Ops.Syncretism is an open source options search engine based on Yahoo! Finance market data.
Dependencies are Clojure, Leiningen, NodeJS, and MariaDB. We try to minimize dependencies to make it easy to install and run locally.
Join us on Discord for any question you might have. Use Github Issues for bug reports & feature requests.
This crawlers targets Yahoo Finance market data to extract options related information.
Dependency on MariaDB or MySQL. Set parameters in datops-crawler/resources/db.edn as illustrated Here.
For example, if your user is datops and the created database is opts, the db.edn becomes:
{:dbtype "mysql"
:dbname "opts"
:user "datops"
:password "12345password"
:rewriteBatchedStatements true}
You should also set datops-crawler/resources/config.edn as follows:
{;; Should we crawl out of market hours
:force-crawl false
;; Do we log debug info
:debug false
;; How often to we save & reorder the queue & the proxies (in seconds)
:t-reorder 60
;; How often do we fully clean the queue (remove expired options & gather new symbols)
:t-clean-queue 86400
;; Number of endpoints to use.
:nb-endpoints 2
;; Max Batch size when writing to db
:batch-size 10000
;; Where to save the options data
:save-path "options"}
Initialize the db with lein run --init.
Run it with lein run
Logs are written in ./opts-crawler.log
The crawler uses a scheduler to make sure that any [Ticker, Date] pair that failed to be
crawled will be given priority in the queue.
With the above configuration file, the data is stored in the folder options.
The crawler creates one folder per day, and one gzipped file per symbol.
In options db:
livetable contains some essential information about the latest measured state of all options. Seelive-table-defindb_init.clj.live_quotecontains essential information about the underlying stock. Seelive-quote-table-defin the same file.fundamentalscontains all the fundamental data available for a given ticker (e.g. earnings, dividends etc...).
As with the crawler, there is a dependency on MariaDB or MySQL. Set parameters in datops-backend/resources/db.edn as illustrated Here.
For example:
{:dbtype "mysql"
:dbname "opts"
:user "datops"
:password "12345password"
:rewriteBatchedStatements true}
Run it with lein ring server-headless, will default to port 3000.
If you want to run it on a different port, simply do
lein ring server-headless 3001
to run it on the port 3001 for example.
Depends on NodeJS.
Change the server address in src/live_opts/communication.cljs to point to your local datops-backend, in the above example
it would become "http://localhost:3000/".
Build with
npm install
./node_modules/.bin/shadow-cljs release :frontend
Copy & paste the /public folder wherever your local http server is.
Depends on NodeJS.
Change the server address in src/live_opts/communication.cljs to point to your local datops-backend, in the above example
it would become "http://localhost:3000/".
Build with
npm install
./node_modules/.bin/shadow-cljs release :frontend
Copy & paste the /public folder wherever your local http server is.
Very early stage, software to do post processing on the crawled data. For the moment:
- can be used to calculate the yield & monthly yield of all non expired options.
- can be used to create aligned time series from the crawled data.
Contain the shared bits of the different utilities above.
Quick examples / experiments regarding what can be done with the data.
Syncretism is licensed under the GNU Affero General Public License 3 or any later version at your choice.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent