Comments (4)
Alright, I have what seems to be a working set of dependencies installed! I haven't deployed the airflow DAGs yet, however I
'll be trying that soon.
Here are the dependency versions I landed on:
eth-hash=0.3.3
web3==5.29.2
eth-abi==2.1.1
eth-rlp==0.2.1
ethereum-etl
from ethereum-etl-airflow.
Yup, I think a lot of this could be solved if ethereum-etl could update to a newer version of eth-utils :-D
Thank you for confirming this is a common problem with composer as I thought maybe I was just going crazy!
I think for other new users it might be worth updating the readme with some of this information so others don't struggle as much as I did. If that's useful I'll happily create a PR for it :)
from ethereum-etl-airflow.
I suggest if you have a working setup, then pin everything (including Composer version) and only update once in a while, since it would be annoying to have to revisit dependency conflicts just for small version bumps.
Composer is remarkably pedantic about conflicts in additional pypi packages. Many times I've had to manually tinker with dependency conflicts where I have had no issues when testing locally. When updating dependencies, I think it does something "cleverer" than just pip install -r requirements.txt
-- it can be infuriating, and I agree with your findings that adding eth-hash
and/or eth-rlp
have sometimes unblocked me with ethereum-etl
and its forks. In fact, that's exactly why eth-hash==0.3.3
is included in requirements_airflow.txt
Fwiw, our current production setup is:
composer-2.0.25-airflow-2.2.5
- pypi packages pinned:
eth-hash==0.3.3, ethereum-etl==2.0.5
- I don't think the version of
ethereum-etl
should make any difference between2.0.3
and current latest2.1.1
- Saying this, I might update
requirements_airflow.txt
for consistency
- I don't think the version of
When I SSH into the airflow-worker and pip list
, it includes:
eth-abi==2.1.1
eth-hash=0.3.3
eth-rlp==0.2.1
ethereum-etl==2.0.5
google-ads==18.0.0
protobuf==3.20.0
web3==5.30.0
So, apart from web3
, very similar to yours in the end but we were fortunate in only having to add eth-hash
, at least for ethereum-etl
.
from ethereum-etl-airflow.
Thanks for the suggestion about eth-utils
. I can't prioritise this work right now, but I will come back to this and try to update the dependencies and improve this.
For now, feel free to raise a PR for updating the README and put me as a reviewer or assignee, happy to help here.
from ethereum-etl-airflow.
Related Issues (20)
- Add contract parsers for SushiSwap HOT 6
- Add DAG that queries data from BigQuery and saves it to csv
- Support parsing anonymous events
- Support parsing anonymous events
- etl & bigquery fee
- Chainlink parsing pointed at the wrong contracts HOT 2
- composer-1.8.3-airflow-1.10.3 not exist HOT 2
- Why is ethereum_load_dag's start date different from the first run date? HOT 2
- Missing logs_by_topic table HOT 3
- BigQuery Aggregator_ETH_USD_event_AnswerUpdated event not functioning
- Issue: Permission bigquery.tables.create denied on dataset blockchain-etl-internal:common (or it may not exist). HOT 1
- Create a function that generates SQL for a table definition that can be run in BigQuery HOT 1
- Missing `blocks` table when running first time HOT 1
- Failure building polygon_etl
- Add verification of empty fields
- add `contracts.bytecode_hash` field
- Missing`_history` table
- DAG import error HOT 1
- tokens table and amended tokens view missing some tokens like PEPE. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ethereum-etl-airflow.