Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Overview

Ethereum ETL

Build Status Join the chat at https://gitter.im/ethereum-eth Telegram Discord

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Full documentation available here.

Quickstart

Install Ethereum ETL:

pip3 install ethereum-etl

Export blocks and transactions (Schema, Reference):

> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Export ERC20 and ERC721 transfers (Schema, Reference):

> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> ethereumetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install ethereum-etl[streaming]
> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 ethereumetl.py

Useful Links

Running Tests

> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

  1. Install Docker https://docs.docker.com/install/

  2. Build a docker image

     > docker build -t ethereum-etl:latest .
     > docker image ls
    
  3. Run a container out of the image

     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io
    
  4. Run streaming to console or Pub/Sub

     > docker build -t ethereum-etl:latest -f Dockerfile .
     > echo "Stream to console"
     > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
     > echo "Stream to Pub/Sub"
     > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects//topics/crypto_ethereum
    

Projects using Ethereum ETL

  • Google - Public BigQuery Ethereum datasets
  • Nansen by D5 - Analytics platform for Ethereum
Comments
  • Add support for internal transactions

    Add support for internal transactions

    Parity provides APIs for retrieving internal transactions. trace_filter API allows retrieving traces for a range of blocks. This method is already supported by web3.py parity module.

    The scope of this task is to

    1. Design the model (the field names from this PR can be used as a basis: https://github.com/blockchain-etl/ethereum-etl/pull/72).
    2. Implement the export job (the job implementation will mostly resemble the flow in ExportTokenTransfersJob.
    3. Write unit tests (an example can be found here)
    4. Add documentation to README (an example can be found here) (5. Add instructions for how to start parity/geth i.e. what options to use).

    =======================

    Remarks:

    • There will be another task to add the BigQuery schema here
    • Geth also provides support for tracing transactions. There will be another task for implementing internal transactions exporter job for Geth. https://github.com/ethereum/go-ethereum/pull/15516, https://www.linkedin.com/pulse/geth-18-iceberg-michael-hunt-jr-
    opened by medvedev1088 21
  • Add Dockerfile

    Add Dockerfile

    The scope for this task is:

    1. Add Dockerfile (this can be used as a basis). Both ipc and rpc should be supported.
    2. Test it.
    3. Update documentation in README with instructions on how to use it.
    help wanted 
    opened by medvedev1088 15
  • Installing Ethereum-ETL

    Installing Ethereum-ETL

    Dear all,

    unfortunately I can't install ethereum-etl based on the instructions in the readme. Running "pip3 install ethereum-etl" as well as cloning the repo and running "pip3 install -e ." always results in:

    Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b79bd0750>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/ethereum-etl/

    Could not find a version that satisfies the requirement ethereum-etl (from versions: ) No matching distribution found for ethereum-etl

    Does anybody has an idea what the problem is?

    Thanks and greets Peter

    opened by PeHo89 13
  • Add support for geth traces

    Add support for geth traces

    The scope of this task:

    1. Design the model for geth traces (trace.py can be used as a basis).
    2. Implement the export and extract jobs. 2.a The export job should use traceBlockByNumber (https://github.com/ethereum/go-ethereum/pull/15516) and output raw JSON responses. It should use callTracer, an example JS API call: debug.traceBlockByNumber('0x250252', {'tracer': 'callTracer'}); the same is available via JSON RPC. The job should use BatchWeb3Provider similar to export_blocks_job.py. 2.b The extract job should take the file with raw JSON responses, parses them, map to our domain entity and output to CSV file.
    3. Write unit tests (real mainnet responses should be used in test cases, so the same tests can be run against geth node).
    4. Add documentation to README (an example can be found here). Include instructions for how to start geth i.e. what options to use (e.g. geth --gcmode archive)

    ======================= Notes:

    • Parity traces are already supported https://github.com/blockchain-etl/ethereum-etl/pull/104.
    opened by medvedev1088 13
  • web3 upgrade

    web3 upgrade

    Here are the changes to upgrade web3.py for issue #308.

    • One major change was dropping the support for python 3.6 on ethereum etl since web3.py dropped 3.6 support
    • You may want to take a look at these notes on upgrading from web3.py v4 to v5 to see if anything jumps out at you.
    • I did still have the infura test falling locally but I think I needed to just dig in a little more and see why my setup wasn't working. If I have time tomorrow I will look at that.
    • I had to add a few of those mock response files. I copied ones you already had in there and modified them a little.
    • Also, I did not test the docker file but noticed it was using python3.6 so just changed the image it was pulling from to python3.7 hope that works.

    Let me know what you think, happy to tweak this some more if needed.

    opened by dbfreem 11
  • Windows - SyntaxError: invalid syntax

    Windows - SyntaxError: invalid syntax

    C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master>python export_blocks_and_transactions.py --start-block 0 --end-block 500000 \ --ipc-path ~/Library/Ethereum/geth.ipc --blocks-output blocks.csv --transactions-output transactions.csv Traceback (most recent call last): File "export_blocks_and_transactions.py", line 4, in from ethereumetl.ipc import IPCWrapper File "C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master\ethereumetl\ipc.py", line 5, in from web3.providers.ipc import get_default_ipc_path, PersistantSocket File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3_init_.py", line 8, in from web3.main import Web3 # noqa: E402 File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\main.py", line 26, in from web3.providers.eth_tester import ( File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers_init_.py", line 8, in from .websocket import WebsocketProvider # noqa: F401 File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers\websocket.py", line 9, in import websockets File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets_init_.py", line 3, in from .client import * File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\client.py", line 20, in from .protocol import WebSocketCommonProtocol File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\protocol.py", line 18, in from .compatibility import asyncio_ensure_future File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\compatibility.py", line 15 asyncio_ensure_future = asyncio.async # Python < 3.5 ^ SyntaxError: invalid syntax

    bug duplicate 
    opened by RaianeQC 11
  • adds support for non-mainnet in etl stream

    adds support for non-mainnet in etl stream

    relates #178

    DEMO on ronin network, an eth sidechain. Notice that latest block can now be read, thanks to middleware injection.

    2021-08-12 19:24:34,321 - ProgressLogger [INFO] - Started work. Items to process: 1.
    2021-08-12 19:24:34,375 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
    2021-08-12 19:24:34,375 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.054600.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Started work.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000169.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Started work.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000143.
    
    opened by numonedad 10
  • Ethereum data export error

    Ethereum data export error

    When i use “ethereumetl export_all -s 0 -e 5999999 -b 100000 -p file://$HOME/Library/Ethereum/geth.ipc -o output”,The following error occurred:“ValueError: Unknown uri scheme /datalake/blockdata/geth.ipc”,Ask what is the reason

    opened by sznariOsmosis 10
  • Add support for extracting Origin Protocol data

    Add support for extracting Origin Protocol data

    This implements a prototype that extracts data generated by Origin Protocol's open source platform for building decentralized marketplaces on Ethereum and IPFS.

    Fixes https://github.com/blockchain-etl/ethereum-etl/issues/215

    opened by franckc 10
  • ValueError: result is None in response (export_all)

    ValueError: result is None in response (export_all)

    I run it on ubuntu

    ethereumetl export_all -s 4900000 -e 4999999 -b 100000 -p https://mainnet.infura.io -o output

    but after n time it has error:

    File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 70, in _export │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 70, in _export total_items=self.end_block - self.start_block + 1 │ total_items=self.end_block - self.start_block + 1 File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 48, in execute │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 48, in execute self.executor.submit(self._fail_safe_execute, work_handler, batch) │ self.executor.submit(self._fail_safe_execute, work_handler, batch) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit self._check_completed_futures() │ self._check_completed_futures() File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_compl│ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_comp eted_futures │leted_futures future.result() │ future.result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result │ File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() │ return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result │ File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception │ raise self._exception File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run │ File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) │ result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe_│ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe execute │_execute work_handler(batch) │ work_handler(batch) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in _export_batch │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in _export_batch blocks = [self.block_mapper.json_dict_to_block(result) for result in results] │ blocks = [self.block_mapper.json_dict_to_block(result) for result in results] File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in <listcomp> │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in <listcomp> blocks = [self.block_mapper.json_dict_to_block(result) for result in results] │ blocks = [self.block_mapper.json_dict_to_block(result) for result in results] File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 58, in rpc_response_batch_to_results │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 58, in rpc_response_batch_to_results yield rpc_response_to_result(response_item) │ yield rpc_response_to_result(response_item) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 67, in rpc_response_to_result │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 67, in rpc_response_to_result raise ValueError(error_message) │ raise ValueError(error_message) ValueError: result is None in response {'jsonrpc': '2.0', 'id': 0, 'error': {'code': -32603, 'message': 'request proc│ValueError: result is None in response {'jsonrpc': '2.0', 'id': 0, 'error': {'code': -32603, 'message': 'request pro essing failed'}}.

    opened by minhtrih 10
  • Error in exporting traces using Parity

    Error in exporting traces using Parity

    @medvedev1088 We used Parity to export traces. It works perfectly for almost every block. BUT error happens in some block ranges, such as 2,280,000~2,400,000 and 3800000~4,000,000. Error:web3.utils.threads.Timeout: 60 seconds image

    Our server has far more than 8 GB memory. Maybe we could change default time. But we could not figure out the correct command

    opened by cxd1995 10
  • Add decoding log functions here

    Add decoding log functions here

    https://github.com/blockchain-etl/ethereum-etl/blob/6710e6b894789d83ce3c28db79b4ec5bf2ec3845/ethereumetl/jobs/export_receipts_job.py#L78

    Add feature to decode logs in the exportation of receipts

    opened by pauline-thrackle 0
  • [Errno 101]

    [Errno 101]

    Command: ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000
    --blocks-output blocks.csv --transactions-output transactions.csv
    --provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c Error: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mainnet.infura.io', port=443): Max retries exceeded with url: /v3/7aef3f0cd1f64408b163814b22cc643c (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4b8fd63910>: Failed to establish a new connection: [Errno 101] Network is unreachable')) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mainnet.infura.io', port=443): Max retries exceeded with url: /v3/7aef3f0cd1f64408b163814b22cc643c (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4b8fd621d0>: Failed to establish a new connection: [Errno 101] Network is unreachable')) how to solve this problem?

    opened by Facebeside 0
  • M1 mac issue Symbolic Execution not available: No module named 'mythril.ether'

    M1 mac issue Symbolic Execution not available: No module named 'mythril.ether'

    System:

    • M1 Mac
    • Python 3.9.15
    • pip 22.1.2
    • Poetry (version 1.2.2)

    pyproject.toml

    [tool.poetry]
    name = "asd"
    version = "0.1.0"
    description = ""
    authors = ["Your Name <[email protected]>"]
    readme = "README.md"
    packages = [{include = "asd"}]
    
    [tool.poetry.dependencies]
    python = "^3.9"
    ethereum-etl = "^2.1.1"
    pyetherchain = "^0.3.2"
    blake2b-py = { git = "https://github.com/ethereum/blake2b-py.git", tag="v0.1.4" }
    mythril = {extras = ["mythril"], version = "^0.23.12"}
    
    
    
    [build-system]
    requires = ["poetry-core"]
    build-backend = "poetry.core.masonry.api"
    

    reprod script

    from ethereumetl.service.eth_contract_service import EthContractService
    
    bytecode_as_str = '608060405234801561001057600080fd5b506040516111893803806111898339818101604052602081101561003357600080fd5b810190808051906020019092919050505060016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000208190555080600260006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff1602179055506b033b2e3c9fd0803ce8000000600381905550600160048190555050611095806100f46000396000f3fe608060405234801561001057600080fd5b50600436106100b45760003560e01c80636924500911610071578063692450091461020d578063957aa58c146102175780639c52a7f114610235578063bf353dbb14610279578063d9638d36146102d1578063ebecb39d14610346576100b4565b80631504460f146100b95780631a0b287e146100e757806329ae81141461012957806336569e7714610161578063495d32cb146101ab57806365fae35e146101c9575b600080fd5b6100e5600480360360208110156100cf57600080fd5b810190808035906020019092919050505061039e565b005b610127600480360360608110156100fd57600080fd5b810190808035906020019092919080359060200190929190803590602001909291905050506105bd565b005b61015f6004803603604081101561013f57600080fd5b8101908080359060200190929190803590602001909291905050506107d7565b005b6101696109dc565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6101b3610a02565b6040518082815260200191505060405180910390f35b61020b600480360360208110156101df57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610a08565b005b610215610b36565b005b61021f610c27565b6040518082815260200191505060405180910390f35b6102776004803603602081101561024b57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610c2d565b005b6102bb6004803603602081101561028f57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610d5b565b6040518082815260200191505060405180910390f35b6102fd600480360360208110156102e757600080fd5b8101908080359060200190929190505050610d73565b604051808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020018281526020019250505060405180910390f35b61039c6004803603606081101561035c57600080fd5b810190808035906020019092919080359060200190929190803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610db7565b005b6000806001600084815260200190815260200160002060000160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff166359e02dd76040518163ffffffff1660e01b81526004016040805180830381600087803b15801561041e57600080fd5b505af1158015610432573d6000803e3d6000fd5b505050506040513d604081101561044857600080fd5b810190808051906020019092919080519060200190929190505050915091506000816104755760006104b1565b6104b061049461048c8560001c633b9aca0061100b565b600354611037565b6001600087815260200190815260200160002060010154611037565b5b9050600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16631a0b287e85836040518363ffffffff1660e01b815260040180838152602001807f73706f740000000000000000000000000000000000000000000000000000000081525060200182815260200192505050600060405180830381600087803b15801561055857600080fd5b505af115801561056c573d6000803e3d6000fd5b505050507fdfd7467e425a8107cfd368d159957692c25085aacbcf5228ce08f10f2146486e84848360405180848152602001838152602001828152602001935050505060405180910390a150505050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610671576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b6001600454146106e9576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f6d617400000000000000000000000000000000000000000000000000000000008214156107315780600160008581526020019081526020016000206001018190555061079f565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450505050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020541461088b576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b600160045414610903576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f706172000000000000000000000000000000000000000000000000000000000082141561093757806003819055506109a5565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a4505050565b600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b60035481565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610abc576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60016000808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a45050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610bea576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60006004819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450565b60045481565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610ce1576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60008060008373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a45050565b60006020528060005260406000206000915090505481565b60016020528060005260406000206000915090508060000160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff16908060010154905082565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610e6b576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b600160045414610ee3576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f7069700000000000000000000000000000000000000000000000000000000000821415610f6557806001600085815260200190815260200160002060000160006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff160217905550610fd3565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450505050565b600080821480611028575082828385029250828161102557fe5b04145b61103157600080fd5b92915050565b600081611050846b033b2e3c9fd0803ce800000061100b565b8161105757fe5b0490509291505056fea265627a7a72315820b20103aad12e7c2a10372688d31c070bc7aa268459f8bc3a58d4298c7f6bdf0964736f6c634300050c003200000000000000000000000035d1b3f3d7966a1dfe207aa4514c12a259a0492b'
    
    def main():
        contract_service = EthContractService()
        func_sigs = contract_service.get_function_sighashes(bytecode_as_str)
        print(func_sigs)
    
    
    if __name__ == '__main__':
        main()
    
    opened by 007vasy 0
  • stream command failed to extract contract information

    stream command failed to extract contract information

    Command : ethereumetl stream --start-block 15789281 -e block,transaction,log,token_transfer,contract --log-file log.txt --provider-uri https://mainnet.infura.io/v3/9aa3d95b3bc440fa88ea12eaa4456161

    Error : 2022-11-30 16:07:55,003 - root [ERROR] - An exception occurred while syncing block data. Traceback (most recent call last): File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/streaming/streamer.py", line 77, in _do_stream synced_blocks = self._sync_cycle() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/streaming/streamer.py", line 98, in _sync_cycle self.blockchain_streamer_adapter.export_all(self.last_synced_block + 1, target_block) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 62, in export_all traces = self._export_traces(start_block, end_block) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 159, in _export_traces job.run() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/jobs/base_job.py", line 30, in run self._end() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/jobs/export_traces_job.py", line 106, in _end self.batch_work_executor.shutdown() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/batch_work_executor.py", line 97, in shutdown self.executor.shutdown() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown self._check_completed_futures() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures future.result() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result return self.__get_result() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/batch_work_executor.py", line 63, in _fail_safe_execute work_handler(batch) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/jobs/export_traces_job.py", line 90, in _export_batch json_traces = self.web3.parity.traceBlock(block_number) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/module.py", line 57, in caller result = w3.manager.request_blocking(method_str, File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/manager.py", line 198, in request_blocking return self.formatted_response(response, File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/manager.py", line 171, in formatted_response raise ValueError(response["error"]) ValueError: {'code': -32601, 'message': 'the method trace_block does not exist/is not available'}

    opened by infocodiste 1
  • can't support proxy contract erc20

    can't support proxy contract erc20

    I looked at the source code and found that bytecode is used to judge whether it is erc20, but many contracts currently use the proxy contract upgrade solution to deploy the contract, which makes it impossible to judge whether it is erc20

    opened by SuperEdison 2
Releases(v2.1.2)
Owner
Blockchain ETL
Facilitating data science on blockchain data. Available in Google BigQuery https://goo.gl/oY5BCQ
Blockchain ETL
Python repo to create blockchain CSVs

staketaxcsv Python repo to create blockchain CSVs for Terra (LUNA), Solana (SOL), and Cosmos (ATOM). CSV codebase for stake.tax Community contribution

null 187 Dec 31, 2022
Lottery by Ethereum Blockchain

Lottery by Ethereum Blockchain Set your web3 provider url in .env PROVIDER=https://mainnet.infura.io/v3/<YOUR-INFURA-TOKEN> Create your source file .

John Torres 3 Dec 23, 2021
J. Brandon Walker 1 May 13, 2022
Bridge between L1 (Ethereum) and L2 (cheapETH)

The ETH chain and the cheapETH chain. We can assume the ETH chain has ~1000x more value than the cheapETH chain.

null 107 Oct 12, 2022
Tutela: an Ethereum and Tornado Cash Anonymity Tool

Tutela: an Ethereum and Tornado Cash Anonymity Tool The repo contains open-source code for Tutela, an anonymity tool for Ethereum and Tornado Cash use

TutelaLabs 96 Dec 5, 2022
A simple Ethereum mining pool

A simple getWork pool for ethereum mining Payouts are still manual. TODO: write payouts when someone mines 10 blocks. Also, make the submit actually

null 93 Oct 5, 2022
A simple Ethereum mining pool

A simple getWork pool for ethereum mining

null 93 Oct 5, 2022
How to setup a multi-client ethereum Eth1-Eth2 merge testnet

Mergenet tutorial Let's set up a local eth1-eth2 merge testnet! Preparing the setup environment In this tutorial, we use a series of scripts to genera

Diederik Loerakker 24 Jun 17, 2022
Basic Ethereum Miner Lib

EthMine ⛏ Basic Ethereum Miner Library. Developers can integrate this algorithm to mine blocks from their ethereum supported chain efficiently. Instal

Jaival Patel 1 Oct 30, 2021
Ceres is a combine harvester designed to harvest plots for Chia blockchain and its forks using proof-of-space-and-time(PoST) consensus algorithm.

Ceres Combine-Harvester Ceres is a combine harvester designed to harvest plots for Chia blockchain and its forks using proof-of-space-and-time(PoST) c

null 38 Nov 14, 2022
Tools for running airdrop and token distribution campaigns on the Solana blockchain.

Overview This repository contains some of the scripts we have used for running our airdrop campaigns and other distributions. Initially, all of these

null 147 Nov 17, 2022
seno-blockchain is just a fork of Chia, designed to be efficient, decentralized, and secure

seno-blockchain https://seno.uno Seno is just a fork of Chia, designed to be efficient, decentralized, and secure. Here are some of the features and b

Denis Erygin 27 Jul 2, 2022
A little side-project API for me to learn about Blockchain and Tokens

BlockChain API I built this little side project to learn more about Blockchain and Tokens. It might be maintained and implemented to other projects bu

Loïk Mallat 1 Nov 16, 2021
This is simple Blockchain ,miner and wallet to send crypto using python

pythonBlockchain-SImple This is simple Blockchain ,miner and wallet to send crypto using python It is simple Blocchain so it can only dobasic work usi

null 3 Nov 22, 2022
Learn Blockchains by Building One, A simple Blockchain in Python using Flask as a micro web framework.

Blockchain ✨ Learn Blockchains by Building One Yourself Installation Make sure Python 3.6+ is installed. Install Flask Web Framework. Clone this repos

Vaibhaw 46 Jan 5, 2023
blockchain address database

Blockchain Address Ownership Database The database is in data/addresses.db This is a SQLite database of addresses from several blockchains. It's obtai

null 37 Nov 26, 2022
A repository for Algogenous Smart Contracts created on the Algorand Blockchain.

Smart Contacts Alogrand Smart Contracts using Choice Coin. Read Docs for how to implement Algogenous Smart Contracts for your own applications. Smart

Choice Coin 3 Dec 20, 2022
A repository for Algogenous Smart Contracts created on the Algorand Blockchain.

Smart Contacts This Repository is dedicated to code for Alogrand Smart Contracts using Choice Coin. Read Docs for how to implement Algogenous Smart Co

Choice Coin 3 Dec 20, 2022
High Performance Blockchain Deserializer

bitcoin_explorer is an efficient library for reading bitcoin-core binary blockchain file as a database (utilising multi-threading).

Congyu 2 Dec 28, 2021