High Performance Blockchain Deserializer

Congyu

Last update: Dec 28, 2021

Related tags

Cryptography parser bitcoin blockchain deserializer pyo3

Overview

bitcoin-explorer

bitcoin_explorer is an efficient library for reading bitcoin-core binary blockchain file as a database (utilising multi-threading).

This package is ported using pyO3 from rust library bitcoin-explorer

Installation

MacOS, Windows, and Linux wheels are provided.

Use pip install bitcoin-explorer to install.

Documentation

This library has a Rust version, go to Rust Documentation

For python documentation, go to Documentation.

Compatibility Note

This package deals with the binary file of another software Bitcoin Core. It might not be compatible with older Bitcoin Core versions.

Examples

It contains one class BitcoinDB.

import bitcoin_explorer as bex

# parse the same path as `--datadir` argument for `bitcoind`.
db = bex.BitcoinDB("~/Bitcoin")

# get the length of the longest chain currently on disk.
db.get_max_height()

# get block of a certain height
db.get_block(1000)

# to retrieve the connected outputs of each inputs as well.
# note that this is inefficient.
# Use `get_block_iter_range(end, connected=True)` for better performance.
db.get_block(1000, connected=True)

# get block hash of a certain height.
db.get_hash_from_height(1000)

# a fast method for getting just the header.
# in memory query, no disk access
db.get_block_header(1000)

# get block of height 1000.
db.get_height_from_hash("some hash")

# get transaction from txid.
# This queries the `levelDB` each time, thus it is relatively slow.
db.get_transaction("some txid")

# get the height of the block which this transaction belongs.
db.get_height_from_txid("some txid")

# get the script type and addresses from a script public key
db.parse_script("some hex script pubic key")

# use iterator
for block in db.get_block_iter_range(start=1000, end=2000):
    do_something_with(block)

# use iterator, iterate over heights
for block in db.get_block_iter_array(heights=[1, 3, 5, 7, 9]):
    do_something_with(block)
    
# use iterator, connect outpoints
# !! This requires a minimal amount of 32GB memory. 
for block in db.get_block_iter_range(end=700000, connected=True):
    do_something_with(block)

A little side-project API for me to learn about Blockchain and Tokens

BlockChain API I built this little side project to learn more about Blockchain and Tokens. It might be maintained and implemented to other projects bu

1 Nov 16, 2021

A workshop to build an NFT smart contract on the polygon blockchain

Polygon NFT Workshop This is an interactive workshop that guides you through the steps to deploy an NFT smart contract on the Polygon blockchain. By t

56 Oct 14, 2022

Blockchain Python Implementation

2 Nov 21, 2021

Accounting Cycle Program with Blockchain Component

In the first iteration of my accounting cycle program, I talked about adding in a blockchain component that allows the user to verify the inegrity of

1 Nov 29, 2021

Best blockchain in the world

alphachain Best blockchain in the world!!! Can be used to implement Layer 2 cryptocurrency protocol just click alphachain.py and it will execute autom

0 Feb 18, 2022

Cryptocurrency with implementet Blockchain

1 Mar 24, 2022

Distributed, blockchain based hashtables middleware for deduplication of file uploads to the cloud

distributed-blockchain-based-secure-file-dedupe Searching is Distributed, Block and Access List for each upload is unique and it is stored in a single

1 Dec 2, 2021

This is simple Blockchain ,miner and wallet to send crypto using python

pythonBlockchain-SImple This is simple Blockchain ,miner and wallet to send crypto using python It is simple Blocchain so it can only dobasic work usi

3 Nov 22, 2022

Retrieve ECDSA signature R,S,Z values from blockchain rawtx or txid.

rsz Retrieve ECDSA signature R,S,Z values from blockchain rawtx or txid. Info The script parse the data of rawtx to fetch all the inputs in the transa

29 Nov 18, 2022

Comments

Erron on initialization (pyo3_runtime.PanicException)

I get following error during initialization (Python 3.8.11).

import bitcoin_explorer as bex db = bex.BitcoinDB(r"D:\BTC\Chain")

[19:02:59] INFO - Start loading block_index [19:03:00] INFO - Longest chain: 702696 thread '' panicked at 'called Result::unwrap() on an Err value: Utf8Error { valid_up_to: 58, error_len: Some(1) }', C:\Users\runneradmin.cargo\registry\src\github.com-1ecc6299db9ec823\leveldb-0.8.6\src\database\error.rs:29:72 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace Traceback (most recent call last): File "C:/Users/T/PycharmProjects/BTC_explorer/main.py", line 3, in db = bex.BitcoinDB(r"D:\BTC\Chain") File "D:\anaconda3\envs\env_btc\lib\site-packages\bitcoin_explorer_init_.py", line 82, in init self.db = _BitcoinDB(str(path), tx_index) pyo3_runtime.PanicException: called Result::unwrap() on an Err value: Utf8Error { valid_up_to: 58, error_len: Some(1) } Process finished with exit code 1

I have data downloaded using Bitcoin Core in D:\BTC\Chain\blocks (.dat files) and D:\BTC\Chain\blocks\index (.ldb files).
bug

opened by TomaszKot2 13

Why the results doesn't match?

I'm using the explorer in a regtest mode bitcoin-core and when I use the function "db.get_transaction()" for a specific transaction, it returns the following data:

[23:35:22] INFO - Start loading block_index
[23:35:23] INFO - Successfully opened tx_index DB!
{'txid': '4f820b821643587d0fb89720e0b2e8e32b6db4533e12c94ec1fee16b3539f7ab', 'input': ({'txid': '39a1aca5790d939d75159e43fc02c77a621c3c6119bc18fd32efee66e17b5657', 'vout': 1},), 'output': ({'value': 2000000000, 'addresses': ('bc1qjpjrryrqxykpe7x7uq63a0jfcc0upral3682a7',)}, {'value': 1999999718, 'addresses': ('bc1qxp54zryddh5jlqnd0u3cmrkt6j7eyrsq2769uu',)})}

If I check the same transaction with bitcoin-cli gettransaction I get the following data:

{
  "amount": 20.00000000,
  "confirmations": 10,
  "blockhash": "0d93629f7e02ec6c5f5872e36a339df37503629ce77ddf217aa6d40640bad60a",
  "blockheight": 203,
  "blockindex": 3,
  "blocktime": 1659992136,
  "txid": "4f820b821643587d0fb89720e0b2e8e32b6db4533e12c94ec1fee16b3539f7ab",
  "walletconflicts": [
  ],
  "time": 1659991965,
  "timereceived": 1659991965,
  "bip125-replaceable": "no",
  "details": [
    {
      "address": "bcrt1qjpjrryrqxykpe7x7uq63a0jfcc0uprale4953y",
      "category": "receive",
      "amount": 20.00000000,
      "label": "",
      "vout": 0
    }
  ],
  "hex": "0200000000010157567be166eeef32fd18bc19613c1c627ac702fc439e15759d930d79a5aca1390100000000feffffff0200943577000000001600149064319060312c1cf8dee0351ebe49c61fc08fbfe6923577000000001600143069510c8d6de92f826d7f238d8ecbd4bd920e000247304402204b3999d93bca110be6985b765d6ad990b0517c1531a862cf252030dbf08261c40220484c01ce5656c79890fb3db9b1d727bbea7def0b17f0fe3335fe52653969287801210382d5768096f162475257659c99556999cea6d622a4c583ddc7d5fd40ea536118af000000"
}

Why the data don't match? for example, doesn't appear in the return of the bitcoin explorer, or if I check the transaction that appears in the input of the bitcoin explorer ('39a1aca5790d939d75159e43fc02c77a621c3c6119bc18fd32efee66e17b5657') bitcoin-cli returns that is invalid.

Thanks for reading.

opened by JavierPorron 1

Releases(v1.2.15)

v1.2.15(Nov 1, 2021)
Update backend to fix plaintable size bug

Use tuple to replace list

Source code(tar.gz)
Source code(zip)
v.1.2.13(Oct 20, 2021)

Update backend to v1.2.11
Source code(tar.gz)
Source code(zip)
v1.2.12(Oct 17, 2021)

Source code(tar.gz)
Source code(zip)
v1.2.11(Oct 16, 2021)
backend update

simplify startup process

Source code(tar.gz)
Source code(zip)
v1.2.10(Oct 14, 2021)
Fix: special treatment for genesis transaction

Fix: get_connected_transaction and get_connected_block now return Err if some outpoints can't be found

Enhancement: add new API get_block_count

Source code(tar.gz)
Source code(zip)
v1.2.9(Oct 14, 2021)

Fixed py36 compatibility issue.
Source code(tar.gz)
Source code(zip)
v1.2.4(Oct 12, 2021)
Critical bug fixes

improve error handling

stop iterator correctly

ensure iterator correct order

Source code(tar.gz)
Source code(zip)
v1.2.3-1(Oct 12, 2021)

Source code(tar.gz)
Source code(zip)
v1.2.3(Oct 12, 2021)

Source code(tar.gz)
Source code(zip)
v1.2.1(Oct 12, 2021)

Reduce memory usage to 4GB for iter_block(connected=True)
Source code(tar.gz)
Source code(zip)
v1.1.3(Oct 10, 2021)

Make correction to docstring
Source code(tar.gz)
Source code(zip)
v1.1.2(Oct 9, 2021)

Update to newer backend crate (not much an impact for python, pyO3 is too slow).
Source code(tar.gz)
Source code(zip)
v1.0.9(Oct 6, 2021)

Fix warning log info blocking issue.
Source code(tar.gz)
Source code(zip)
v1.0.8(Oct 6, 2021)
add back TXID duplicate warning

compress TXID to 128 key (reduce memory)

Source code(tar.gz)
Source code(zip)
v1.0.6(Oct 5, 2021)
Reduce Memory Usage (simplify Simple Block Format)

Improve iterator performance

Source code(tar.gz)
Source code(zip)
v1.0.5(Oct 4, 2021)

Fix error message typo
Source code(tar.gz)
Source code(zip)
v1.0.4(Oct 4, 2021)

Fix an issue where bitcoinDB failed to initialize due to levelDB panic.
Source code(tar.gz)
Source code(zip)
v1.0.3(Oct 2, 2021)

update package documentation
Source code(tar.gz)
Source code(zip)
v1.0.2(Oct 2, 2021)

Update bitcoin-explorer crate to v1.0.2
Source code(tar.gz)
Source code(zip)

Owner

Congyu

GitHub https://pypi.org/project/bitcoin-explorer/

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions.

2.3k Jan 1, 2023

Learn Blockchains by Building One, A simple Blockchain in Python using Flask as a micro web framework.

Blockchain ✨ Learn Blockchains by Building One Yourself Installation Make sure Python 3.6+ is installed. Install Flask Web Framework. Clone this repos

46 Jan 5, 2023

Tools for running airdrop and token distribution campaigns on the Solana blockchain.

Overview This repository contains some of the scripts we have used for running our airdrop campaigns and other distributions. Initially, all of these

147 Nov 17, 2022

seno-blockchain is just a fork of Chia, designed to be efficient, decentralized, and secure

seno-blockchain https://seno.uno Seno is just a fork of Chia, designed to be efficient, decentralized, and secure. Here are some of the features and b

27 Jul 2, 2022

Ceres is a combine harvester designed to harvest plots for Chia blockchain and its forks using proof-of-space-and-time(PoST) consensus algorithm.

Ceres Combine-Harvester Ceres is a combine harvester designed to harvest plots for Chia blockchain and its forks using proof-of-space-and-time(PoST) c

38 Nov 14, 2022

High Performance Blockchain Deserializer

Related tags

Overview

bitcoin-explorer

Installation

Documentation

Compatibility Note

Examples

You might also like...

A little side-project API for me to learn about Blockchain and Tokens

A workshop to build an NFT smart contract on the polygon blockchain

Blockchain Python Implementation

Accounting Cycle Program with Blockchain Component

Best blockchain in the world

Cryptocurrency with implementet Blockchain

Distributed, blockchain based hashtables middleware for deduplication of file uploads to the cloud

This is simple Blockchain ,miner and wallet to send crypto using python

Retrieve ECDSA signature R,S,Z values from blockchain rawtx or txid.

Comments

Erron on initialization (pyo3_runtime.PanicException)

Why the results doesn't match?

Releases(v1.2.15)

v1.2.15(Nov 1, 2021)

v.1.2.13(Oct 20, 2021)

v1.2.12(Oct 17, 2021)

v1.2.11(Oct 16, 2021)

v1.2.10(Oct 14, 2021)

v1.2.9(Oct 14, 2021)

v1.2.4(Oct 12, 2021)

v1.2.3-1(Oct 12, 2021)

v1.2.3(Oct 12, 2021)

v1.2.1(Oct 12, 2021)

v1.1.3(Oct 10, 2021)

v1.1.2(Oct 9, 2021)

v1.0.9(Oct 6, 2021)

v1.0.8(Oct 6, 2021)

v1.0.6(Oct 5, 2021)

v1.0.5(Oct 4, 2021)

v1.0.4(Oct 4, 2021)

v1.0.3(Oct 2, 2021)

v1.0.2(Oct 2, 2021)

Owner

Congyu

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Learn Blockchains by Building One, A simple Blockchain in Python using Flask as a micro web framework.

Tools for running airdrop and token distribution campaigns on the Solana blockchain.

seno-blockchain is just a fork of Chia, designed to be efficient, decentralized, and secure

Ceres is a combine harvester designed to harvest plots for Chia blockchain and its forks using proof-of-space-and-time(PoST) consensus algorithm.

blockchain address database

A repository for Algogenous Smart Contracts created on the Algorand Blockchain.

A repository for Algogenous Smart Contracts created on the Algorand Blockchain.

Python implementation of a blockchain.

An BlockChain Based solution for storing the medical records