How to setup a multi-client ethereum Eth1-Eth2 merge testnet

Overview

Mergenet tutorial

Let's set up a local eth1-eth2 merge testnet!

Preparing the setup environment

In this tutorial, we use a series of scripts to generate configuration files, and these scripts have dependencies that we need to install. You can either install these dependencies on your host or you can run those scripts inside a docker container. We call this environment setupenv.

Preparing the setup environment on your host:

apt-get install python3-dev python3-pip python3-venv golang

# Check that you have Go 1.16+ installed
go version

# Create, start and install python venv
python -m venv venv 
. venv/bin/activate
pip install -r requirements.txt

# Install eth2-testnet-genesis tool (Go 1.16+ required)
go install github.com/protolambda/eth2-testnet-genesis@latest
# Install eth2-val-tools
go install github.com/protolambda/eth2-val-tools@latest
# You are now in the right directory to run the setupenv commands below.

Alternatively, you can use docker:

docker build -t setupenv .
docker run -i -v $PWD:/mergenet-tutorial \
  -v /etc/passwd:/etc/passwd -h setupenv \
  -u $(id -u):$(id -g) -t setupenv \
  /bin/bash
# docker spawns a shell and inside that run:
cd /mergenet-tutorial
# You are now in the right directory to run the setupenv commands below.

Create chain configurations

Set eth1_genesis_timestamp inside mergenet.yamlto the current timestamp or a timestamp in the future. To use the current timestamp run:

sed -i -e s/GENESIS_TIMESTAMP/$(date +%s)/ mergenet.yaml

Otherwise tweak mergenet.yaml as you like. The current default is to have the Eth2 genesis 10 minutes after the Eth1 genesis.

"$TESTNET_NAME/public/eth1_config.json" # Configure Eth2 chain python generate_eth2_conf.py > "$TESTNET_NAME/public/eth2_config.yaml" ">
# Inside your setupenv: Generate ETH1/ETH2 configs
export TESTNET_NAME="mynetwork"
mkdir -p "$TESTNET_NAME/public" "$TESTNET_NAME/private"
# Configure Eth1 chain
python generate_eth1_conf.py > "$TESTNET_NAME/public/eth1_config.json"
# Configure Eth2 chain
python generate_eth2_conf.py > "$TESTNET_NAME/public/eth2_config.yaml"

Configure tranche(s) of validators, edit genesis_validators.yaml. Note: defaults test-purpose mnemonic and configuration is included already, no need to edit for minimal local setup. Make sure that total of count entries is more than the configured MIN_GENESIS_ACTIVE_VALIDATOR_COUNT (eth2 config).

Prepare Eth2 data

# Inside your setupenv: Generate Genesis Beacon State
eth2-testnet-genesis merge \
  --eth1-config "$TESTNET_NAME/public/eth1_config.json" \
  --eth2-config "$TESTNET_NAME/public/eth2_config.yaml" \
  --mnemonics genesis_validators.yaml \
  --state-output "$TESTNET_NAME/public/genesis.ssz" \
  --tranches-dir "$TESTNET_NAME/private/tranches"

# Build validator keystore for nodes
#
# Prysm likes to consume bundled keystores. Use `--prysm-pass` to encrypt the bundled version.
# For the other eth2 clients, a different secret is generated per validator keystore.
#
# You can change the range of validator accounts, to split keys between nodes.
# The mnemonic and key-range should match that of a tranche of validators in the beacon-state genesis.
export VALIDATOR_NODE_NAME="valclient0"
eth2-val-tools keystores \
  --out-loc "$TESTNET_NAME/private/$VALIDATOR_NODE_NAME" \
  --prysm-pass="foobar" \
  --source-min=0 \
  --source-max=64 \
  --source-mnemonic="lumber kind orange gold firm achieve tree robust peasant april very word ordinary before treat way ivory jazz cereal debate juice evil flame sadness"

Start nodes

This documents how to build the binaries from source, so you can make changes and check out experimental git branches. It's possible to build docker images (or use pre-built ones) as well. Ask the client devs for alternative install instructions.

mkdir clients

mkdir "$TESTNET_NAME/nodes"

You can choose to run clients in two ways:

The docker instructions include how to configure each of the clients. Substitute docker volume-mounts with your own directory layout choices, the instructions are otherwise the same.

Genesis

Now wait for the genesis of the chain! actual_genesis_timestamp = eth1_genesis_timestamp + eth2_genesis_delay

Bonus

Test ETH transaction

Import a pre-mined account into some web3 wallet (e.g. metamask), connect to local RPC, and send a transaction with a GUI.

Run example_transaction.py.

Test deposit

TODO

Test contract deployment

TODO

Comments
  • Add instructions on how to handle bootnode discovery.

    Add instructions on how to handle bootnode discovery.

    Using bootnodes.txt from nocturna works for me in the pyrsm example. The instructions for using a local bootnode via protolambda:eth2-bootnode are yet to be improved.

    opened by clefru 3
  • Prysm Mergenet Installation Instructions

    Prysm Mergenet Installation Instructions

    This PR includes instructions for how to build and run Prysm to comply with the mergenet tutorial. Currently, it requires building from source, but we plan on providing docker images as we make more progress on this throughout the week.

    opened by rauljordan 3
  • Path error in Teku for Docker

    Path error in Teku for Docker

    The command to run Teku from docker contains a typo:

    flag: --data-path "/testnet/nodes/teku0/beacondata" \

    should be: --data-path "/testnet/public/nodes/teku0/beacondata" \

    opened by jschp 1
  • Provide a docker environment to run config generation cmds.

    Provide a docker environment to run config generation cmds.

    There are certain pitfalls with installing the Python3 requirements.txt on other distributions. For instance on the most recent Debian (buster), we see bdist_wheel failures during pip install, and the installed Go version is too old. It is much more convenient and reliable to also use docker to generate configs.

    opened by clefru 1
  • Remove Quotes from ETH2 Config for Prysm

    Remove Quotes from ETH2 Config for Prysm

    Prysm does not like hex values in quoted strings, and it seems like the genesis fork versions are unnecessarily quoted after yaml generation in the python tool. This PR removes the unnecessary quotes.

    @protolambda is there a reason these config values are quoted right now?

    opened by rauljordan 1
  • Block Explorer + misc fixes

    Block Explorer + misc fixes

    Adds the beaconcha.in explorer to the setup. Thanks @guybrush and explorer contributors, this is very useful for debugging. Next up is to to fork the code and implement basic Eth1 transaction indexing.

    opened by protolambda 0
  • Multi client refactor

    Multi client refactor

    • Document to describe compiling clients from source code. (we should clean up the readme later)
    • Script to run a local multi-client testnet (nethermind-lighthouse problem though)
    • Update eth1 config configuration, to work around a special bug

    Supported clients:

    • teku
    • nethermind
    • besu (implemented, needs more testing)
    • teku
    • lighthouse
    • prysm (implemented, needs more testing)
    • nimbus (highly experimental integration, disabled for now)
    opened by protolambda 0
  • Update Teku+Catalyst docker instructions

    Update Teku+Catalyst docker instructions

    Changes

    • Replaces old --rpc with new --http in Catalyst CLI
    • Adds Catalyst docker instructions
    • Updates Teku docker instructions

    Both docker containers have been tested on Linux

    opened by mkalinin 0
  • Add deposit instructions

    Add deposit instructions

    Added minimal support to test deposits. The user needs to generate more keystores so that they are already imported by their validators. Then generates the depositdata using eth2-val-tools and finally submits the raw transactions using a script similar to example_transaction.py

    opened by potuz 0
  • Lighthouse errors connecting to eth1 nodes while Catalyst or Nethermind running.

    Lighthouse errors connecting to eth1 nodes while Catalyst or Nethermind running.

    Running into issues with Lighthouse via Docker failing to connect to eth1 nodes. First tried running Catalyst, switched to Nethermind, both result in the same error (below).

    Additional context about my configuration:

    • Minimal changes from the tutorial. Adjusted genesis_validators.yml count to 75, and generated 75 validator keystores
    • teku running fine in client mode (omitted --validator-keys )
    • start_validator.sh running fine via Docker
    • catalyst client successfully binds to 127.0.0.1:8545 when running via command line (not at same time as nethermind)
    • nethermind successfully binds to 0.0.0.0:8545 when running via docker (not at same time as catalyst)

    Any ideas what may be off?

    Error logs from /lh_scripts/start_beacon.sh docker:

    rayonism: Pulling from sigp/lighthouse
    Digest: sha256:65e61d4a57a2a64989d8bc322dc363567d556eb2e97f897bf73d85cebb75a172
    Status: Image is up to date for sigp/lighthouse:rayonism
    docker.io/sigp/lighthouse:rayonism
    Apr 28 02:48:01.902 INFO Lighthouse started                      version: Lighthouse/v1.3.0-6401a63
    Apr 28 02:48:01.902 INFO Configured for network                  name: custom loaded from mynetwork/public/eth2_config.yaml
    Apr 28 02:48:01.902 INFO Data directory initialised              datadir: /mynetwork/nodes/lighthouse_docker
    Apr 28 02:48:01.902 WARN Running HTTP server on port 5052
    Apr 28 02:48:01.902 INFO Deposit contract                        address: 0x4242424242424242424242424242424242424242, deploy_block: 0
    Apr 28 02:48:01.921 INFO Starting beacon chain                   method: resume, service: beacon
    Apr 28 02:48:01.935 INFO Block production enabled                method: json rpc via http, endpoints: ["http://localhost:8545"]
    Apr 28 02:48:01.935 INFO Beacon chain initialized                head_slot: 0, head_block: 0x84c6…ce8b, head_state: 0x87c6…3907, service: beacon
    Apr 28 02:48:01.935 INFO Timer service started                   service: node_timer
    Apr 28 02:48:01.935 INFO UPnP Attempting to initialise routes    service: UPnP
    Apr 28 02:48:01.937 INFO Libp2p Service                          peer_id: 16Uiu2HAmK2cK6EZc7MpJfe3HrU3GysSeEj33rmLY8X2WWNvrwj22, service: libp2p
    Apr 28 02:48:01.937 INFO ENR Initialised                         tcp: Some(9000), udp: None, ip: None, id: 0xc14f..2597, seq: 1, enr: enr:-KO4QPeF9chLLL-3e7r4guqos77cAvUsLVfO7XT7XJm5JJOqZFQhyusZY8jJ1osXH1dmrwNzhTUmNjZ4Io0ZYPj8Le8Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB6RgHmAAAHAP__________gmlkgnY0iXNlY3AyNTZrMaEDXq_7I6Wzxi8WkK1nhD1teUrYRyZElospotMlvR2r8YODdGNwgiMo, service: libp2p
    Apr 28 02:48:01.938 INFO Listening established                   address: /ip4/0.0.0.0/tcp/9000/p2p/16Uiu2HAmK2cK6EZc7MpJfe3HrU3GysSeEj33rmLY8X2WWNvrwj22, service: libp2p
    Apr 28 02:48:01.938 INFO HTTP API started                        listen_address: 127.0.0.1:5052
    Apr 28 02:48:01.952 WARN Eth1 endpoint is not synced             action: trying fallback, last_seen_block_unix_timestamp: 1619574826, endpoint: http://localhost:8545, service: eth1_rpc
    Apr 28 02:48:01.952 CRIT Couldn't connect to any eth1 node. Please ensure that you have an eth1 http server running locally on http://localhost:8545 or specify one or more (remote) endpoints using `--eth1-endpoints <COMMA-SEPARATED-SERVER-ADDRESSES>`. Also ensure that `eth` and `net` apis are enabled on the eth1 http server, warning: BLOCK PROPOSALS WILL FAIL WITHOUT VALID, SYNCED ETH1 CONNECTION, service: eth1_rpc
    Apr 28 02:48:01.952 ERRO Failed to update eth1 cache             error: Failed to update Eth1 service: "All fallback errored: http://localhost:8545 => EndpointError(FarBehind)", retry_millis: 7000, service: eth1_rpc
    Apr 28 02:48:07.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
    Apr 28 02:48:07.001 INFO Searching for peers                     current_slot: 443, head_slot: 0, finalized_epoch: 0, finalized_root: 0x0000…0000, peers: 0, service: slot_notifier
    Apr 28 02:48:07.001 WARN Syncing eth1 block cache                est_blocks_remaining: initializing deposits, msg: sync can take longer when using remote eth1 nodes, service: slot_notifier
    Apr 28 02:48:08.949 WARN Eth1 endpoint is not synced             action: trying fallback, last_seen_block_unix_timestamp: 1619574826, endpoint: http://localhost:8545, service: eth1_rpc
    Apr 28 02:48:08.949 CRIT Couldn't connect to any eth1 node. Please ensure that you have an eth1 http server running locally on http://localhost:8545 or specify one or more (remote) endpoints using `--eth1-endpoints <COMMA-SEPARATED-SERVER-ADDRESSES>`. Also ensure that `eth` and `net` apis are enabled on the eth1 http server, warning: BLOCK PROPOSALS WILL FAIL WITHOUT VALID, SYNCED ETH1 CONNECTION, service: eth1_rpc
    Apr 28 02:48:08.949 ERRO Failed to update eth1 cache             error: Failed to update Eth1 service: "All fallback errored: http://localhost:8545 => EndpointError(FarBehind)", retry_millis: 7000, service: eth1_rpc
    Apr 28 02:48:11.981 INFO UPnP not available                      error: IO error: Resource temporarily unavailable (os error 11), service: UPnP
    
    opened by anuzis 2
Owner
Diederik Loerakker
Platform architect, specialized in Ethereum R&D. Building Eth2. Twitter: @protolambda
Diederik Loerakker
A simple Ethereum mining pool

A simple getWork pool for ethereum mining Payouts are still manual. TODO: write payouts when someone mines 10 blocks. Also, make the submit actually

null 93 Oct 5, 2022
A simple Ethereum mining pool

A simple getWork pool for ethereum mining

null 93 Oct 5, 2022
Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions.

Blockchain ETL 2.3k Jan 1, 2023
Basic Ethereum Miner Lib

EthMine ⛏ Basic Ethereum Miner Library. Developers can integrate this algorithm to mine blocks from their ethereum supported chain efficiently. Instal

Jaival Patel 1 Oct 30, 2021
Tutela: an Ethereum and Tornado Cash Anonymity Tool

Tutela: an Ethereum and Tornado Cash Anonymity Tool The repo contains open-source code for Tutela, an anonymity tool for Ethereum and Tornado Cash use

TutelaLabs 96 Dec 5, 2022
Lottery by Ethereum Blockchain

Lottery by Ethereum Blockchain Set your web3 provider url in .env PROVIDER=https://mainnet.infura.io/v3/<YOUR-INFURA-TOKEN> Create your source file .

John Torres 3 Dec 23, 2021
Electrum - Lightweight Vertcoin client

Electrum - Lightweight Vertcoin client Electrum-VTC is a rebase of upstream Electrum and pulls in updates regularly. Donate VTC to support this work:

Vertcoin 4 Oct 14, 2022
Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse.

Tink A multi-language, cross-platform library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse. Ubuntu

Google 12.9k Jan 5, 2023
😈 Shining is a tool that enables engineers to remotely pull smart contract code in multi-file situations.

?? Shining ?? Shining is a tool that enables engineers to remotely pull smart contract code in multi-file situations. Shining is the name of one of my

xxxeyJ 15 Jun 17, 2022
Web3 Ethereum DeFi toolkit for smart contracts, Uniswap and PancakeSwap trades, Ethereum JSON-RPC utilities, wallets and automated test suites.

Web3 Ethereum Defi This project contains common Ethereum smart contracts and utilities, for trading, wallets,automated test suites and backend integra

Trading Strategy 222 Jan 4, 2023
Resources for the Ki testnet challenge

Ki Testnet Challenge This repository hosts ki-testnet-challenge. A set of scripts and resources to be used for the Ki Testnet Challenge What is the te

Ki Foundation 23 Aug 8, 2022
switching computer? changing your setup? You need to automate the download of your current setup? This is the right tool for you :incoming_envelope:

?? setup_shift(SS.py) switching computer? changing your setup? You need to automate the download of your current setup? This is the right tool for you

Mohamed Elfaleh 15 Aug 26, 2022
Fabric mod where anyone can PR anything, concerning or not. I'll merge everything as soon as it works.

Guess What Will Happen In This Fabric mod where anyone can PR anything, concerning or not (Unless it's too concerning). I'll merge everything as soon

anatom 65 Dec 25, 2022
A Telegram Video Merge Bot by @AbirHasan2005

VideoMerge-Bot This is very simple Telegram Videos Merge Bot by @AbirHasan2005. Using FFmpeg for Merging Videos. Features: Merge Multiple Videos. User

Abir Hasan 57 Nov 12, 2022
Merge multiple PDF files into one.

PDF Merger Merge multiple PDF files into one. Usage % python pdf_merger.py -h usage: pdf_merger.py [-h] [-o OUTPUT] [-f [FILES ...]] optional argumen

Duo Apps 6 Oct 3, 2022
Fetch McDonald invoices from mailbox and merge them to one PDF file.

concatenate Fetch McDonald invoices from mailbox and merge them to one PDF file. Description This script will fetch all McDonald invoice pdfs from a p

null 3 Oct 6, 2022
Small python-gtk application, which helps the user to merge or split pdf documents and rotate, crop and rearrange their pages using an interactive and intuitive graphical interface

Small python-gtk application, which helps the user to merge or split pdf documents and rotate, crop and rearrange their pages using an interactive and intuitive graphical interface

null 1.8k Dec 29, 2022
A Sublime Text package that allows a user to view all the available core/plugin commands for Sublime Text and Sublime Merge, along with their documentation/source.

CommandsBrowser A Sublime Text package that allows a user to view all the available core/plugin commands for Sublime Text and Sublime Merge, along wit

Sublime Instincts 26 Nov 15, 2022
This repo is for scripts to run various clients at the merge f2f

merge-f2f This repo is for scripts to run various clients at the merge f2f. Tested with Lighthouse! Tested with Geth! General dependecies sudo apt-get

Parithosh Jayanthi 2 Apr 3, 2022
Showing potential issues with merge strategies

Showing potential issues with merge strategies Context There are two branches in this repo: main and a feature branch feat/inverting-method (not the b

Rubén 2 Dec 20, 2021