Celebrating 5 years with the Ethereum Community!

https://medium.com/ethereum-cat-herders/celebrating-5-years-with-the-ethereum-community-fd3dfd701bdd

Ethereum Cat Herders Anniversary — achievements, initiatives, and recognitions.

This marks another milestone for the Ethereum Cat Herders, a group synonymous with dedication, coordination, and unwavering support for the Ethereum ecosystem. As we celebrate the anniversary, it’s a moment to reflect on our remarkable achievements, contributions, and initiatives.

Key Achievements

Ethereum Improvement Proposals (EIPs)

We are honored to have contributed to managing and streamlining the EIP process, ensuring that proposals for network improvements and other Ethereum Standards are thoroughly discussed, and implemented efficiently before moving to `Final` status.

Since 2019, we have organized, facilitated, and documented notes for:

Hard Fork Coordination

Many of you who followed Ethereum Cat Herders’ origin story may know about our active involvement with coordinating the Ethereum network upgrades, communicating the testnet and mainnet timelines, and educating about the proposals involved. Preparing the Ethereum community for the upcoming upgrade and facilitating seamless transitions.

In 2024, we are expecting the Dencun upgrade which is currently deployed on the Goerli Test network. Ethereum Cat Herders planned the transition to test network Livestream.

  • We also added a Testnet page to share information on existing and deprecated testnets. Each of the active testnets contains details such as explorer, faucets, and the expectancy to have the testnet around.
  • The community may follow the specs of devnets for Dencun, the introduction of Holesky Testnet and the limited availability of Goerli testnets on a trusted and updated resource at the Ethereum Cat Herders website.

Community Engagement and Education

Through various initiatives, the Cat Herders have significantly boosted community engagement. We’ve provided educational resources, making complex technical topics accessible to a broader audience, and fostering a more informed and involved community. Apart from representing Ethereum Cat Herders in various, blockchain events, Pooja Ranjan participated in

She joined as a guest to the Consensys Academy course to talk about the “Understanding Ethereum Network Upgrades: Dencun”.

PEEPanEIP

We recorded over a dozen of episodes to explain proposals and devnets discoveries with the PEEPanEIP Playlist. It includes all 9 proposals, Deneb specifications, the introduction of the Holesky Public testnet & more.

Learn2Earn — Dencun NFT

We brought back the Learn2Earn app so people can learn about Ethereum Improvement Proposals to be deployed on the mainnet with the Dencun upgrade in a fun and rewarding way. If you have not heard of Learn2Earn (L2E) before, it is an educational platform that rewards users for learning about Ethereum protocols and application standards. Learn the protocol, take the quiz, and earn an NFT when you pass.

Earn Dencun NFT by answering a few questions: https://l2e.ethereumcatherders.com/

Developer Meetings and Governance

By documenting notes, and uploading Ethereum developer meetings, we’ve ensured that key development discussions are productive, inclusive, and transparent. We encourage a more democratic decision-making process within the Ethereum community. This include

Initiative

Ecosystem Project Demo

https://medium.com/media/4e1415722f73241301760a23546f04a6/href

To support application, tooling, and other projects of the Ethereum ecosystem, we launched a video series Ecosystem Project Demo. The goal of the series is to provide a showcase opportunity for upcoming and existing projects.

Recognition & Testimonials

The Ethereum community has widely recognized the Cat Herders for our unwavering commitment to transparency and accountability, evident in our regular updates and reports.

A survey report on Client Diversity on Ethereum Network and our efforts in promoting diversity and inclusion have not gone unnoticed. We support creating a more vibrant and varied Ethereum community.

On multiple occasions, prominent ecosystem developers, projects, and individuals from the community expressed appreciation for our efforts in streamlining communication and decision-making processes.

Some of the popular mentions include

Meet The Herders

https://medium.com/media/32f6d57659926fc3d5ac804576475a35/href

The Ethereum Cat Herders have been more than just facilitators. We’re proud to be one of the driving forces behind Ethereum’s continued growth. The series, Meet The Herders highlights the contribution of contributors to the open-source community.

Our work ensures that the Ethereum ecosystem remains decentralized, democratic, and forward-moving, despite the challenges of managing a diverse and global community of developers, stakeholders, and users.

Public Goods Funding Grants

We are immensely grateful to the grant organization for their generous funding towards the Ethereum Cat Herders. This support plays a pivotal role in our mission to cultivate and enhance public goods within the Ethereum ecosystem.

This funding not only bolsters our capacity to support the Ethereum network but also reinforces the value and impact of public goods in the blockchain space. We deeply appreciate their belief in our mission and their commitment to fostering innovation and collaboration in the Ethereum community.

As we celebrate this anniversary, it’s clear that the Ethereum Cat Herders’ journey is one of impactful collaboration and innovation. Their contributions have been invaluable in shaping Ethereum into a robust, dynamic, and leading blockchain platform.

Support the Ethereum Cat Herders

Here’s to many more years of our continued success and support of the Ethereum ecosystem! 🐱


Celebrating 5 years with the Ethereum Community! was originally published in Ethereum Cat Herders on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ethereum Cat Herders Update #55

https://medium.com/ethereum-cat-herders/ethereum-cat-herders-update-55-7bb3877a21b3

Dencun Upgrade — EIPs, Devnet 8 / 9, Testing Overview, Other Client Dev meeting, EIP-7495, EIP-5069, PEEPanEIP, EIPs Insight, ECH initiatives, Public Goods Funding and Other Community updates!

It’s been a while since we last connected through the Ethereum Cat Herders Newsletter, however, our commitment to delivering valuable insights, exciting updates, and engaging content has never wavered. We’re thrilled to be back in your inbox!

Let’s take a quick read into the Ethereum Protocol & Research update, EIPs, Cat Herder’s initiative, and what happened in the Ethereum ecosystem since the last update.

Summary

  • Dencun Upgrade — Devnet 9 to be launched this week, Dencun Testing Overview, PEEPanEIP
  • Other Client Dev Meetings — ACDC, ACDE, Execution layer P2P, EOF Implementers, Verkle Implementers
  • EIPs updates — EIP-7495, EIP-7329, EIP-5069, EIP Editing Office Hour, PEEPanEIP, EIPs Insight
  • ECH initiatives — Ecosystem Project Demo, Meet The Herders
  • Public Goods Funding — ENS, Octant, Gitcoin Grant
  • Other Community updates.

Dencun Upgrade

Deneb on the Consensus Layer and Cancun on the Execution Layer, together known as Dencun, is the next planned upgrade on the Ethereum blockchain. EIPs are selected, clients have implemented, and are currently on the Devnet 8 (Developers testnet).

Dencun Devnet

Devnet 1–7 were more focused on EIP-4844. However, Devnet-8 was supposed to be a feature-complete developers’ testnet. All clients except Reth are participating in devnet.

As per devnet 8 specs, Pull Requests included are

In the last ACDC call, the protocol testing team, Parithosh, Barnabus, and Mario indicated the planned launch of Devnet 9, tomorrow, 12th September 2023.

Devnet explorers available are:

Dencun Testing Overview

https://medium.com/media/e2aeb71d5d774189f58a35c4128d84b9/href

To understand the various testing approaches and bases they are covering, we highly recommend watching the PEEPanEIP- Dencun Testing with Pari, Mario, Barnabus

In addition to that, the team released a Dencun testing overview, a document created to help comprehend the various parts that will be changed and the corresponding tests required across the EL/CL.

  • Sync testing is being run by Nethermind
  • Chaos testing is WIP will be completed by the end of this month.
  • MEV-related test — Issue with Capella genesis. As of yesterday, it has been fixed. CL side tested mock MEV workflow.

If interested in learning more, please follow Dencun Interop Testing Call 30 here.

Other Client Dev meetings

EIPs updates

EIP-7495: SSZ StableContainer

Having a unified structure for data storage and transmitting over the EL & CL is something that Ethereum devs are striving for. In early January, Etan Kissling, a developer at Nimbus proposed a solution to use SSZ on both layers. His latest proposal EIP-7495 will introduce a new Simple Serialize (SSZ) type to represent StableContainer[N] values.

Though it is late to include any EIP for Deneb, this proposal was discussed at length for inclusion at the last ACDC meeting. A decision may be expected on the ACDC call on 14th September.

EIP Governance

EIP-7329: ERC/EIP Repository split

After years-long discussion, the wait for the EIP-ERC repo split is almost over. Following past EIPIP meetings, you may know that EIP editors came to a rough consensus of splitting the repo.

As per the decision, a Meta EIP-7329: ERC/EIP Repository split has been documented by Matt Garnett and Danno Ferrin. Currently in the Last Call , the EIP describes the motivation and rationale for splitting the EIP repositories into an EIP repository, targeting core Ethereum changes and an ERC repository, targeting application layer specifications.

EIP-5069: EIP Editor Handbook

Sam Wilson, one of the EIP editors, proposed the Ethereum Improvement Proposal Charter to outline the governance and the role of EIP Editors in the process. A big portion of the proposal is now added to EIP-5069: EIP Editor Handbook, another Meta EIP addition to the EIP repository.

PEEPanEIP

In the past few months, we reached out to multiple Ethereum developers and EIP authors to create resources for the Dencun upgrade. We’re happy to share the PEEPanEIP Playlist and EIPs that are selected for the upgrade. For the first time, the Ethereum upgrade includes Consensus Layer EIPs.

EIP Editing Office Hour

https://medium.com/media/87a240e03a7391636a34eb5c6d32a309/href

If you’re an author willing to document your EIP and need help, the EIP Editing Office hour is the meeting that you’d like to attend. EIP editor Sam Wilson takes time to review the proposals and respond to authors’ queries to help move the proposal towards Final status.

Should you have a pull request to discuss with the EIP Editor, drop the link to the Agenda and join the next office hour. The join link will be available on EthCatHerders Discord.

EIPs Insight

EIPs Insight (September 2023) – HackMD

In Sep 2023, EIPs repository received 3 Final, 8 Draft proposals, 5 in Last Call 4 in Review and 1 Stagnant EIPs. Check the full report here.

ECH initiatives

Ecosystem Project Demo

https://medium.com/media/4e1415722f73241301760a23546f04a6/href

Extending our support to Ethereum staking and application devs, we are inviting ecosystem projects to demo for the Ethereum Community. We begin the series with a project by Staders Labs to provide an opportunity for the Ethereum community to be able to join ETHStaking with less than 32 ETH. Check out the conversation with Anoothi Kumar, Stader Labs: Ecosystem Project Demo #1

Meet The Herders

Ever wondered what goes into coordinating upgrades to Ethereum; a multibillion-dollar decentralized network?

In this new interview series, we’ll meet the Ethereum Cat Hearders who behind the scenes orchestrate this remarkable feat; by aligning multiple teams, documenting every step, and ensuring the community remains well-informed. In this series, we hear their stories, experiences & insights into the Ethereum community. Stay tuned for the release soon!

Public Goods Funding

We can not thank enough the Ethereum ecosystem to keep us afloat with Public Goods Funding. In Q2-Q3, we received grants from different organizations.

ENS

Public Goods Large Grants

The ENS Large Grants process has been experimental to create a pathway to fund Public Goods demonstrating exceptional usefulness and impact in the Ethereum ecosystem.

In Q2, four recipients were chosen to receive the 50k USDC grant. All of these projects have provided exceptional usefulness and impact on the Ethereum or Web3 ecosystems. Excited to share that Ethereum Cat Herders was one of the four recipients.

Octant Epoch Zero

Regens have spoken! Epoch Zero results are in

Back in June, the project Octant announced Epoch Zero: a special pre-launch event for experiments in decentralized governance which is funded and developed by the Golem Foundation.

Based on a community poll, the results of Epoch Zero were announced. Together, the project distributed $1M worth of ETH among 10 high-impact web3 public goods projects with an excellent track record including Ethereum Cat Herders as one of the grant recipients.

Gitcoin Grant 18

Gitcoin | Explorer

Ethereum Cat Herders also participated in Round 18 and received community support.

As we’re close to the end of Q3, we want to express our heartfelt gratitude for your continued support. Your trust in us is what keeps Ethereum Cat Herders alive and thriving. Whether you’ve been with us from the beginning or are just joining our community, we want you to know that you’re a vital part of what we do. 🙏

Other Community updates

Ethereum Execution Layer Specification

After more than a year in development, the Ethereum Execution Layer Specification (affectionately known as EELS) is announced. EELS is a Python reference implementation of the core components of an Ethereum execution client focused on readability and clarity.

Intended as a spiritual successor to the Yellow Paper that’s more programmer-friendly and up-to-date with post-merge forks, EELS can fill and execute state tests, follow mainnet1, and is a great place to prototype new EIPs.

KZG Ceremony Special Contributions

The Special Contribution Period for the KZG Ceremony allowed participants to contribute in ways that may not have been possible in the Open Contribution period. While the Ceremony only needs a single honest participant to provide a secure output, Special Contributions provide additional assurances beyond a standard entropy contribution.

Share your questions, comments, and suggestions with me on Twitter. Thank you for reading 🙏.

— The Ethereum Cat Herders


Ethereum Cat Herders Update #55 was originally published in Ethereum Cat Herders on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to the Ethereum Shapella Upgrade!

https://medium.com/ethereum-cat-herders/welcome-to-the-ethereum-shapella-upgrade-3c6f90d5fcf

EIPs & CL changes, PEEPanEIP, and a Watch party invite.

Ethereum blockchain is expecting Shapella Upgrade on April 12, 2023. It will be will activate on the Ethereum network at epoch 194048, scheduled for 22:27:35 UTC as announced in the mainnet announcement blog post by Ethereum Foundation.

“Shapella” is a derived upgrade name (Shanghai + Capella) to suggest two layers of the blockchain are being upgraded simultaneously.

Shapella is the first network upgrade since TheMerge. It will deploy changes to the Execution Layer with the Shanghai upgrade; whereas the Capella upgrade will bring some spec changes to the Consensus Layer.

Activating staked ETH Withdrawal

With the concept of network upgrade centered around a particular proposal, EIP-4895: Beacon chain push withdrawals as operations is the “anchor EIP” of the Shapella upgrade. This upgrade will deploy EIP-4895 on the Execution clients to enable the Withdrawal of staked ETH for Ethereum validators along with three other EIPs deployed on the mainnet.

Since the launch of the Beacon chain in Dec 2020, Ethereum staking is unidirectional. Users may stake ETH to become a validator, perform duties and earn rewards in their wallet. This upgrade will allow them to withdraw their ETH partially (rewards only) or fully (staked ETH+rewards).

This is the first opportunity to enable staked ETH withdrawal along with EVM changes and preparation of future protocol changes since the merge of the beacon chain (PoS) and Ethereum PoW chain in September 2022.

Shanghai upgrade will deploy 3 more Core EIPs on the Execution Layer.

EIP-3651: Warm COINBASE— Direct COINBASE payments are becoming increasingly popular but accessing COINBASE is overpriced because the address is initially cold under the access list framework introduced in EIP-2929 (Berlin upgrade). This proposal will warm the COINBASE address at the start of transaction execution making it cheaper for users.

EIP-3855: PUSH0 instruction— Many instructions expect offsets as inputs, which in a number of cases are zero. This proposal will introduce the PUSH0 (0x5f) instruction, which pushes the constant value 0 onto the stack. This is a useful feature upgrade for clients & contract developers.

EIP-3860: Limit and meter initcode — This is an extension to EIP-170, to limit the maximum size of initcode and ensure initcode is fairly charged (most importantly cost is proportional to initcode’s length). It will simplify EVM engines by the explicit limits and have a cost system that is extendable in the future.

Readers should be aware that the Shanghai upgrade on Execution Layer is accompanied by the Capella upgrade on Consensus Layer which will be upgraded at the same time.

Capella will introduce the following changes to Consensus clients:

  • Full and partial withdrawals for validators
  • BLSToExecutionChange messages, which allow validators using a BLS_WITHDRAWAL_PREFIX to update it to an ETH1_ADDRESS_WITHDRAWAL_PREFIX, a prerequisite for withdrawals
  • Independent state and block historical accumulators, replacing the original singular historical roots

It is important to note that EIP-4895 introduces a system-level “operation” to support validator withdrawals that are “pushed” from the beacon chain to the EVM. Enabling withdrawal is only possible with the specs change at Consneus Layer.

The Shapella Upgrade specs also include a Meta EIP-6049: Deprecate SELFDESTRUCT.

EIP-6049 is only a deprecation warning. Client teams expect SELFDESTRUCT semantics to change in future network upgrades, but the opcode’s behavior remains unchanged in Shanghai. EIPs for the spec change at the protocol level are still under discussion.

PEEPanEIP

To provide an overview of the proposals involved, testing, and client preparation in the Shapella upgrade, Ethereum Cat Herders recorded multiple talks with authors and client developers.

https://medium.com/media/006bf95b8ebdbe71ffa8f89255e933d8/href

The playlist includes talks by authors and implementers to explain why they are included in the Shapella Network upgrade. Other featured talks are:

PEEPanEIP is an hour-long deep dive into Ethereum Improvement Proposals and upcoming changes. Highly recommend watching for a better understanding of the concept shared by the authors of the proposal. Also, available on Ethereum Cat Herders Podcast.

Watch Party

Shapella Watch Party

You’re invited to the Shapella Livestream party!

EthStaker & Ethereum Cat Herders are organizing an upgrade watch party. We hope to be joined by you and

  • Protocol Researcher — Vitalik Buterin
  • ACDE/ACDC coordinators — Tim Beiko & Danny Ryan
  • EIP Authors — Alex Stokes, Alex Beregszaszi, Paweł Bylica, William Morriss.
  • Testing team rep — Parithosh Jayanthi & Mario Havel
  • Ethereum client teams
  • EthStaker crew
  • ECH community

We welcome the Ethereum community to the complete Proof of Stake Ethereum blockchain world. Stay tuned for upcoming upgrades & changes in the Ethereum ecosystem with Ethereum Cat Herders!

Support the Ethereum Cat Herders

Share your question, comments, and suggestions with me on Twitter. Thank you for reading 🙏.

— The Ethereum Cat Herders

PS: Original image by Unsplash.


Welcome to the Ethereum Shapella Upgrade! was originally published in Ethereum Cat Herders on Medium, where people are continuing the conversation by highlighting and responding to this story.

Exploring Ethereum’s Client Ecosystem

https://medium.com/@poojaranjan19/afc9affa84dd

An analysis of Client Diversity based on a survey conducted in Q1 2023

We thank Tim Beiko for reviewing and providing feedback on this report and the ETHStakers community for encouraging validators to respond to the survey.

Key Findings

  • The primary reason for node running is identified as ETH staking. However, a small number of users also run Ethereum nodes for data accessibility, and privacy, to become ecosystem contributors, and less than 1 % run Ethereum nodes for metrics collection.
  • Over 95% of the respondents of the survey are running nodes for themselves.
  • Geth is indeed the favorite client for the Execution chain followed by Nethermind, Teku & Erigon.
  • Nethermind and Erigon turned out to be the clients of choice for switching if needed. Many users have shared not-so-good experiences with Besu since after the Merge. Interestingly, Reth is also being considered.
  • Lighthouse is the #1 choice of respondents for the Consensus client, followed by Prysm, Teku, Nimbus & Lodestar.
  • Teku & Nimbus are the preferred choices in case of a switch from the existing node-running clients. A few users expressed interest in Grandine.
  • Geth-Lighthouse (21%) is the most popular client combination. Followed by Nethermind-Lighthouse, Besu-Lighthouse & Geth-Prysm at ~12% each.
  • Availability of documentation along with the client/dev support is the main reason for the selection of clients to run EL/CL nodes.
  • About 80% expressed concern with the present client diversity and ~25% of respondents listed “Client Diversity” as the main reason for the switch to another client for EL/CL node running.
  • To increase client diversity, suggestions include standardization, campaigns, rewards & penalties.

Please note statistics shared in this report are based on the responses collected from the survey conducted by Ethereum Cat Herders in Q1–2023. Due to the limited survey size (<1% validators), the client diversity may NOT be an actual reflection of the present network share. However, users’ responses about their preferences, critical feedback, and improvement suggestions are valuable data points for the enhancement of existing Ethereum Execution & Consensus clients and eventually lead to an ideal distribution over the network.

Our sincere thanks to all users who responded to the survey. 🙏

Background

The Execution & Consensus Layer of Ethereum has multiple client implementations developed by different teams and written in different programming languages. Having multiple client implementations allow a healthy level of competition and encourages innovation among client developers.

The distribution of clients across the network isn’t ideal, especially with most nodes running the Geth client for the Execution Layer. Around 80% of survey respondents are concerned about the state of client diversity in the Execution Layer.

Client diversity reduces the risk of a single point of failure, making the network more resilient and less vulnerable to attacks.

We received responses from over 40 countries with the majority coming from the United States, France, Germany, and other European countries. We also received responses from users in South Korea, Japan, and Indonesia.

95% of respondents identified themselves as stakers, with the rest being non-stakers.

Ethereum node(s) running

Most of the respondents reported node running for personal use, while the remaining 7.5% of users running nodes for different organizations such as Aestus, Relay, Avado, Chainlink, Circle Data Science, Cryptocompare, Jump Crypto, Keynodes, Lido, Minipool Node, Rocket pool, Stafi, Stakewise, Swell, The Cryptocrats Partnership or personal projects.

Chart: Reasons for running Ethereum node(s)

As part of the survey, we asked users to select their primary reason or use for running Ethereum node(s) from four available multi-select options. The results showed that approximately 95% of users were running their nodes for staking purposes, while 46–53% were focused on Decentralization/ Privacy and contributing to the Ethereum ecosystem.

Client Diversity

Source: https://clientdiversity.org/

According to the Client Diversity website charts, Geth is the majority client on Execution Layer and Prysm is the leader on the board with 58% and 39.8% network share respectively.

The stats. collected from our survey may not be 100% consistent with the Client Diversity website chart, but provides a pretty much similar idea of the uneven distribution of clients that we have on the Mainnet today.

Chart: EL/CL clients on the network as reported

Survey data suggests that with a little over 50% of the network share, Go-Ethereum (Geth) is indeed the most popular client for the Execution chain followed by Nethermind (30.8%), Besu (31.4%) & Erigon (9.4%).

On the Consensus chain, Lighthouse (54%) is the #1 choice of respondents for node running followed by Prysm (23.3%), Teku (23.3%), Nimbus (15.7%) & Lodestar (1.9%).

As a reminder, the above chart (EL/CL clients on the network) is based on limited data collected from the survey with a sample size of less than 1% of node operators on the Mainnet.

Number of client nodes and EL-CL combinations on the Mainnet

In addition, our survey revealed that the majority of users are currently running a single client node, and only 3–5% of respondents are running more than 5 Execution and/or Consensus nodes.

Approximately 74% of respondents are currently running a single client combination, 23% running 2–5 client combinations, and only 4% are running more than 5 client combinations.

How important is it for you to run a minority client on the network? Why?

Over 75% of respondents mentioned running a minority client on the network is important for various reasons:

  • It minimizes the severity of the slashing risk due to bugs in the majority of clients.
  • It’s risk containment to avoid getting marooned if there’s a consensus-impacting bug.
  • Support security and avoid penalties for a majority bug.
  • The liveliness of the network in case of bugs is at stake.
  • Don’t attest to a wrongly finalized chain.
  • To promote decentralization/resilience, diversity, and redundancy. Keeping things honest, trust yet verify!
  • Important for the health of the network, and to improve the ecosystem.
  • diversity → protect my investment
  • Makes me feel good that I’m contributing to diversity.

However, some users are hesitant to switch to a minority client due to the additional workload that comes with it. They prefer to stick with a working system, and a minority client may come with more issues and have fewer developers to support it. They would be more willing to run another client if there were well-written documentation available. Some users have considered running a minority client, but are hesitant due to potential problems and are waiting for better tooling to make the process easier.

While diversity is important, some users prioritize keeping their existing setup running smoothly as there is a lot of value at stake for them. They want to use the most mature, best-working, and most stable client available.

Highlighted users comments

– It is important to foster decentralization.

– It’s definitely important. Lighthouse was a minority client, when I started to use it. However, as large staking operators switched from Prysm to Lighthouse, I’m now running a majority client. I’m very much of the opinion, that large staking providers are the ones to hold accountable and urge them to switch as opposed to solo stakers. It’s still quite a hassle to setup a new staking rig and get familiar with a different client. Professional operators are better quipped to handle this.

– Running a minority client is important for chain stability: if the major client fails due to a bug, a minor client written in a different code base can keep the chain going. Ultimately, all clients should have similar shares. I would have preferred to run the failover node as a combination of minority clients, but some CL clients still have compatibility issues with failover support.

– Important, but currently all other execution clients are RAM resource intensive.

– If Erigon were the majority client, I’d still be running it since I need the smaller archive node size.

– Important, if my system would support them.

– Very important — the usual reasons. however, if the (somer) docs weren’t available, i’d have gone the popular route without a thought.

What impacts the selection of clients?

To better understand users’ thought processes, we asked users about the factors that influenced their decision to run a specific client. The majority of respondents cited Client team & dev support to be the most influencing factor followed by the availability of documentation, Specific feature, and majority bias. According to the responses, these factors play a vital role in the selection of clients, especially for new users.

Interestingly, many of the respondents reported having previously used Geth and Prysm clients but had since switched due to concerns about their super-majority status and the increasing diversity of available clients.

Switch clients?

As part of our survey, a few questions were added to collect the willingness to run a different client node and understand the challenges if any.

What would encourage you to run a different client than the one you are running currently — the missing element?

Based on the responses from our survey, it is clear that users value a client that uses fewer system resources and has good documentation, reliability, and fast sync. They are also interested in a minority client that has distinguishing features, such as lower storage/memory requirements, but want to understand any tradeoffs that may exist. Switching costs are high, so users would prefer a client that offers improved performance, stability, disk usage, or monitoring over their current client.

However, users also mentioned that the sync time for switching to a new execution client is a big hurdle, which can lead to procrastination. Therefore, a well-documented and fully functioning migration guide would be useful to make the process easier. Additionally, users would like to see incentives for running minority clients, even if they are small. Some users have tried running a minority client but felt like they were beta testers, which wasn’t worth the headache without incentives.

In case of switching to a different client, Nethermind and Erigon turned out to be the Execution clients of choice, if needed. Interestingly, Reth is also being considered.

Chart: Prefered clients to switch today

Nimbus & Teku are preferred choices in case of switching from the existing node running Consensus clients. Despite being a closed source at the time, a few users expressed interest in Grandine.

Chart: Present EL/CL Client combinations on the network

What features (that currently don’t exist) would you like to see an EL/CL client implement?

According to our survey respondents, they would like to see better incentives for individual node operators, such as proof of solo authenticity, to encourage more participation in the network. Some users suggested that they feel the separation between EL and CL is unnecessary and that the clients should handle both layers. They also suggested that combo EL/CL software packages would provide a vastly better user experience and could help more inexperienced users become home stakers.

Other features that users would like to see include pruning (although it already exists), simpler or inbuilt monitoring, Beaconchain-like features for earnings, tax forms for staking income, automatic client switching, data management and clean-up, improved and informative logging on errors and sync status, MEV smarts and other profit maximization, alerting capabilities, and reduced hardware requirements.

Some users suggested that they would like to see a unified EL/CL application that handles both functions in one application, as the current format can be clunky. Faster sync times on the EL side would also be welcomed by users, as switching clients without redundancy or failover configurations can result in a downtime of 1–2 days.

Lastly, some users would like to see a proper user interface for validator management and reporting, as they feel it is not enough to rely solely on Beaconcha.in. Additionally, users would like better access to and display of internal stats.

EL-CL Client packages & testing

  • Feel the EL/CL separation is unnecessary. The clients should handle both layers.
  • A ‘plug and play’ software package for EL/CL combinations. One application that handles both functions. [Multiple mentions]
  • Automatic client switching.
  • A temporary failover option for CL when a given EL is offline.
  • How about the client host a web server at some port that I can just go to and find out information about its state — CPU, memory, and other statistics? I have to do the whole Prometheus, graphana garbage. and the stats don’t even mean anything to me.
  • One-click to run the whole darn stack.
  • Differential fuzzing between client pairs.
  • Proof of Keys test (dry run to withdraw for example). On-chain encrypted emergency backup of keys. Graffiti plugins
  • Single binary

UI, monitoring, and other features

  • Simpler or inbuilt monitoring.
  • More graphs and interfaces, an easy dashboard
  • Graphical interfaces.
  • Improved and informative logging on errors and sync status
  • Alerting capabilities
  • Auto update on startup
  • vastly better UX
  • CL Team provides a nice Grafana dashboard
  • 1-word command updates, more ease of use.
  • Not interested in features. They should be as resource efficient as possible, maintaining themself configurable in a closed loop,
  • Monitor SSD space: currently LH only sends data about local disk, and not data-dir.
  • A proper UI for validator management and reporting. Not enough to rely on Beaconcha.in all the time.
  • Better internal stats access and display
  • it would be nice to be able to run on non-super-enterprise-grade hardware & NVMEs
  • Native Web UIs
  • Better logging, stats, and other on-chain analysis
  • Browser-based UI and toolset
  • Would be good for clients to report their vendors to validators, this way everybody would know the exact number and proportion of different clients.
  • Better troubleshooting

MEV boost

  • Built-in MEV boost, MEV smarts and other profit maximization.
  • The clients should consider local mempool before accepting bids from relays. It might rarely trigger any change, but just as a matter of principle, the relays (especially if they’re censoring) should only be able to buy the blockspace if they outbid the mempool.
  • There should also be a feature that says relays have to outbid mempool by at least x ETH before the client will accept it.

Database

  • online state pruning; better database compression. pruning of ancient blocks
  • store chain data in a SQL database for easy querying. No, graphql is not good enough
  • Some kind of internal checkpointing, e.g. on a weekly basis, would let you roll back from database corruption without a complete resync!
  • Data management and clean up
  • I would like the duties to be sent to different CLs depending on whether they are Att, Blocks, or Sync committees. Also, we could go back to the N:N model between CL/EC. that would help a lot in redundant systems.

Execution clients specific

  • Faster sync times on the EL side would be great (i.e. dropping the load of storing/validating historic blocks/states). If I had to switch clients without redundancy or failover configurations, there’s going to be a downtime of ~1–2 days which is not the end of the world, but still suboptimal and thus a barrier to change (at least for the EL).
  • Faster sync times (something like checkpoint sync we have in the CL). Better reliability against sudden crashes (reduce risk of DB corruption). Reduced DB size.
  • Multi-EL clients for failover
  • Self-prune in other EL clients, similar to Bonsai Trees in Besu would be great.
  • An easy health check
  • Execution client on HDD
  • Getting added to Linux kernel. DB corruption protection.
  • Reduced hardware requirements. Lower memory / CPU / disk usage.

Reason for switching to another client

While self-pruning was greatly appreciated, Besu users reported significant problems after the merge and were unable to get it to run reliably, losing more than 50% of attestations over several days of use. They also noted poor performance and stability issues. Erigon users reported difficulties with continuous updates. As for execution clients, most users switched to Nethermind due to its better stability and lower resource consumption.

Other highlighted users’ comments for the reason to switch to another client are:

– I choose based off of ease of use and TB footprint for an archive node.

– Switched from Geth to Besu for client diversity. Switched back to Geth after the merge for stability.

– Switched from Geth to Erigon/Besu to reduce supermajority risk.

– Earlier Geth, switched to Erigon because it’s much faster, uses less space and Geth has a super majority.

– Originally ran Geth and Prysm. Switched to Lighthouse for diversity and everything worked fine. Tried switch to besu for diversity but was unable to get it to start syncing and felt it lacked documentation. Then tried Nethermind and while I was able to get it to sync, I had frequent missed attestations (about 4 per validator per day) and low 90% effectiveness according to beaconcha.in. I upgraded my equipment (from 8GB of RAM to 32GB RAM and from a 2 TB sata SSD to a 4 TB nvme SSD). I still had the same issues. I switched back to Geth and had no issues with very, very rare attestation misses and 99% effectiveness.

– Main machine runs Geth/Prysm. I tried with Besu/Prysm. Switched back to Geth due to missed attestations daily with Besu. Besu continues running on my back-up machine. Another back up machine runs Geth/Lighhouse. Reason for 1 Main + 2 Backups is to have maximum client diversity.

– Switched from Geth to Besu to improve client diversity. Switched from Besu to Nethermind for better stability, lower resource consumption (at the expense of manual prunning).

– Used Besu but post-merge it missed several attestations a day and produced blocks with zero transactions included, so switched back to Geth.

– I used besu but after the merge it suffered significant issues and I could not get it to run reliably again (losing >50% attestations over several days of use)

– Initially Lighthouse and Geth. Switched to Nimbus and Besu because of client diversity, but later ditched Besu for Nethermind after numerous issues during the Merge.

– Earlier used Geth and Prysm, switched Besu-Lighthouse when autopruning was added. Reason for switching was to enhance client diversity

– Earlier used Prysm. I switched because the load on the system was too high.

– Switched from Geth/Lighthouse to Besu/Teku before the merge to avoid using majority execution client after merge. Also Lighthouse was in some metricas over 33% at that time.
Stand-by system is geth/nimbus on a less powerful machine also used for gathering some statistics of the network/daily balances.

Client interaction frequency

We asked users how often they interact with their EL/CL clients. The results show that approximately 42% of respondents interact with their clients every week, while 40% interact during network upgrades. Around 13% of users interact with their clients daily, and a small percentage, ~3.8%, interact with their clients multiple times per day.

What do users think of their clients?

With unique features and h/w — s/w requirements, Ethereum node runners have multiple options available to select the client. From the survey, below is the list of merits and improvements requested by node runners. Here are some key highlights of what users think of each client.

Execution clients

Besu

Validators and node operators currently using Besu, appreciated the client for

  • auto pruning,
  • stability,
  • ease of installation,
  • documentation,
  • a trusted & active dev community,
  • a responsive team,
  • regular updates,
  • professional support, and
  • having a minority client implementation with no common code with Geth.

However, users [20+ mentions] reported stability issues, slow syncing, missed/wrong attestation, low performance, and the need for better documentation.

Reliability was poor following the merge, but much better now.

Technical issues surrounding the merge were not well communicated.

Documentation could be improved and performance tuning tips can be explained more clearly and frequently.

Makes me nervous during hard forks (besu)

Besu has come down like 4–5 times post merge that’s required a re-sync

Took forever to stabilize and stop missing attestations.

Slow syncing, missing attestations several times per day

Bugs. Java.

Besu’s memory usage and leaks.

Frequent RocksDB corruption issues with Besu

The performance of besu is lower than of Geth, especially for large blocks. Leading to a few wrong attestations because blocks were not validated in time.

I still miss one attestation every few days, and missed about 10% sync committee participation

Besu — some releases are unstable. Long time to fix issues.

I wish there was an in-line command interface similar to what Geth has.

Besu has a lot of problems, currently with open issues and a month of downtime.

It was fairly difficult (Compared to Geth) to get it configured and running. There was lots of fiddling around, Googling error messages and asking devs about things on Discord before it finally worked.

Erigon

Erigon is highly appreciated for

  • Less TB for archive, light archive node,
  • archive node features for data analysis,
  • fast sync, good custom APIs for querying blockchain data small archive node.
  • Easy to monitor fast restart.

However, a few users [5+ mentions] reported that Erigon

a lot of RAM usage

requires large disk space,

too many changes and resyncs every few months.

Sometimes the database gets corrupted and I have to resync fully which takes days.

Erigon is stable up to version 2.34.0, at 2.36. it is stable again but takes a lot of disk space.

Geth

Geth is the #1 choice of node operators. The main reasons as highlighted are

  • Easy to use, good documentation, very effective, de facto standard.
  • Excellent, trustworthy dev team with a long proven record;
  • Preference for Rust/Go
  • Better documentation & guides in comparison to other EL clients
  • Geth is stable and takes a low-space disk.
  • Reliability [10+ mentions] and low hardware requirements
  • Really good Prometheus metrics
  • Rare bugs & Issues
  • Stable Geth-Prysm & Geth-Lighthouse Client Combination.
  • Stable, quiet, peace-of-mind operations; hardened, secure software; low frequency of updates, yet critical patches will be fixed immediately.
  • Performance, and support of some Geth-specific RPC

But, at the same time, users are also worried about the issue of being a majority client and slashing/penalties risk associated with the bugs in the client. Other reported concerns are:

No online or auto-pruning [5+ mentions]

Geth sometimes gets random and inexplicable database corruptions that require a full resync

Geth corrupts easily with shutdowns, and restarts take a long time to synchronize.

High storage requirement.

Geth: disk footage. 2TB Nvme ssd is quite expensive and 50% of it is used from day 1 for the node.

Having to migrate to a 4TB ssd within the next 2 years would be unfortunate.

No GUIGeth: Too many nodes running, but it’s still the most viable option.

Very IOPS dependent trying to prune geth the first time messed up the whole db. had to resync from start

Geth is very sensitive to unsafe shutdown [5 mentions] and takes a long time to heal and takes hours slow to add features.

Nethermind

Multiple users shared that Nethermind is great for simplicity and ease of use as compared to other EL clients.

  • Nethermind is easy to install,
  • has regular updates
  • is compatible with most of the Geth scripts and methods.
  • auto-pruning
  • easier to use fast sync,
  • good custom APIs for querying blockchain data
  • Nethermind is quick and has great support.
  • Developers’ timely response to bugs & problems.

Nethermind is the most popular client after Geth on Execution Layer. However, there is some room for improvement. Multiple mentions to improve memory usage and pruning issues. Related feedbacks are:

Not sure how to prune it [7 mentions], it begins to be quite big.

Nethermind uses too much memory [5 mentions], don’t like the .net ecosystem either, but it works.

RAM consumption and heavy CPU usage on initial sync, but I put 64GB so its not a problem. [3 Mentions]

The lack of documentation in the docsNethermind has somewhat confusing documentation. [3 mentions]

Logs are hard to read, it’s not colored.

The community is not as big as the one from the other clients, it’s sometimes hard to get support

Prune time

Online pruning does not seem to work well for Nethermind. Considering switching to Besu (and Bonsai), but have seen many complaints about bugs in Besu, so will wait for now.

Consensus Clients

Lighthouse

Stability, reliability, and dev team [20+ mentions] are highly appreciated by users.

  • Lighthouse is rock solid. They make me feel fuzzy when they fuzz their code Stability, ease of installation, and documentation are rock solid, they just work.
  • Simple and generally stable and performant.
  • Lighthouse is developed in Rust
  • not a majority client, popularity is growing, need to switch again
  • Stable, quiet, peace-of-mind operations; hardened, secure software; low frequency of updates, yet critical patches will be fixed immediately.
  • The Lighthouse team is awesome, every time I have a question, there is someone to answer me
  • Rust. I trust the sigma prime team, I have high confidence in the product, and it’s the client I am most able to operate.
  • Easy to maintain, update notifications/explanations on Discord (they only tag @everyone when there’s a new update
  • Has a good reputation in the community, not the majority client (if Lighthouse became the majority client, I would switch)

Though some users have

future majority concerns

The name lighthouse is not unique so searches get polluted with irrelevant results.

Not being able to download the latest, but having to put a specific file

Using DockerConfigurations is sometimes buried in discord,

no easy way to upgrade versions.

No GUIvery IOPS dependent

I don’t need a GUI/TUI, but I think it’s a valuable feature that ought to be provided by all consensus clients.

All of the clients are still out of reach for novice users.

Graphical solutions like dappnode ought to be developed upstream and made accessible to every client.

One sudden crash of the database led to missing proposal.

Nimbus

Known for light node, light on resources.

  • High reliability
  • Great documentation and rock solid
  • Nimbus is lovely, and
  • also has low resource usage, disk usage

Some users are

concerned about state issues

Logs unclear. I miss a few attestations every day, but I don’t understand why

releases and release notes seem random,

sometimes breaking features get added randomly with no mention in the release notes.

Prysm

Reported to be

  • Stable,
  • plenty of documentation,
  • ease of upgrade
  • been rock solid, they just work
  • key manager API
  • auto-update when launching
  • Prysm had a great product early in the Eth2 development cycle and had better support then. Now I appreciate the metrics they have.

Apart from being one of the leading clients, Prysm received very few concerns as

it may not be as efficient with disk space or other resources.

That I had to learn Linux all over again after leaving it for about 15 years

Teku

Many users switched to Teku for client diversity. It being a minority client was the key motivator along with others like

  • Teku and Besu work very well together,
  • Regular updates
  • professional support

Reportedly, Teku

is not implementing state/validator_balance for arbitrary epochs for computing daily profits (lighthouse and nimbus do).

A lot of RAM usage

no useful entries in logs / seems that Teku is overwhelmed in busy network situations.

Intermittent high CPU usage

What can be done to increase client diversity?

Most of the users are on board with the importance of multiple client implementations. We received multiple suggestions to increase client diversity. Good support and client documentation are important. The community groups like EthStaker as other avenues to get information are also helpful.

Easy switching

Make switch ezpz for noobz

Easy switching, and easier tooling for setting up and migrating existing EL/CL pairs that bake in diversity could be helpful. According to a user,

Honestly, switching clients is already too easy for this to be taking so long.

However, some suggest that having specific switching guides, for combinations of different setups eg Dappnode, Rocketpool, etc. will be nice to have. Node operators want to be assured that it’s not going to break if left alone for a few weeks.

  • More 1-click install staking solutions, with enough credibility to encourage institutional adoption.
  • Improve docs, provide stats for attestation and proposal performance, and show stats for CPU, disk, memory, and bandwidth usage.
  • Execution clients catch up to Geth in effectiveness.
  • Improve the robustness of minor ECs and add some graphical interfaces.

Standardization

Each client is running their own chain database (RocksDB, LevelDB etc.), and there is no interoperability between them. I.e. Besu cannot use the same DB as Nethermind. With a standard database schema shared across multiple clients, one could have the clients switch automatically, continuing recording the blocks, based on the current diversity spread. Although this is unlikely in practice.

  • Make swapping clients easier through more standardization. Likely not possible, but if installing, configuring, and running clients were more similar, people might be willing to trade out one client for another.
  • Standardize confit/command line values
  • A standard database format or a DB conversion tool?
  • Some features/tools that reduce or eliminate resyncing would encourage change.

Rewards & Penalities

According to a few respondents incentive matters. Having economic incentives for running minority clients may provide better encouragement. The suggestion is to change the way validators are selected for execution/consensus rewards, so users make more if they run a minority client and that’s compensation for taking a risk. Minority clients should get higher staking rewards. A few others suggested penalties for running staking nodes with a client with > 50 % market share (e.g. Geth).

A few users think that having more POAPs/NFTs rewards, and giving badges may help promote client diversity.

Campaigns

Some sort of campaign, and good people like Superphiz 🙂

Conclusion

Overall, these survey results provide valuable insights into the motivations and demographics of Ethereum node runners. Having a better distribution of different clients with no single client dominating over 50% of the network is critical for the security, stability, and resilience of decentralized Ethereum networks.

Furthermore, it helps avoid scenarios that can result in disruption, monetary loss, and centralization, which can be detrimental to the network and its users. Therefore, it is essential to strive for a well-distributed client diversity across the network to ensure its smooth operation and safety.


Exploring Ethereum’s Client Ecosystem was originally published in Ethereum Cat Herders on Medium, where people are continuing the conversation by highlighting and responding to this story.