Optimism Censorship Bug Disclosure

## Introduction

On 16 July 2023, Bernard Wagner, a security researcher at iosiro reported a bug to Optimism Labs in their sequencer stack that could be exploited to censor a subset of accounts on Optimism Mainnet. Optimism Labs patched the issue and thereafter paid a bounty.

## Background

A primary security concern of Optimism Mainnet, as well as other single-sequencer protocols, is the potential for issues in the sequencer to be a single point of failure for the whole network. As of the [Optimism Bedrock](https://community.optimism.io/docs/developers/bedrock/differences/) upgrade, all transactions, including transactions submitted through 3rd parties such as Alchemy, are routed through the same endpoint.

As a counterexample, on Ethereum Mainnet there is no centralized network-level component that can be targeted in such a way. There is redundancy in nodes and RPC providers, ensuring users always have some method of broadcasting transactions. Evidence of this can be seen in the relatively weak enforcement of OFAC sanctions on Tornado Cash. While there are some strategies of enforcement on different layers of the network (like wallets and RPC providers), in practice they rely more on intimidating users than technological chokepoints. This highlights the protocol’s censorship resistance; one of the foundational properties of any robust blockchain.

## Quick Context

Optimism’s sequencer is positioned behind `proxyd`, a custom JSON-RPC load balancer, which restricts the volume of requests the main sequencer has to manage. This setup applies rate limiting on a per-sender basis to avoid being bombarded with signed transactions causing a DOS (Denial of Service) of the sequencer.

To determine whether an account should be rate limited, `proxyd` tracks the number of signed transactions per account received within a time window. Once the threshold is reached, all transactions signed by the message sender will be rejected until the start of the next time window.

Transactions are discarded before contributing to the rate limit if the nonce is lower than the account’s current nonce, preventing a malicious actor from censoring an account by replaying previous transactions from that chain.

Source IP rate limiting is also performed by `proxyd` and will generally come into effect before the message sender rate limit, however, this is trivial to bypass with the use of an anonymizing proxy service.

In terms of the network stack, `proxyd` is part of the main sequencer or `op-node` within the Optimism Network, even though it operates as a separate Kubernetes service.

## Bug Details

The bug stemmed from the oversight that the chain ID was not validated before incrementing the rate limiter. As a result, signed transactions from other chains could be replayed, as long as the transaction nonce was higher than the current nonce of the account on Optimism Mainnet. Ultimately, the submitted transaction would be rejected by the sequencer due to [EIP-155](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-155.md), however, it would still activate the rate limiter.

While rate limited, the message sender could not submit any further transactions to Optimism Mainnet, leading to a "sender is over rate limit" JSON-RPC error. An account could be kept in this rate limited state indefinitely by spamming a large amount of replayed transactions.  

At the time of disclosure, around 1.3 million accounts were vulnerable to the attack. This included some of the most notable protocols and users of Optimism, such as LayerZero’s Executor and Oracle accounts, Across’ Relayer, Synapse’s Executor, Hop Bridge’s ETH Bonder, LayerSwap’s Relayer, Aave’s Deployer, Uniswap’s Deployer, and even many of Optimism’s multisig owners and the usual wallet transaction executor. A table at the end of this post gives some further details.

A [Dune Analytics query](https://dune.com/queries/2999264) was crafted to better understand the full scope of the attack.

## Impact

The issue was submitted as a critical risk issue. At the time of disclosure, the closest relevant category on Immunefi was causing “a transient consensus failure”, which is graded as high risk. Because there is a single sequencer, the impact of a transient consensus failure or general network DOS is complete temporary network outage. Using this as a starting point, it was argued that the ability to censor specific accounts for an attacker-controlled period is a significantly higher impact due to the following considerations.

1. Censoring specific accounts results in the ability to alter the effective state of the chain. Specific accounts will have their transactions censored while the rest of the network will continue to function as normal. This creates potentially profitable edge cases in the network.
2. A consensus failure is a catastrophic case and is immediately evident to all observers of the network. This attack left no on-chain remnants of censorship and was very difficult to detect. The only way to detect any abnormalities would be to [view `proxyd` logs](https://github.com/ethereum-optimism/optimism/blob/develop/proxyd/server.go#L291).
3. A consensus failure would be a point-in-time attack and resolve when an upgrade was deployed. This attack could be performed at strategic intervals controlled by the attacker to interfere with specific applications or user operations.
4. The attack was trivial to sustain and could be performed over an arbitrarily long period of time.  

Apart from generally undermining the availability of the network, a few specific attack scenarios presented to Optimism Labs are listed below; however, there are potentially many more application-specific edge cases that exist.

- Censoring of LayerZero’s Executor and Oracle, the two top transactors on Optimism. Doing so would deny service to LayerZero applications using the default relayer on Optimism, such as Stargate and Aptos. If exploited, bridging of assets from Aptos to Optimism would halt and could potentially need to be retried on Aptos once resolved.
Further impact to LayerZero could not be confirmed as LayerZero’s off-chain relayer code was not publicly available. After reaching out to LayerZero’s security team, who were highly responsive, they assured us that there would be minimal impact due to their queueing mechanism and enforcement of nonces.
- Censoring of bridges / relayers (e.g. Across, Synapse, LayerSwap, etc.) would result in bridging activities being temporarily DOSed.
- Delaying of push-based Oracle updates resulting in stale prices, such as WooOracle.
- Interfering with protocol keeper bots, potentially profiting from public keeper functions.
- Censoring centralized exchanges’ hot wallet addresses, preventing withdrawals.
- Censoring of 5 of the 7 owners of Optimism’s ProxyAdmin multisig wallet, as well as the usual transaction executor. It is described on Optimism’s website as: “If access to this account is lost, we will not be able to upgrade in an emergency”. Signatures could be collected offline and executed by another address, but a further delay in a high stress situation could be created.
- Prevent white hats or protocol owners from taking action during an ongoing attack. As became evident during the recent Curve Pool hacks, any delay in response could be catastrophic. Especially given that all actions would succeed when testing in a forked environment, but would simply timeout when broadcasting the transaction.

It should again be noted that none of these attacks would be possible with a transient consensus failure.

## Risk Rating

Optimism decided to downgrade the risk to medium risk on the basis that only a subset of users were affected by the issue. Their reasoning is provided below:

```diff
We have discussed this within the team and we believe that this should be classified as Medium, and are ready to pay $10,000 for this Medium priority vulnerability. Here is our reasoning behind the medium classification.

1. We do not currently have a category listed on Immunefi for a DoS attack such as this.
2. In our Sherlock audit contest, a DoS on critical services is classified the same severity as a consensus failure.
3. In our ImmuneFi program, transient consensus failures are classified as High.
4. 1 and 2 together means a DoS on critical services is at most High in the ImmuneFi program.
5. However, the DoS attack in this report can only be launched on targeted signers, and requires a high degree of difficulty to execute by identifying valid targets and creating a mechanism by which to profit from it.
```

Unfortunately, our arguments for the increased impact were never addressed.

We maintained that due to the number of affected accounts (again, over 1.3 million), as well as the fact that almost every major protocol on Optimism Mainnet could in some way be targeted through the attack, the scope of the attack should not be used as a mitigating factor.  

We are still of the opinion that a medium risk rating does not adequately capture the potential impact of the issue. However, in retrospect, submitting the bug as high risk would likely have been most appropriate given that an immediate large financial impact could not be demonstrated. As the bug was marked as “out of scope” by Immunefi and Optimism, we were unfortunately not in a position to argue the point any further.

## The Fix

Optimism prepared and deployed the fix in a [Github Pull Request](https://github.com/ethereum-optimism/optimism/pull/6358/files).

The test cases provided were expanded and feedback was given to Optimism that the change effectively addressed the issue given that the newly introduced `allowed_chain_ids` only contains a single `chain_id` for each `proxyd` cluster. If the array was to contain both Optimism Mainnet and Optimism Testnet the vulnerability could present itself again, however, for a much smaller subset of accounts.

The fix was pushed live 13 days after the initial disclosure.

## Déjà vu

During the disclosure process, it emerged that a similar vulnerability had previously been reported to Optimism in January 2023. In this case, any user on Optimism Mainnet could be targeted by replaying any signed transaction by the message sender. To mitigate this, the aforementioned nonce validation was implemented. The original fix from [commit 30db328](https://github.com/ethereum-optimism/optimism/pull/4797/files) is provided below as a reference:

```diff
@@ -631,7 +631,7 @@ func (s *Server) rateLimitSender(ctx context.Context, req *RPCReq) error {
return ErrInvalidParams(err.Error())
}

ok, err := s.senderLim.Take(ctx, msg.From().Hex())
ok, err := s.senderLim.Take(ctx, fmt.Sprintf("%s:%d", msg.From().Hex(), tx.Nonce()))
if err != nil {
log.Error("error taking from sender limiter", "err", err, "req_id", GetReqID(ctx))
return ErrInternal
```

However, this fix was insufficient and continued to allow for a subset of Optimism's users to be censored by replaying transactions from other networks with a higher user-nonce.

## Conclusion

Ultimately, Optimism offered to double their medium risk payout from $10,000 to $20,000 on the basis that it was in line with the previously reported bug. Furthermore, Optimism added the specific class of vulnerability as a medium risk to their bug bounty program. While we may have had a difference of opinion on the final risk rating, we appreciate Optimism Labs’ commitment to their bug bounty program.

On reflection, the disagreement highlights a concerning disparity between bug bounty programs for protocols and the underlying applications and protocols which rely on them. The impact is often framed as less severe from a network provider perspective than from the affected protocols’ perspectives. For example, LayerZero’s famous $15m bug bounty, would generally not apply in cases where the root cause of an issue lies in the networks on which it operates.

Hopefully, this dynamic will change in future and protocols and their applications can work together to ensure that incentives are better aligned to appropriately reward security researchers.

## Timeline

- 16 July 2023 - Reported the issue via Immunefi with a full write-up and a working POC.
- 16 July 2023 - Immunefi closed the report and marked it as out-of-scope.
- 17 July 2023 - Requested Immunefi to assist in contacting Optimism, reiterating the risk. Immunefi maintained that the report was out-of-scope, kept the report closed and took no further action.
- 18 July 2023 - Reached out to Optimism team members via Twitter.
- 18 July 2023 - Optimism team member responded via Twitter, acknowledging the report.
- 20 July 2023 - Optimism sent the public pull request containing the fix. The fix was reviewed and additional recommendations and tests were provided.
- 29 July 2023 - Fix was deployed. Optimism started impact assessment.
- *Further discussions related to impact omitted for brevity.*
- 24 August 2023 - Reward payout received.

## List of Notable Accounts Affected

| Name | Optimism Address | Note |
| --- | --- | --- |
| LayerZero Executor | [0xe93685f3bba03016f02bd1828badd6195988d950](https://optimistic.etherscan.io/address/0xe93685f3bba03016f02bd1828badd6195988d950) | The account with the most transactions on Optimism. |
| LayerZero Oracle | [0xb8ff877ed78ba520ece21b1de7843a8a57ca47cb](https://optimistic.etherscan.io/address/0xb8ff877ed78ba520ece21b1de7843a8a57ca47cb) | The account with the second most transactions on Optimism. |
| LayerZero Multisig Executor | [0x2458BAAbfb21aE1da11D9dD6AD4E48aB2fBF9959](https://optimistic.etherscan.io/address/0x2458BAAbfb21aE1da11D9dD6AD4E48aB2fBF9959) |  |
| StarGate Deployer | [0x1d7c6783328c145393e84fb47a7f7c548f5ee28d](https://optimistic.etherscan.io/address/0x1d7c6783328c145393e84fb47a7f7c548f5ee28d)  | Also responsible for executing Stargate multisig wallet transactions.  |
| Optimism L2 ProxyAdmin Owner | [0x7871d1187a97cbbe40710ac119aa3d412944e4fe](https://optimistic.etherscan.io/address/0x7871d1187a97cbbe40710ac119aa3d412944e4fe#readProxyContract) | A 5-of-7 multisig wallet, with 5  vulnerable signers. [0xad70ad7ac30cee75eb9638d377eacd8dfdfe0c3c](https://optimistic.etherscan.io/address/0xad70ad7ac30cee75eb9638d377eacd8dfdfe0c3c) the only executor that has been used to date on Optimism was also vulnerable.  |
| Optimism Foundation Multisig | [0x2501c477d0a35545a387aa4a3eee4292a9a8b3f0](https://optimistic.etherscan.io/address/0x2501c477d0a35545a387aa4a3eee4292a9a8b3f0#readProxyContract) | A 3-of-5 multisig wallet, with 3  vulnerable signers. |
| Hop Protocol ETH Bonder | [0x710bda329b2a6224e4b44833de30f38e7f81d564](https://optimistic.etherscan.io/address/0x710bda329b2a6224e4b44833de30f38e7f81d564)  |  |
| Synapse Bridge Executor | [0x230a1ac45690b9ae1176389434610b9526d2f21b](https://optimistic.etherscan.io/address/0x230a1ac45690b9ae1176389434610b9526d2f21b) |  |
| Across Relayer | [0x428ab2ba90eba0a4be7af34c9ac451ab061ac010](https://optimistic.etherscan.io/address/0x428ab2ba90eba0a4be7af34c9ac451ab061ac010) |  |
| LayerSwap Relayer | [0x2fc617e933a52713247ce25730f6695920b3befe](https://optimistic.etherscan.io/address/0x2fc617e933a52713247ce25730f6695920b3befe)  |  |
| WooOracle Keeper | [0x491e59c255c790d4e3a53cec2632524088f1aaa4](https://optimistic.etherscan.io/address/0x491e59c255c790d4e3a53cec2632524088f1aaa4) |  |
| Aave Deployer | [0x4365f8e70cf38c6ca67de41448508f2da8825500](https://optimistic.etherscan.io/address/0x4365f8e70cf38c6ca67de41448508f2da8825500) |  |
| Uniswap V3 Deployer | [0x6c9fc64a53c1b71fb3f9af64d1ae3a4931a5f4e9](https://optimistic.etherscan.io/address/0x6c9fc64a53c1b71fb3f9af64d1ae3a4931a5f4e9) |  |
| Owlto Finance Bridge | [0x45a318273749d6eb00f5f6ca3bc7cd3de26d642a](https://optimistic.etherscan.io/address/0x45a318273749d6eb00f5f6ca3bc7cd3de26d642a) |  |

This is by no means a comprehensive list. It is simply a list of interesting accounts that showed up during research into better understanding the potential impact.

‍