Simulating Cross-Chain Communication for Incident Response Drills

Many DeFi projects now operate accross L1 and L2 networks leading to an increase in the complexity of monitoring and incident response. Practicing coordination of responses across Layer 1 (L1) and Layer 2 (L2) networks requires not only a deep understanding of cross-chain communication but also a testing environment to simulate real-world scenarios. In this post, we’ll explore a framework for simulating cross-chain communication on parallel network forks, focusing on Ethereum L1 and Optimism L2 interactions, and how it can be used for incident response drills.

Find the GitHub repo here.

To request a wargame drill for your protocol visit https://www.securityalliance.org/wargames

---

The Challenge with Multi-Chain Incidents

With L2s like Optimism, Arbitrum, and zkSync, protocols now span multiple chains, leveraging the scalability and cost-efficiency of L2s while maintaining the security of L1. However, this multi-chain architecture introduces significant challenges for incident response:

• Coordinated Actions: Responding to incidents often requires simultaneous actions on both L1 and L2, such as pausing bridges or freezing assets.
• Realistic Testing: Incident response drills must mirror real-world conditions, including realistic user activity and cross-chain interactions.
• Cross-Chain Communication: Understanding and testing how messages are passed between chains is critical for effective response.

Without a realistic testing environment, teams risk being unprepared for cross-chain incidents, which can lead to prolonged downtime or loss of funds. We can now simulate these scenarios through an updated framework for wargame drills.

---

A Framework for Cross-Chain Drills

This framework is designed to simulate cross-chain communication between Ethereum L1 and Optimism L2, with a focus on testing bridge transactions and message passing. By creating a realistic testing environment, teams can practice incident response in a controlled setting that mirrors production conditions.

---

Core Architecture

Test Environment Setup

To simulate cross-chain communication, we use forked environments of both Ethereum and Optimism. These environments are created using Tenderly Virtual Testnets, though the setup can be extended to other tools like Anvil. Key features of the test environment include:

• Forked Chains: Both Ethereum and Optimism are forked at specific blocks to replicate the state of the mainnet.
• Configurable Start Blocks: Teams can configure the starting block for each chain to simulate different scenarios.
• Cheat Codes & Modifications: By running drills on testnets we can make small modifications like introducing bugs to smart contract code, changing chain IDs to avoid replay with main networks, and introduce potential faulty upgrades.
• Account Impersonation: Send transactions as if they came from a specific address without needing keys to sign (Useful also for simulating governance/ multisig actions during the simulation)

Key Components

The framework consists of three core components that work together to simulate cross-chain communication:

1. Message Relayer

The Message Relayer is responsible for monitoring and relaying messages between L1 and L2. It:

• Monitors CrossDomainMessenger events on both chains.
• Replicates Optimism’s deposit specification
• Ensures that messages are relayed accurately and in the correct sequence.

2. User Simulation

To create a realistic testing environment, we use automated bots to simulate user activity. These bots:

• Perform bridge operations, including ERC20 and ETH transfers.
• Mimic realistic transaction patterns to replicate real-world usage.
• Generate activity on both L1 and L2 to test cross-chain interactions.
• During drills these bots are modified to include typical transactions with the relevant protocols being tested

3. Monitoring & Metrics

A monitoring system is essential for tracking cross-chain activity during these drills. Our setup includes:

• Bridge Transaction Tracking: Monitoring the flow of assets between chains.
• Bridge Status: Measure if bridges have deposits or withdrawals paused on L1 or L2
• Grafana + Prometheus: A visualization dashboard for real-time monitoring and analysis.

---

Demo: Cross-Chain Transactions in Action

Example Bridge Transaction

Below are example test transactions showing a bot bridging ETH and USDC from our forked L1 to the forked L2. The same cross domain messenger can be used for more complex transactions including cross chain governance.

ERC20 Deposit Initiated

Events

ERC20DepositInitiated
{
    "l1Token":"0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"
    "l2Token":"0x7f5c764cbc14f9669b88837ca1490cca17c31607"
    "from":"0xe16b5254a61757c30581b46fdefa4465ec79efef"
    "to":"0xe16b5254a61757c30581b46fdefa4465ec79efef"
    "amount":"725811265"
    "extraData":"0x"
}
SentMessage
{
    "target":"0x4200000000000000000000000000000000000010"
    "sender":"0x99c9fc46f92e8a1c0dec1b1747d010903e884be1"
    "message":"0x0166a07a0000000000000000000000007f5c764cbc14f9669b88837ca1490cca17c31607000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48000000000000000000000000e16b5254a61757c30581b46fdefa4465ec79efef000000000000000000000000e16b5254a61757c30581b46fdefa4465ec79efef000000000000000000000000000000000000000000000000000000002b43004100000000000000000000000000000000000000000000000000000000000000c00000000000000000000000000000000000000000000000000000000000000000"
    "messageNonce":"1766847064778384329583297500742918515827483896875618958121606201292775919"
    "gasLimit":"200000"
}

ERC20 Deposit Finalized

Events

RelayedMessage
{
    "msgHash":"0x733c45f903bcebd6120fd66d34c6a38e5643b47916932b99f7b4aa5019430035"
}
DepositFinalized
{
    "_l1Token":"0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"
    "_l2Token":"0x7f5c764cbc14f9669b88837ca1490cca17c31607"
    "_from":"0xe16b5254a61757c30581b46fdefa4465ec79efef"
    "_to":"0xe16b5254a61757c30581b46fdefa4465ec79efef"
    "_amount":"725811265"
    "_data":"0x"
}

ETH Deposit Initiated

Events

ETHDepositInitiated
{
  "from": "0x867e85779c4446a27489efcc91d6736b4e6029aa",
  "to": "0x867e85779c4446a27489efcc91d6736b4e6029aa",
  "amount": "857319916551426049",
  "extraData": "0x"
}
SentMessage
{
  "target": "0x4200000000000000000000000000000000000010",
  "sender": "0x99c9fc46f92e8a1c0dec1b1747d010903e884be1",
  "message": "0x1635f5fd000000000000000000000000867e85779c4446a27489efcc91d6736b4e6029aa000000000000000000000000867e85779c4446a27489efcc91d6736b4e6029aa0000000000000000000000000000000000000000000000000be5cfebfcea500100000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000",
  "messageNonce": "1766847064778384329583297500742918515827483896875618958121606201292775921",
  "gasLimit": "200000"
}
SentMessageExtension1 (Required for passing native value)
{
  "sender": "0x99c9fc46f92e8a1c0dec1b1747d010903e884be1",
  "value": "857319916551426049"
}

ETH Deposit Finalized

Events

DepositFinalized
{
  "_l1Token": "0x0000000000000000000000000000000000000000",
  "_l2Token": "0xdeaddeaddeaddeaddeaddeaddeaddeaddead0000",
  "_from": "0x867e85779c4446a27489efcc91d6736b4e6029aa",
  "_to": "0x867e85779c4446a27489efcc91d6736b4e6029aa",
  "_amount": "857319916551426049",
  "_data": "0x"
}
MessageRelayed
{
  "msgHash": "0xef8c6d1944657ba46fbca235d0a8ff887225a5ecd68066512cb2826839907672"
}

Comparison with Mainnet

To validate the accuracy of our simulation, we compare the test transaction with real bridge deposits on Optimism mainnet. We can inspect the logs and see the same events are emitted and transfers/ calls occur.

ERC20 Deposit Finalized

Tx: https://optimistic.etherscan.io/tx/0x16ecd5c848702f73af5aa0a6f506ccdbb58ebae52babaf224a01cf0a3af1fa13

ETH Deposit Finalized

Tx: https://optimistic.etherscan.io/tx/0x43b510b74c6e4dea775bd7fd525d259861ec996b08895c75ca6ed13a37a0ab5f

Message Passing Flow

The message relayer service tracks SentMessage events from L1 and relays them on L2. Key steps include:

1. L1 Deposit Initiation: The user initiates a deposit on Ethereum L1.
2. Message in CrossDomainMessenger: The deposit triggers a message in the CrossDomainMessenger contract.
3. L2 Transaction: The message is relayed to Optimism L2, where the corresponding transaction is executed.

Monitoring Dashboard

The Grafana dashboard provides real-time insights into cross-chain activity, including transaction volumes, deposits initiated and finalized. This can be extended to include relevant metrics for the protocol being tested.

---

Building the Infrastructure

Setting Up Test Networks

The framework uses Tenderly Virtual Testnets to create forked environments of Ethereum and Optimism. Key steps include:

• Forking mainnet at specific blocks.
• Modifying ChainId as needed to prevent replay of test transactions
• Verifying modified contracts on the fork explorer

Message Passing Implementation

The framework follows Optimism’s deposit specification to implement message passing. Key aspects include:

• Interacting with the CrossDomainMessenger contract.
• Managing the message queue to ensure proper ordering.
• For native transfers, set the balance of aliased L1 CrossDomainMessenger address using a cheat code to the value of the relayed message
• Impersonate the aliased L1 CrossDomainMessenger contract to send the transaction without needing a private key or system transaction method
• Handling edge cases and error conditions.

Monitoring Setup

The monitoring system tracks events on both chains and compares them with mainnet metrics. It includes:

• Event tracking using Prometheus.
• Real-time visualization with Grafana.
• Custom metrics for bridge activity and message relay performance.

---

Current Limitations

While the framework provides a robust testing environment, it has some limitations:

• L1 → L2 Deposits Only: The current implementation supports deposits but not withdrawals.
• Normal Rollup Operation: The framework assumes normal rollup operation and does not simulate fault conditions.
• Future Expansion: We plan to extend the framework to support additional L2s and more complex scenarios.

---

Open Source Repository

The framework is available as an open-source project on GitHub. Key features include:

• Easy to extend bots & services
• Docker Compose setup for monitoring.
• Setup scripts for configuring fork networks

---

Next Steps

We’re actively working on improving the framework, with plans to:

• Add support for L2 → L1 withdrawals.
• Extend the framework to other L2s
• Introduce additional monitoring metrics and dashboards.
• Welcome contributions from the community.

---

Resources

• GitHub Repository: https://github.com/security-alliance/drill-template-multichain
• Optimism Documentation:
  • Deposit Specification
• Related Tools and Projects:
  • Tenderly Virtual Testnets

---

By simulating cross-chain communication, teams can better prepare for real-world incidents and ensure the security and reliability of their protocols. We invite you to explore the framework, contribute to its development, and share your feedback.