When performing an on-chain transaction, you must first broadcast it into a waiting room known as the mempool. You can do this using your node, or broadcast your transaction via a third-party node, and eventually, your transaction details will make the rounds.
As other nodes pick up the transaction, it’s added to their local mempool and becomes part of the greater public mempool. Miners then seek out transactions housed in this wider mempool and use them to compile the transactions they wish to put into a block.
Every node has a local mempool, and every node may have policies in place to manage its mempool size or prioritise transactions. Some nodes also remove transactions from the mempool after a certain period of time if they haven’t been included in a block.
When it’s your node, it’s your rules.
For the most part, nodes tend to read and sing from the same hymn sheet, but there can be times when some nodes might deviate from others.
This can be due to local censorship, for example, a public miner or public company can filter their node to avoid transactions from OFAC-sanctioned addresses, or other nodes like Knots and Ordisrespector runners, turn their noses up at any meta protocol transactions.
When transactions are excluded from enough nodes, they look to the fringe nodes willing to take them on, and these nodes can be categorised as dark mempools.
What is the general Mempool?
The standard mempool is public and transparent. Anyone running a node can see pending transactions, including their details such as sender, recipient, amount, and associated fees.
So, where does the general mempool get all the transactions from? Well, the gossip, of course.
The girls are gossiping
The Bitcoin mempool is not a single entity but rather a collection of individual memory pools that each Bitcoin full node maintains. To ensure that transactions spread across the network and reach miners for inclusion in blocks, nodes employ a “gossip protocol.”
Periodically, a node will select a few random peers from its list of connected nodes. It then shares information about the new transactions it has in its mempool with these selected peers.
The peers receiving this information then perform validation checks on the newly received transactions. If valid and not already in their mempool, they add them to their local mempool and, in turn, gossip these new transactions to a few of another set of random peers.
This process of nodes gossiping new transactions to their peers continues, creating a ripple effect that helps to rapidly disseminate transaction information across the entire Bitcoin network. Even if some nodes are temporarily offline or have limited connectivity, the gossip protocol’s redundancy ensures that valid transactions eventually reach the majority of nodes, including those of miners.
Where do Dark Mempools come from?
A dark mempool, as the name suggests, operates in the shadows. It’s a private transaction pool that isn’t visible to the general public or typical network participants. Only specific miners, mining pools, or trading entities have access to these exclusive transaction pools.
In essence, dark mempools create a two-tier system where certain transactions bypass the public waiting room and get fast-tracked for inclusion in blocks through private channels and often involve out-of-band payments for transactions.
How Dark Mempools Work?
Dark mempools are run by miners or businesses with direct relationships with mining pools. Through these direct relationships, they can submit a local mempool of transactions to miners who will produce the blocks.
Here’s how the process typically works:
- Private Agreements: Large players establish relationships with mining pools, offering guaranteed fees in exchange for priority transaction processing.
- Transaction Submission: Instead of broadcasting transactions to the public network, participants send them directly to specific miners via private channels.
- Priority Processing: Miners include these transactions in blocks they mine, often ahead of public mempool transactions even when the latter might offer higher fees.
- Settlement: The transactions appear on the blockchain once the block is mined, with no indication that they bypassed the public mempool.
Why Dark Mempools Exist
Dark mempools emerged primarily for the following reasons:
1. Front-Running Protection
In public mempools, transactions are visible before confirmation. This visibility creates opportunities for front-running, where observers can see a pending high-value transaction and submit their own similar transaction with higher fees to get processed first.
For large traders or institutional investors, this transparency can be costly.
Dark mempools protect against front-running by keeping transactions private until they’re already included in a block, preventing others from seeing and exploiting trading intentions.
2. Guaranteed Transaction Processing
During periods of network congestion, transaction fees can spike dramatically as users compete for limited block space. Dark mempools offer predictability through pre-arranged fee structures and processing guarantees.
For example, a service like SlipStream by Mara allows users to submit their transactions directly to Mara for inclusion in their next block, and because they’re such a large miner, they can offer this service knowing they’ll reasonably secure enough blocks per day.
3. Reduced Fee Volatility
For entities executing large volumes of transactions, sudden fee spikes in competitive public mempools can significantly increase operational costs. Private arrangements through dark mempools often feature stable fee structures and even allow for deals to be paid in currencies other than Bitcoin.
For example, dark mempools can accept Altcoins and stablecoins like USDT or bank transfer instead of whatever the market rate could be for fees in sats per vbyte.
The Impact of Dark Mempools
The existence of dark mempools has several implications for cryptocurrency ecosystems:
Market Efficiency and Fairness
Dark mempools create information asymmetry in what many believe should be open and transparent systems. While they protect certain participants from front-running, they also give those with privileged access advantages over regular users.
This raises questions about market fairness and whether blockchain networks are truly as decentralised as they claim to be.
Network Security Considerations
Some argue that dark mempools might impact network security by creating financial incentives for miners to prioritise private transactions over public ones. This could potentially lead to centralisation risks if a small group of miners captures most of the profitable private transaction flow.
MEV (Maximal Extractable Value)
Dark mempools are also a common tactic for traders looking to avoid MEV (Maximal Extractable Value) – the profit miners can extract from reordering, including censoring transactions.
While public MEV extraction is observable, MEV extraction through dark mempools remains largely hidden, making it difficult to quantify its total economic impact.
MEV has been primarily an issue for smart contract chains like Ethereum, where DEFI trades are processed on-chain, miners or validators can review the mempool and then place competing bids and front-run certain large trades.
According to Coindesk, the percentage of Ethereum users opting for dark mempool submission was about 7% in September 2022, but after Ethereum transitioned into a proof-of-stake network, the portion has jumped to 15% since the start of 2024.
While this was previously not an issue for Bitcoin since users were only trading one asset, with the introduction of meta protocols like Ordinals, STAMPS and RUNES, we’ve seen a growing use for MEV and sniping bots.
Filtering for Non-standard transactions
The push to filter out non-standard transactions from Bitcoin has not yet reached a scale where dark mempools are required, with Knots only sitting at 9% of active nodes on the network.
But had Bitcoin Core pushed for filters or Knots gained a sizable adoption, the need for dark mempools would have increased among the cohort of users who wish to trade on the Bitcoin blockchain using meta protocols.
Are Dark Mempools Problematic?
Whether dark mempools are beneficial or harmful depends largely on perspective:
For large traders and institutions, dark mempools provide necessary protection against predatory trading practices and ensure reliable transaction processing.
For blockchain purists, they represent a deviation from the core principles of transparency and equal access that underpin cryptocurrency philosophy.
For average users, dark mempools might mean longer wait times for transaction confirmation during network congestion, as miners prioritise their private arrangements.
There is no one mempool
Dark mempool is a natural reaction to the maturing Bitcoin market as more people want to use the network, and it meets the needs of a global audience.
Bitcoin is a network and a medium of exchange for enemies, and everyone has different ideas and interpretations. As long as the transaction parameters remain within consensus, they can continue to broadcast and secure block space.
While they may seem at odds with blockchain’s transparent ethos, they address real needs for certain participants and showcase Bitcoin’s censorship resistance.
For traders and investors, awareness of dark mempools and their effects on transaction processing and market dynamics is increasingly important for developing effective strategies in the cryptocurrency space. Even if you never directly interact with a dark mempool, its existence shapes the market in which all participants operate.
