redb for BDK

This summer I have been working on implementing a persistence backend (bdk_redb) for bdk_wallet and bdk_chain structures using redb as part of Summer Of Bitcoin'25 . redb is a pure Rust lightweight key-value database.

The motivation behind this is that the current persistence mechanism in BDK is based on sqlite which forces some application developers using redb for their application data to use two types of databases. Also another goal is to replace the file_store which is the current persistence backend used by the team for testing.

The Journey

In December, looking at BDK’s friendly discord and proposal to incorporate silent-payments in BDK made me feel that I am at the right place! Fast forward to March 2025 when project proposals for BDK came up I decided to go with the redb one since both databases and networking were new to me (math major here!) and I had spent more time with the wallet code. I really did not realize the importance of the project at that time. Fast forward to May 2025 I started working on the project after the first call with my mentor, notmandatory.

The main structure to be persisted is the Wallet which depends on structures from the bdk_chain: the TxGraph, the Indexer and the LocalChain. TxGraph holds the transactions provided by the chain source like electrum, esplora and bitcoind. The Indexer is used to index the transactions in the TxGraph and the LocalChain holds the block data provided by the chain source. Now, it is clearly wasteful to persist the whole structure each time it changes so rather we have the concept of ChangeSet. As the name suggests this structure records the changes to the corresponding structure. Then these ChangeSets are persisted in memory. So functions in bdk_redb take in ChangeSets rather than the actual structures! Along with these we also need to persist the Network and the descriptors corresponding to the wallet

Honestly, the project involved a lot of back and forth wherein we started with persisting the whole ChangeSets as it is. After the first iteration, notmandatory suggested that we could do the persitence more smartly like in rusqlite by persisting each individual fields of ChangeSets instead since some fields may change more frequently than others.

The next challenge was to decide the Rust types for the key and value in our redb database.

Conceptually, a redb database consists of tables which contain data in the form of key-value pairs. To use a certain structure as key and value we need to implement Key and Value trait respectively.

Initially we used primitive types (for which Key and Value trait are already implemented) to accomplish this and then after a suggestion from notmandatory we created wrappers around the structures we wanted to use as Key and Value and implemented the required traits for the same. Since the referential relationship between tables is unique to relational databases we tried replicating the behavior in bdk_redb using external logic.

Since we wanted to support multiple wallets in one db file we were forced to iterate over multiple irrelevant entries in a table which would clearly cause a performance issue. But my mentor in a masterstroke, suggested that we create a new table per structure and per wallet. We checked redb’s issues page and found out that performance is log(no. of tables) so this should not be an issue.

Owing to BDK’s focus on modularity we tried making bdk_redb as independent of bdk_wallet as possible by using feature flags and using bdk_chain structures wherever possible. This would allow users of bdk_chain structures to also use bdk_redb to persist their data.

Then came the error handling to replace the clones and unwraps. It is now that I realized that implementing Key and Value for the wrapper types does not allow us to return serialization and deserialization errors! So back to primitive types…
I initially started out with nested errors but while incorporating bdk_redb into bdk-cli I decided it would be cleaner for downstream users if we remove the nesting.

Panics are tools for developers and correspond to invariants which should never be broken by the code!

Afterwards due to my mentor’s suggestions we decided to modify the API to use references to databases instead of a redb::Database since that helps many applications to use the same redb::Database which was one of the motivations! We used Arc since it is thread-safe.

From the initial stage itself notmandatory advised me to have tests so that we have something working end to end. And this helped a lot! I felt a lot more confident introducing changes into my persistence functions since I knew that I have tests written already. Code-coverage tools are a great way to aid testing as they identify branches (which turned out helpful for me), regions, lines which are not being tested by the tests!

Unit tests should be focused on the function/scenario they are trying to test. It is much cleaner and maintainable to create a common test setup for the unit tests.

Learnings

At the start of the internship I was of the opinion that the production ready code that we see in BDK or Bitcoin Core is what a BOSS dev would come up with as soon as they seen an issue but I was so wrong. I learnt that great code actually happens in iterations where the first few iterations are for the POC stage and the rest build up to production ready code!

One of the outcomes of the project for me is increased proficiency in Rust. Before the project, I only knew the basic syntax and how to use the docs to find out relevant functions for the job! But the project made me go through multiple chapters from TRPL(the Holy Book of Rust) like Traits, Generics, Macros, Lifetimes, Concurrency, Error Handling, Closures etc., not to mention the countless blogs and Rust forum questions!

If you are ever stuck in implementing a trait take a look at how it has been implemented for the primitive types.

I learned seeing BDK’s codebase that generic design has more to it than just Rust generics! Rather than a collection, the parameter to a function should be an iterator so that any collection can be passed by converting to iter. A practice promoting forward compatibility is to use bounds when doing impl instead of applying bounds in the struct definition!

I also gained confidence in the BDK codebase. Earlier I had no idea how different parts of BDK interact and was particularly scared of bdk_chain but now after spending some time persisting these parts :sweat_smile: and going over PRs on the repo I feel better about bdk_chain :). The whole ChangeSet concept is actually very beautiful!

Trying to maintain the repo and keeping the commit history clean made me learn almost all common git commands: pull, fetch, push, add, commit, remote, squash, rebase, cherry-pick etc.

If you want to identify whether a particular use case is supported by a library, try to look in the repo’s open/closed issues.

Also being organized with questions and all the work done in the past week helps in dev-calls as it saves the time spent and back-and-forth done in trying to recall what all happened.

Work Done So Far

Currently bdk_redb contains a Store wrapper around a redb database with methods to persist and read BDK structures. Store also implements WalletPersister under the wallet feature flag which allows bdk_wallet users to use the Store for persistence. This Store can be used in place of the sqlite backend currently supported by BDK. We support persisting multiple wallets in a single database file. The crate has been published on crates.io now. Although the crate is still EXPERIMENTAL. DONOT use with MAINNET wallets. Also opened a PR to integrate bdk_redb into BDK’s playground called bdk-cli.

Looking Back

Honestly it was a marvelous feeling publishing my first crate (after starting out from scratch)! I hope this project will help BDK users and the Bitcoin ecosystem :)

Also I am really thankful for such an awesome mentor ! Getting this far wouldn’t have been possible without his constant help and guidance. Thank you Adi and SOB for such an awesome project and helping me get one step closer to my BOSS dream!

To all college students reading this, I highly recommend Summer Of Bitcoin, a chance to work with awesome people on an awesome tech that is helping people around the globe!

What Next ?

For developers implementing their own persistence crates it would be helpful if there is a generic persitence testing crate so that they do not need to write their own tests! The unit tests in bdk_redb can be extracted out to make this testing crate! This is what I have been working on lately.

So bye! See you next time! Till then,

Whoever you are whatever you do, do it for Bitcoin and Bitcoin will find you!