MagnumDB Database Project
MagnumDB is an open-source database engine built in Rust for exploring storage engines, SQL execution, transactions, Write-Ahead Logging (WAL) durability, MVCC, and database internals.
What is MagnumDB?
MagnumDB is an experimental embedded KV and SQL database engine written in clean, idiomatic Rust. It combines core database engine components into an inspectable, self-contained codebase.
Why MagnumDB?
MagnumDB addresses three distinct workflows for database enthusiasts and systems developers.
Understand Database Internals
Understand how storage engines, transactions, Write-Ahead Logging (WAL), B+ Trees, and query execution work underneath high-level SQL abstractions.
Embed Inside Rust Apps
Use MagnumDB as an embedded database engine inside Rust desktop software, local-first applications, and command line utilities requiring local data storage.
Hack & Research Infrastructure
Modify the engine, experiment with page sizes or cache replacement algorithms, and study how database systems are implemented in memory-safe Rust.
What can you use MagnumDB for?
- •Educational database projects and academic coursework
- •Embedded relational or KV storage inside Rust applications
- •Local-first desktop software and offline CLI tools
- •Developer tools and local test harness environments
- •Database internals research and storage engine experiments
- •Custom SQL query planner or executor algorithm prototyping
- •Systems programming learning projects in Rust
Features
| Feature | Description | Documentation |
|---|---|---|
| B+ Tree | Disk-backed B+ Tree storage engine with slotted 4 KB pages. | Docs |
| WAL | Append-only Write-Ahead Logging for crash recovery durability. | Docs |
| Buffer Pool | LRU page frame manager with pinning and dirty writeback. | Docs |
| MVCC | Multi-Version Concurrency Control with xmin/xmax versioning. | Docs |
| Transactions | BEGIN, COMMIT, ROLLBACK statements under Read Committed isolation. | Docs |
| SQL Engine | Relational query execution supporting DDL, DML, JOINs, and CTEs. | Docs |
| PostgreSQL Protocol | Wire-protocol TCP server support for psql and Postgres drivers. | Docs |
| Embedded Mode | Direct in-process Rust API access via Database, Config, Executor. | Docs |
| Server Mode | Standalone TCP process listening on port 5432. | Docs |
Architecture Overview
Database Internals Documentation
Detailed technical documentation explaining each underlying database engine primitive.
4 KB page layout, slotted tuple arrays, and free-lists.
Key indexing, internal nodes, leaf sequence links, and page splits.
In-memory LRU page frame caching, pinning, and writeback.
Fixed 4096-byte I/O page block abstraction and file headers.
Append-only log records, CRC32 checksums, and fsync calls.
Startup WAL record replay to reconstruct committed state.
BEGIN, COMMIT, ROLLBACK semantics and WAL log tagging.
Multi-Version Concurrency Control using xmin and xmax versioning.
RwLock synchronization and table lock management.
Rule-based AST transformation into physical execution plans.
Iterator tuple streams implementing open(), next(), and close().
Frontend/backend packet format and TCP socket handling.
SQL Support Matrix
| SQL Command | Support Status | Execution Notes |
|---|---|---|
| CREATE TABLE | Supported | Supports PRIMARY KEY, UNIQUE, NOT NULL, DEFAULT |
| INSERT INTO | Supported | Appends tuples into B+ Tree slotted leaf pages |
| SELECT | Supported | Filtering via WHERE clauses and binary expressions |
| UPDATE | Supported | Creates new tuple version with current xmin under MVCC |
| DELETE | Supported | Marks tuples as deleted by setting xmax transaction ID |
| DROP TABLE | Supported | Deallocates relation B+ Tree metadata and pages |
| ALTER TABLE | ADD COLUMN Supported | Appends new attribute to schema definition |
| JOIN | INNER / LEFT Supported | Evaluated by Volcano physical join operators |
| GROUP BY & HAVING | Supported | Aggregates tuple groups and filters calculated values |
| ORDER BY & LIMIT | Supported | Sorts result sets and applies offset boundaries |
| CTEs (WITH clause) | Supported | Evaluates named temporary CTE result streams |
| Window Functions | ROW_NUMBER Supported | Evaluates partition window frame functions |
| UNION / UNION ALL | Supported | Combines query result streams |
| FOREIGN KEY | Parsed, NOT Enforced | Syntax parses cleanly, but referential integrity is not enforced |
FOREIGN KEY syntax is parsed by the lexer for DDL script compatibility, but referential integrity constraints are NOT currently enforced by the query execution engine.
Getting Started Preview
Integrating MagnumDB into your Rust project requires only adding the magnumdb crate dependency.
Once added, call Database::open(config) to initialize the disk pager and LRU buffer pool.
123456789101112131415use magnumdb::{Config, Database}; use magnumdb::sql::{Executor, Parser}; fn main() -> anyhow::Result<()> { // 1. Initialize database path and buffer pool let config = Config::default().with_path("./my_database"); let mut db = Database::open(config)?; // 2. Execute SQL query statements let mut exec = Executor::new(&mut db); let ast = Parser::parse("CREATE TABLE users (id INT PRIMARY KEY, name TEXT);")?; exec.execute(ast)?; Ok(()) }
Embedded vs Server Mode
MagnumDB can be deployed as an in-process Rust library or as a standalone PostgreSQL wire protocol network server.
Embedded Mode
In-Process Library- •Links directly into Rust binaries without managing external server processes.
- •Direct API access to
Database,Config, andExecutor. - •Zero network socket IPC overhead for local data operations.
Server Mode
Standalone TCP Process- •Runs standalone TCP listener accepting PostgreSQL frontend/backend wire protocol frames.
- •Connect via standard PostgreSQL client drivers and the
psqlCLI client. - •Supports MD5 password challenge authentication negotiation for user principals.
Security Policy & Known Limitations
MagnumDB does not implement TLS encryption or modern network hardening. Deploy strictly within local loopback interfaces (127.0.0.1) or isolated development networks.
Network socket communication over TCP port 5432 is unencrypted plaintext.
Password verification uses legacy MD5 challenge responses without SCRAM-SHA-256.
The default postgres role operates with full superuser permissions without RBAC scoping.
Snapshot isolation is limited to Read Committed. Non-repeatable reads may occur.
Transaction commits acquire exclusive write locks, serializing throughput.
FOREIGN KEY clause syntax parses, but referential integrity constraints are not enforced.
Performance
MagnumDB does not currently publish comparative benchmark results.
Sequential & random tuple insertion rate
Key-based point lookup latency
Leaf page range scan bandwidth
Multi-client socket scaling & lock contention
WAL commit fsync & write operations
Expression evaluation & Volcano iterator overhead
LRU buffer pool footprint & page frame bounds
Slotted page header space amplification
MagnumDB vs Other Database Engines
MagnumDB is positioned as an educational and experimental engine for studying database implementation techniques.
| Engine | Primary Purpose | Embedded | SQL | KV | WAL | MVCC | Wire Protocol | Maturity |
|---|---|---|---|---|---|---|---|---|
| MagnumDB | Database Internals Education | Yes | Subset | Yes | Yes | Read Committed | PostgreSQL | Alpha / Experimental |
| SQLite | Embedded Relational Storage | Yes | Full | No | Yes (WAL Mode) | No (Lock-based) | None | Production Stable |
| PostgreSQL | Primary Application RDBMS | No | Full (ANSI) | JSONB | Yes (pg_wal) | Full MVCC | Native PostgreSQL | Enterprise Production |
| RocksDB | LSM Key-Value Store | Yes | No | Yes | Yes | Sequence Numbers | None | High-scale Production |
| redb | Embedded Rust KV Store | Yes | No | Yes | Copy-on-Write | No (CoW B-Tree) | None | Stable Rust Crate |
Roadmap & Project Status
Implemented Features (v0.4.8)
- •Embedded Rust library API (Database, Config, Executor)
- •Fixed 4 KB page disk pager and slotted page layout
- •B+ Tree engine with internal node splits & leaf overflow pages
- •Write-Ahead Logging (WAL) with CRC32 checksums & fsync
- •Startup WAL replay crash recovery
- •LRU buffer pool manager with page frame pinning
- •MVCC with xmin / xmax versioning and Read Committed isolation
- •Volcano-style query executor (open, next, close)
- •Rule-based SQL query planner & AST generator
- •PostgreSQL wire protocol TCP server & psql CLI compatibility
Planned Future Milestones
Note: Planned milestones represent future research goals and are not implemented in version 0.4.8.
- •WAL Checkpointing (Periodic fuzzy checkpointing to truncate log replay length)
- •TLS / SSL Socket Encryption for Server Mode
- •Differential Fuzz Testing against SQLite & PostgreSQL test suites
- •Foreign Key Referential Integrity Enforcement
- •SCRAM-SHA-256 Password Authentication
- •Repeatable Read & Snapshot Isolation Modes
- •Query Optimization Predicate Pushdown Rules
- •Comparative Performance Benchmarking Suite
Community & Contributing
MagnumDB development, architectural discussions, and issue tracking occur transparently on GitHub.
Built by Soham Das
MagnumDB is built and maintained as an open-source database project developed under the Blobly ecosystem, focused on database internals, storage engines, SQL, transactions, and systems programming in Rust.