In a cloud-only IoT tutorial, network connections are perfect. In the real world—especially on factory floors, mining sites, or remote solar grids—network partitions are a daily occurrence. Heavy machinery EMI, cellular signal drops, and local IT maintenance will regularly cut your edge gateway off from the internet.
If your edge gateway simply tries to write telemetry directly to a remote broker or API without an offline fallback, you will suffer immediate data gaps. In mission-critical environments, these gaps break predictive maintenance models, skew regulatory reports, and ruin audit trails.
To solve this, industrial engineers implement a Store-and-Forward architecture. This guide walks you through the design principles and database choices for building a highly resilient gateway buffering system.
The Architecture: Decoupling Ingestion from Egress
A resilient gateway must never block sensor data ingestion due to a network failure. We achieve this by splitting our software into two decoupled processes running asynchronously:
- The Ingestion Thread: Listens to local machine interfaces (Modbus, OPC-UA, CAN Bus) and immediately writes the incoming data into a local embedded database. This operation must be lightning-fast and run completely offline.
- The Forwarding (Egress) Thread: Monitors network availability. When connected, it queries the local database in chunks, transmits the data to the cloud MQTT broker or HTTPS ingestion endpoint, and deletes successfully acknowledged entries from the local database.
Choosing the Right Embedded Database
An edge gateway operates on constrained hardware (SD cards, eMMC flash) and must handle sudden power cutoffs. Traditional client-server databases are too heavy. Instead, we use embedded databases:
1. SQLite (Relational / WAL Mode)
SQLite is the most robust general-purpose choice. Because it stores data in a single file, it is extremely easy to manage. To use SQLite for high-frequency writes on flash storage, you must enable Write-Ahead Logging (WAL) mode:
PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;WAL mode allows concurrent reads and writes, and significantly increases write throughput while protecting against database corruption if the gateway loses power mid-write.
2. BadgerDB or RocksDB (Key-Value LSM Tree)
If you are dealing with very high frequency sensor data (e.g., thousands of vibration samples per second), SQLite's write overhead might be too high. Key-value databases built on Log-Structured Merge-trees (LSM) like BadgerDB (written in pure Go) or RocksDB (C++) are optimized for write-heavy workloads. They write to sequential log files, minimizing disk head movement and maximizing write speed.
Production Checklist: Crucial Store-and-Forward Considerations
Designing a store-and-forward pipeline involves solving three major edge-case behaviors:
1. Database Cap and Ring-Buffering
What happens if the internet goes down for a week? The gateway will continue storing data until it fills up the disk, crashing the operating system. You must enforce a hard database size limit. When that limit is reached, adopt a ring-buffer (FIFO) policy—discard the oldest records to make room for new ones, and flag a critical local alert.
2. Backpressure and Cloud Ingestion Spike Mitigation
If the gateway goes offline for 12 hours, it accumulates a backlog of millions of telemetry points. Once the network is restored, if the gateway tries to push all 12 hours of data at once, it might overload the cloud ingestion server or trigger DDoS protection rules. Your forwarding script must implement rate-limiting (backpressure control)—syncing data in controlled, batch sizes with a small delay between each batch.
3. De-duplication and Out-of-Order Ingestion
If the gateway forwards a batch of data, but the network drops right before receiving the MQTT broker's acknowledgment, the gateway will assume the transmission failed and send the same batch again. The cloud application must handle these duplicate payloads. Furthermore, the cloud data warehouse must support out-of-order ingestion, as the delayed "forwarded" historical data will arrive alongside the latest real-time telemetry.
Conclusion
Store-and-Forward is the difference between a toy IoT demo and a production-grade industrial solution. By decoupling data collection from cloud egress using an embedded WAL-mode SQLite database or LSM-tree KV store, you ensure that physical events on the factory floor are captured reliably—no matter the state of the network connection.
AdaptNXT builds resilient edge gateway software and high-throughput data synchronization engines for challenging industrial environments. Speak with our engineers about designing a bulletproof data pipeline for your company.
