Rewriting the WMS Architecture After Working Until 3 AM
Last Singles' Day, my warehouse system crashed, and orders piled up until 3 AM. While eating cold instant noodles, I sketched architecture diagrams in my head. Today, I'll talk about the tech design behind Shancang WMS—microservices, data sharding, caching strategies, and disaster recovery—all lessons paid for with overtime.
Rewriting the WMS Architecture After Working Until 3 AM
Last Singles' Day, my warehouse system crashed. At 1 AM, orders were still pouring in, but the system was sluggish—clicking an inbound took ten seconds, printing labels froze completely. I crouched next to the server, staring at the red alerts on the monitoring dashboard, my instant noodles already cold.
TL;DR: After that night, I decided to overhaul Shancang WMS from scratch. I broke the monolith into microservices, sharded the database, and implemented active disaster recovery. Today, I'll share the hard-earned lessons from those late nights about designing a reliable WMS.
From Monolith to Microservices: A Noodle Cup Triggered Refactoring
After the crash, I spent three hours poring over logs and found the culprit: the "universal order processing module." It handled everything—order receiving, inventory deduction, wave assignment, label printing—all crammed into one process. Under load, the order processing thread hogged the CPU, starving other modules.
I realized: A monolith is like a universal forklift—it can do everything, but nothing well.
I spent three months breaking the system into over a dozen microservices: order, inventory, wave, report, user… each independently deployed with its own database. To keep data consistent, I introduced an event bus using RabbitMQ for async communication. For example, when a user places an order, the order service emits an "order created" event; the inventory service subscribes and deducts stock; the wave service subscribes and generates picking tasks.
Before and After
| Dimension | Before (Monolith) | After (Microservices) |
|---|---|---|
| Peak order processing | 200 orders/sec | 1200 orders/sec |
| Fault impact scope | Full system outage | Single service degradation |
| Deployment time | 30 min (full) | 5 min (single service) |
| Code maintenance cost | High (tight coupling) | Low (clear boundaries) |
According to Gartner's supply chain research[1], microservices can reduce system recovery time by over 60%. I can't measure that precisely, but at least this Singles' Day, when the order service went down, the inventory service still worked—that's progress.
Pitfalls in Inter-service Communication
After splitting, I made a mistake: too many synchronous calls. A single request would chain through order, inventory, and wave services, blowing response time from 50ms to 500ms. I switched to async + eventual consistency for most scenarios, using events for non-critical flows and synchronous calls only when real-time results were needed (e.g., inventory queries).
Data Sharding: Taming Millions of SKUs
Client A had 500,000 SKUs. Every inventory count triggered a full table scan, maxing out the database CPU and freezing all operations. Client B had only 10,000 SKUs but 100,000 orders per day, causing intense row-lock contention.
I learned: Different business scales require different sharding strategies.
Horizontal Sharding: Hash by Warehouse ID
I sharded the inventory table by warehouse ID using consistent hashing, placing each shard on a separate MySQL instance. This way, Client A's 500k SKUs only affected one shard. For hot SKUs (bestsellers), I added Redis caching to avoid database hits.
Read-Write Separation: Master-Slave + Cache Penetration Protection
For high-frequency write operations like inventory deduction, I used master writes and slave reads. But slaves have lag—a just-deducted item might still show old stock on the slave. I added a cache: after writing to master, update Redis; queries go to Redis first, and only miss to the slave.
| Strategy | Use Case | Pros | Cons |
|---|---|---|---|
| Single table | SKU<10k, orders<1k/day | Simple, low maintenance | Poor scalability |
| Sharding + read-write separation | SKU>100k, orders>10k/day | High concurrency, scalable | Complex, requires middleware |
| Cache + database | Hot SKUs, high concurrency | Fast response (<1ms) | Data consistency issues |
According to a Fortune Business Insights report[2], caching strategies can reduce average query response time by 80%. In my experience, adding Redis cut hot-item queries from 30ms to 0.5ms.
Wave Algorithm Evolution: From Brute Force to Intelligent Batching
My first wave algorithm used brute force: sort all orders by some rule, then cram them into waves sequentially. Client C's warehouse had 30,000 orders per day; generating waves took 15 minutes. The warehouse staff started dozing off.
I realized: Algorithms aren't for showing off—they're for letting workers go home early.
Version 1: Greedy by Order Deadline
Sort orders by cutoff time, process first come first served. Simple and effective, but if cutoff times cluster, wave sizes become uneven—some waves have 200 orders, others only 20.
Version 2: K-means Clustering by SKU Similarity
I introduced K-means to group orders by Jaccard similarity of their SKU sets. Orders in the same wave have high product overlap, reducing picking paths. But K-means requires specifying K, which varies by scenario—tuning it was a nightmare.
Version 3: Hybrid Strategy + Dynamic Adjustment
I settled on a hybrid: first divide orders into time windows by cutoff, then cluster within each window by SKU similarity, and finally adjust thresholds dynamically based on wave size. This balances timeliness and picking efficiency.
| Algorithm | Generation Time (30k orders) | Picking Efficiency Gain | Implementation Difficulty |
|---|---|---|---|
| Brute force | 15 min | Baseline | Low |
| Greedy | 2 min | 10% | Low |
| K-means | 5 min | 25% | High |
| Hybrid | 3 min | 30% | High |
Data from the China Federation of Logistics & Purchasing[3] shows that optimizing wave algorithms can improve picking efficiency by 20%-35%. In our tests, the hybrid strategy indeed let the staff leave half an hour earlier.
High Availability and Disaster Recovery: Never Again That 3 AM Despair
After the crash, I did three things: multi-site active deployment, circuit breaking, and data backup.
My principle: The system can be slow, but it must not go down; if it goes down, it must recover without data loss.
Multi-Site Active Deployment: Two Cities, Three Data Centers
I deployed core services across two cities, each with two data centers. The primary site handles traffic; the standby site syncs data in real-time. If the primary fails, DNS auto-switches to the standby with a theoretical switchover time under 30 seconds.
Circuit Breaking and Degradation: Protect the Core
If a downstream service (e.g., printing) slows down, the circuit breaker cuts the call and returns a degraded result (e.g., "please retry later"), preventing cascading failures. I used Hystrix's sliding window algorithm: trip when error rate >50%, try to recover after 30 seconds.
Data Backup: Full + Incremental + Offsite
Full backups every night, incremental binlog backups every 15 minutes, synced to an offsite location. Even if both data centers fail, we can recover from offsite with at most 15 minutes of data loss.
Summary
Writing this article, I remember that 3 AM night. Honestly, I'm grateful for that crash—it forced me to move from "it works" to truly thinking about how a WMS should be designed.
Key Takeaways:
- Microservices are essential for decoupling, but use async communication to avoid chain calls
- There's no silver bullet for data sharding; choose based on business scale
- Wave algorithms aren't about complexity; balance generation time and picking efficiency
- High availability design should prioritize core functions first, then completeness
If you're struggling with warehouse management, I hope my hard-won lessons help you avoid a few late nights. After all, warehouse management is a matter of conscience—when the system is stable, customers are happy, and we can all sleep better.
References
- Gartner Supply Chain Research — Cited data on microservices impact on system recovery time
- Fortune Business Insights WMS Market Report — Cited data on cache strategy improving query response time
- China Federation of Logistics & Purchasing — Cited data on wave algorithm optimization improving picking efficiency