Event-driven is a philosophy. Go makes it practical. But most Magento teams that \u201cgo event-driven\u201d discover that adding a queue between systems isn\u2019t the hard part \u2014 handling failure, reprocessing, and consumer coordination is.<\/p>\n
Most Magento architectures start synchronous. An admin saves a product. The observer fires. The ERP gets updated via HTTP. The PIM gets notified. The search index rebuilds.<\/p>\n
Then the ERP is slow one morning. The HTTP timeout fires. The observer throws an exception. And suddenly saving a product in the admin panel is failing.<\/p>\n
The synchronous coupling is the problem. Not the observer pattern. Not Magento. The assumption that all connected systems must be available right now.<\/p>\n
Event-driven architecture breaks that assumption. When a product is updated, you publish an event. Go workers consume it. Systems get updated asynchronously, independently, and with retry logic that doesn\u2019t block the merchant.<\/p>\n
It does not mean \u201cwe have RabbitMQ now.\u201d<\/p>\n
Event-driven is a boundary decision. You\u2019re saying: the source of truth publishes facts, and consumers react to those facts on their own schedule. Neither side knows \u2014 or cares \u2014 about the other\u2019s availability.<\/p>\n
In commerce, the relevant events are:<\/p>\n
These events don\u2019t all need the same consumers. A price change event might feed a PIM system for record-keeping, a search index for faceted filtering, a pricing cache invalidation job, and a CDN purge for cached product pages.<\/p>\n
Each consumer runs independently, at its own pace, with its own retry strategy.<\/p>\n
Go\u2019s concurrency model \u2014 goroutines and channels \u2014 makes building event consumers natural. But \u201cnatural\u201d doesn\u2019t mean \u201cautomatic.\u201d You still need to design for failure.<\/p>\n
If you have one worker consuming one queue, you have a single point of failure and a throughput ceiling.<\/p>\n
Consumer groups solve both problems. Multiple Go worker instances compete to consume from the same queue. Messages are distributed across workers. If one dies, others continue.<\/p>\n
With RabbitMQ, this means multiple consumers on the same queue. With Kafka, it means a consumer group where each partition is owned by one worker at a time. The semantics differ, but the principle is the same: horizontal scaling without coordination overhead.<\/p>\n
<\/a>\/\/ Worker pool \u2014 competing consumers, context-cancellable<\/span><\/span>\n<\/a>func<\/span> startWorkerPool(<\/span>ctx context.<\/span>Context,<\/span> queue Queue,<\/span> workers int<\/span>)<\/span> {<\/span><\/span>\n<\/a> for<\/span> i :=<\/span> 0<\/span>;<\/span> i <<\/span> workers;<\/span> i++<\/span> {<\/span><\/span>\n<\/a> go<\/span> func<\/span>()<\/span> {<\/span><\/span>\n<\/a> for<\/span> {<\/span><\/span>\n<\/a> select<\/span> {<\/span><\/span>\n<\/a> case<\/span> msg :=<\/span> <-<\/span>queue.<\/span>Messages():<\/span><\/span>\n<\/a> processWithRetry(<\/span>ctx,<\/span> msg)<\/span><\/span>\n<\/a> case<\/span> <-<\/span>ctx.<\/span>Done():<\/span><\/span>\n<\/a> return<\/span><\/span>\n<\/a> }<\/span><\/span>\n<\/a> }<\/span><\/span>\n<\/a> }()<\/span><\/span>\n<\/a> }<\/span><\/span>\n<\/a>}<\/span><\/span><\/code><\/pre>\n<\/div>\nScale workers based on queue depth, not wall-clock time. If your PIM sync is behind during flash sales, spin up more consumers \u2014 don\u2019t wait for the queue to drain overnight.<\/p>\n
Reprocessing and Idempotency<\/h3>\n
Messages will be delivered more than once. Network partitions, consumer crashes, acknowledgment failures \u2014 all of these cause redelivery.<\/p>\n
Your handlers must be idempotent. The same message processed twice must produce the same result.<\/p>\n
For a PIM sync worker consuming Magento product update events, idempotency means keying on product_sku<\/code> + updated_at<\/code>, ignoring messages with timestamps older than the last successful sync, and using upsert semantics rather than insert-or-fail.<\/p>\nThis isn\u2019t theoretical. In production, during a queue catch-up event (consumer down for 30 minutes, then restored), you will see thousands of duplicate deliveries. Systems that aren\u2019t idempotent produce duplicate data, corrupted records, or silent failures.<\/p>\n
Poison Messages<\/h3>\n
Some messages will always fail. The ERP is down. The schema changed. The product ID no longer exists.<\/p>\n
If you retry indefinitely, one bad message blocks your entire consumer. The queue backs up. Fresh events stop processing.<\/p>\n
Dead-letter queues (DLQs) are the answer. After N retries, move the message to a DLQ with the original payload, the error, and metadata. Your main consumer keeps moving. You process the DLQ separately \u2014 manually, or with a repair worker.<\/p>\n
\n<\/a>func<\/span> processWithRetry(<\/span>ctx context.<\/span>Context,<\/span> msg Message)<\/span> {<\/span><\/span>\n<\/a> var<\/span> lastErr error<\/span><\/span>\n<\/a> for<\/span> attempt :=<\/span> 0<\/span>;<\/span> attempt <<\/span> maxRetries;<\/span> attempt++<\/span> {<\/span><\/span>\n<\/a> if<\/span> err :=<\/span> handler.<\/span>Process(<\/span>ctx,<\/span> msg);<\/span> err !=<\/span> nil<\/span> {<\/span><\/span>\n<\/a> lastErr =<\/span> err<\/span>\n<\/a> backoff :=<\/span> time.<\/span>Duration(<\/span>attempt*<\/span>attempt)<\/span> *<\/span> time.<\/span>Second<\/span>\n<\/a> time.<\/span>Sleep(<\/span>backoff)<\/span><\/span>\n<\/a> continue<\/span><\/span>\n<\/a> }<\/span><\/span>\n<\/a> return<\/span> \/\/ success<\/span><\/span>\n<\/a> }<\/span><\/span>\n<\/a> \/\/ All retries exhausted \u2014 send to DLQ<\/span><\/span>\n<\/a> dlq.<\/span>Publish(<\/span>ctx,<\/span> DeadLetter{<\/span><\/span>\n<\/a> OriginalMessage:<\/span> msg,<\/span><\/span>\n<\/a> Error:<\/span> lastErr.<\/span>Error(),<\/span><\/span>\n<\/a> Attempts:<\/span> maxRetries,<\/span><\/span>\n<\/a> FailedAt:<\/span> time.<\/span>Now(),<\/span><\/span>\n<\/a> })<\/span><\/span>\n<\/a>}<\/span><\/span><\/code><\/pre>\n<\/div>\nExponential backoff. Bounded retries. DLQ for human inspection. This is non-negotiable in production.<\/p>\n
Real Commerce Use Cases<\/h2>\nPIM \u2192 Magento Sync<\/h3>\n
Your PIM is the source of truth for product content. Magento is the publishing target.<\/p>\n
When a product editor updates content in the PIM, an event fires. A Go worker consumes it, transforms the PIM schema to Magento\u2019s attribute structure, and calls the Magento REST API via go-m2rest<\/a>.<\/p>\nThe worker handles rate limiting (Magento\u2019s API has limits), retry on 500\/503 responses (auto-retry is built into go-m2rest), conflict resolution based on timestamps, and DLQ routing for schema mismatches.<\/p>\n
The PIM doesn\u2019t know Magento exists. Magento doesn\u2019t know the PIM exists. They communicate via events and a shared schema contract.<\/p>\n
CRM Sync on Order Events<\/h3>\n
When an order is placed, customer behavior data flows to your CRM. New customer? Create a contact. Repeat purchase? Update the loyalty score. High-value order? Trigger a sales notification.<\/p>\n
A Go worker consumes order.placed<\/code> events and fans out to multiple CRM operations. Use a message broker topic (Kafka) or exchange (RabbitMQ) to route events to multiple consumer groups.<\/p>\nEach consumer group is independent:<\/p>\n
\n- CRM sync worker \u2014 loyalty score update<\/li>\n
- Fraud detection worker \u2014 risk scoring<\/li>\n
- Email marketing worker \u2014 post-purchase flow trigger<\/li>\n<\/ul>\n
One event, three independent systems, zero coupling.<\/p>\n
ERP Stock Sync<\/h3>\n
ERP systems push stock updates \u2014 sometimes batch, sometimes near-real-time. Either way, Magento needs to reflect current inventory.<\/p>\n
A Go worker consumes stock update events and calls Magento\u2019s inventory API. The ordering challenge is real: if you receive two updates for the same SKU out of order, you might set stock to an older value.<\/p>\n