Export raw data from Magento. Transform it somewhere else. This is the pattern.<\/p>\n
Every Magento integration starts the same way. Someone writes a module that reads order data, reformats it into the ERP\u2019s expected structure, and pushes it via API \u2014 all inside a cron job. It works for the first integration. By the third, you have a transformation layer, a retry mechanism, and error handling logic buried inside Magento\u2019s runtime. You didn\u2019t plan to build middleware. But that\u2019s what you built \u2014 bad middleware, running inside your commerce engine.<\/p>\n
There\u2019s a cleaner approach, and it starts with accepting one thing: Magento should export raw data. Everything else \u2014 mapping, retries, failure handling \u2014 belongs in a dedicated middleware layer.<\/p>\n
It\u2019s the same story every time. A new integration requirement comes in \u2014 push orders to SAP, sync inventory from a WMS, send shipment confirmations to a 3PL. The team looks at what they have: Magento\u2019s cron, its queue consumers, its DI container. Why introduce another service when you can write a module?<\/p>\n
Here\u2019s the trap:<\/p>\n
var\/log\/<\/code> with no alerting. Failed records get silently skipped because nobody built a dead-letter queue inside a Magento module.<\/li>\n<\/ul>\nThe real problem isn\u2019t technical \u2014 it\u2019s organizational. Agencies scope Magento hours. Nobody budgets for middleware. So transformation logic grows inside the platform like mold behind drywall.<\/p>\n
The Pattern: Export Raw, Transform Outside<\/h2>\n
The architecture is straightforward:<\/p>\n
Magento exports raw commerce data.<\/strong> Orders, prices, inventory levels, customer updates \u2014 whatever the integration needs. The format is Magento\u2019s native structure. No transformation. No ERP-specific fields. Just clean commerce payloads pushed to a queue or exposed via API.<\/p>\nMiddleware receives, maps, and delivers.<\/strong> A Go service (or any dedicated middleware) picks up the raw payload, transforms it into the target system\u2019s DTO, and pushes it to the ERP, WMS, or 3PL. This is where field mapping lives. This is where business rules for transformation live. And this is where retry logic belongs.<\/p>\nThis separation has three immediate benefits:<\/p>\n
\n- Magento stays focused on commerce.<\/strong> No SAP field mappings polluting your order export module. No ERP connection timeouts blocking your cron runner.<\/li>\n
- Transformation logic is testable in isolation.<\/strong> You can unit test a DTO mapping function without bootstrapping Magento\u2019s DI container.<\/li>\n
- Each integration is independent.<\/strong> SAP\u2019s retry policy doesn\u2019t affect your shipping provider integration. Failures are isolated.<\/li>\n<\/ul>\n
DTO Mapping: Keep It Boring<\/h2>\n
DTO mapping sounds trivial. It\u2019s not \u2014 at scale.<\/p>\n
An order in Magento has nested structures: items, addresses, payment info, custom attributes, extension attributes. An ERP expects a flat document with specific field names, date formats, and enum values that don\u2019t match Magento\u2019s. The mapping between these two is where most integration bugs live.<\/p>\n
The rules are simple:<\/p>\n
\n- One mapper per integration target.<\/strong> Don\u2019t build a generic \u201ctransform anything\u201d engine. You\u2019ll end up maintaining a DSL nobody understands. Write explicit struct-to-struct mappings.<\/li>\n
- Validate before sending.<\/strong> After mapping, validate the output DTO against the target system\u2019s requirements. Catch missing fields and format errors before they become API failures.<\/li>\n
- Version your DTOs.<\/strong> When the ERP updates their API, you change the mapper. The Magento export doesn\u2019t change. The queue format doesn\u2019t change. Only the last mile changes.<\/li>\n<\/ul>\n
In Go, this is particularly clean. You define typed structs for both source (Magento payload) and target (ERP document), write a function that maps one to the other, and test it with table-driven tests. No reflection magic. No runtime surprises.<\/p>\n
If you\u2019re building Go services that consume Magento\u2019s REST API, go-m2rest<\/a> gives you typed structs and a search criteria builder out of the box \u2014 so you\u2019re not hand-rolling HTTP calls and JSON parsing for every integration.<\/p>\nRetries and Backoff: The Non-Negotiable<\/h2>\n
External APIs fail. ERP endpoints go down for maintenance. Rate limits get hit during peak sync windows. If your integration doesn\u2019t handle retries properly, you lose data.<\/p>\n
But \u201cjust retry\u201d is not a strategy. Here\u2019s what actually works:<\/p>\n
\n- Exponential backoff with jitter.<\/strong> Don\u2019t hammer a struggling endpoint with fixed-interval retries. Back off exponentially, add random jitter to prevent thundering herd, and cap the maximum delay.<\/li>\n
- Retry budget per message.<\/strong> Set a maximum retry count (typically 3-5). After that, the message moves to the dead-letter queue. Don\u2019t retry forever \u2014 you\u2019ll mask the real problem.<\/li>\n
- Distinguish transient from permanent failures.<\/strong> A 503 is retryable. A 400 with a validation error is not. Retrying a permanent failure wastes time and obscures the root cause.<\/li>\n
- Make retries idempotent.<\/strong> If you retry pushing an order to the ERP, the ERP must handle receiving the same order twice. This means idempotency keys, deduplication checks, or upsert semantics. Without this, every retry is a potential duplicate.<\/li>\n<\/ul>\n
In Magento\u2019s cron model, retry logic becomes a state machine managed in database columns \u2014 retry_count<\/code>, last_attempt<\/code>, status<\/code>. It works until you have 50,000 queued records and your cron job takes 45 minutes to scan the table. In a Go service, retries are part of the message processing loop with in-memory state and configurable policies per consumer.<\/p>\nDead-Letter Queues: Where Failed Messages Go to Be Fixed<\/h2>\n
A dead-letter queue (DLQ) is not a graveyard \u2014 it\u2019s a triage room. Messages end up here when they\u2019ve exhausted their retry budget. The DLQ gives you:<\/p>\n
\n- Visibility.<\/strong> You know exactly which records failed and why. No more grepping through Magento logs hoping to find the error.<\/li>\n
- Replay capability.<\/strong> Fix the root cause (bad mapping, API change, missing field), then replay the failed messages. No data loss. No manual re-entry.<\/li>\n
- Alerting.<\/strong> A message hitting the DLQ triggers an alert. Your team knows something is broken before the business notices missing orders.<\/li>\n<\/ul>\n
The pattern in practice: your Go consumer reads from a main queue (RabbitMQ, Kafka, SQS). If processing fails after N retries, it publishes the message to a DLQ topic with metadata \u2014 original timestamp, error message, retry count. A separate dashboard or CLI tool lets you inspect, fix, and replay.<\/p>\n
I built this exact pattern into tracking-updater<\/a> \u2014 failed shipment tracking updates move to a failed directory with full context, and can be reprocessed once the issue is resolved.<\/p>\nThe Magento Side of This<\/h2>\n
Magento\u2019s job in this pattern is minimal \u2014 and that\u2019s the point.<\/p>\n