SAP Financial Operations
Internal, Atulaya
- Role
- Software Engineer
- Type
- Internal · Production
- Systems
- SAP Service Layer · MongoDB
- Scale
- 9 match categories · 7 defense invariants
Objective
SAP Business One holds the financial source of truth — invoices, credit notes, down payments — across every branch. Two problems sat on top of it: branch-to-branch balances had to be manually reconciled and cleared every cycle, and a subtle race condition in the posting pipeline was silently creating duplicate financial documents. This system automates the reconciliation math, posts the resulting SAP documents automatically, and closes the duplicate-posting hole for good — with fine-grained, role- and location-scoped access on top.
Why this is hard: this system writes real money into a financial system of record — there is no undo for a wrongly-posted SAP invoice, and the actual bug wasn't a logic error, it was a timing assumption (that an HTTP timeout means the write didn't happen) that was wrong just often enough to quietly duplicate financial documents in production before anyone noticed.
Decisions
The reconciliation engine
Each branch can carry outstanding balances against a shared business partner across multiple other centres. The engine computes, per partner per cycle: M_own (the branch's own invoices minus credit notes), M_clear (net clearance across every other branch), and D (any outstanding down payment). A strategy selector reads the sign combination of (M_own, M_clear) — nine categories in a 3×3 matrix — and each has its own SAP document-generation recipe. Matching runs on computed totals, not SAP's stored DocTotal, because that can drift by a small rounding difference that could hide a genuine duplicate.
The duplicate-posting incident
SAP's Service Layer can commit a write several minutes after the HTTP request that sent it has already timed out. A production audit found duplicate documents traced to two causes: an automatic-retry race on submission, and re-uploaded import batches with no cross-batch duplicate check. Three independently reasonable retry layers — an HTTP client's auto-retry, the job queue's own retry policy, and a duplicate-check that failed open — were collectively the problem.
The fix: invariants, not a patch
No automatic retries on writes. An idempotency token embedded in the document's own free-text field. Check-before-post, refuse-on-retry. A killed-worker safety net that surfaces an unresolved outcome as its own alertable state. A three-state outcome model — synced, errored, or outcome-unknown — replacing a binary success/fail that had no honest way to represent "SAP might have accepted this, we can't tell yet." Import-level dedupe against already-synced records. And audit rows that are archived, never deleted, after a prior retention job had deleted the only local record of what had been told to SAP.
Architecture
Runtime architecture
Admin (Nova) → Permissions gate → Job orchestrator → Reconciliation engine → SAP connector → SAP Service Layer
- The connector logs every raw request/response pair to a MongoDB audit store, separate from the run log.
- Every SAP request is tagged with a named flow and timed, feeding brownout-style alerting on the slow-write condition that caused the original incident.
SAP write outcome model
Submit (idempotency token attached) → Check-before-post → Write attempt (no auto-retry) → Synced / Errored / Outcome unknown → Alert
- The whole point: a write is never allowed to just silently disappear.
Result
Branch-to-branch reconciliation that used to be manual per-cycle work now runs as an automated batch pipeline. The duplicate documents that had been quietly accumulating in SAP were traced to root cause and stopped — not by adding a lock, but by removing every layer of automatic retry and replacing silent ambiguity with a visible, alertable third outcome state. Access is role-scoped by business function and independently scoped by branch, with every SAP write attributable to a real person.
By the numbers
Stack
Laravel, Nova, Horizon, Saloon (SAP Service Layer), Spatie Permissions, MongoDB
Lessons learned
- The duplicate-posting incident happened because three independently reasonable retry layers were never reviewed together — I'd audit every layer that can retry a write as a single cross-cutting concern before adding any one of them in isolation next time.
- The reconciliation engine needed a follow-up pass to undo AI-assisted over-engineering (cut by roughly a third without changing behavior) — I'd budget that simplification review as a standard step after any AI-generated implementation, not an afterthought.
- The three-state outcome model should have been the default from the start for any external write — "succeeded or failed" is a convenient lie whenever the other system's commit isn't synchronous with its response.