One order, six authorizations: tracking down a payment fan-out

The symptom the business saw

A manufacturer that takes card payments on sales orders ships in pieces when stock is split. The first shipment captures against the card authorization; when the second shipment is ready, the order needs a fresh authorization for what is left. A payment gateway bundle handles that: flag the order for reauthorization, and a scheduled job voids the old authorization and places a new one.

Most of the time it worked. Some of the time the second shipment went out the door and the invoice never came. The order sat on a failed-payment hold. Somebody in accounting would notice days later, fix it by hand, and move on — until it happened again on a different order, then on three at once.

That is the kind of problem that never gets a ticket. It looks like noise. It is not noise.

What it actually cost

• Revenue that shipped but didn’t bill. Goods left the building and no invoice was created. Every day an order sits on hold is a day the receivable doesn’t exist, which means a day it can’t be collected, borrowed against, or reported.
• Customers seeing several pending charges. Each reauthorization places a hold on the customer’s card. A cardholder who sees five pending holds for the same order does not think “integration defect.” They call, or they call their bank.
• Staff time spent hand-fixing the same failure. Every fix was a person reading a hold, finding the live authorization, correcting the order and rerunning billing — skilled time spent on a defect nobody had named.
• The slow erosion of trust in the system. Once people expect billing to miss some orders, they start checking all of them. That is the most expensive cost and it never shows up on a report.

A failure that happens to one order in twenty is easy to ignore and impossible to fix by watching.

How we tracked it down

The temptation was to blame the bundle. It was third-party code, it wrote the bad value onto the order, and it was the last thing to touch the record. We didn’t touch it. Instead we built a timeline from what NetSuite already records.

1. The authorization history, as data

The bundle stores every authorization event as a custom record linked to its sales order. Queried directly with SuiteQL, that history becomes a table you can count: for each affected order, how many authorizations, how far apart, and which ones the gateway rejected.

SELECT ae.order_ref, COUNT(*) AS attempts,
       MIN(ae.created) AS first_try, MAX(ae.created) AS last_try
FROM   auth_event ae
WHERE  ae.created >= SYSDATE - 30
GROUP  BY ae.order_ref
HAVING COUNT(*) > 1
ORDER  BY attempts DESC

The pattern was immediate. Orders that failed didn’t have one bad authorization. They had several good ones, about thirty seconds apart, followed by one the gateway refused as a duplicate. The refused one was the one written onto the order.

2. Our own audit log

Months earlier we had added a small log record to the script that flags orders for reauthorization: every flag it sets, with the amount and the time. That log proved the flag was set once per order. So whatever was multiplying the attempts happened downstream of us.

3. The deployment and its input

The scheduled job reads its work from a saved search, configured on the deployment. We opened that search. Its criteria were right: one row per sales order, main line only. Its results columns included one more field — an authorization status, joined through the child authorization records.

4. Counting, not guessing

The last step was to line the theory up against every case. For each failed order, the number of authorization records on file at run time against the number of attempts the job made:

Every order matched. And it explained the part that had made the problem look random: new orders have one authorization record, so they worked. Orders that had been around long enough to be renewed a few times had several, so they failed. The defect got worse with age, which is exactly why it seemed to come and go.

The cause

A joined column on a child record changes the shape of a search. One sales order with five authorization records becomes five result rows. The job did exactly what it was built to do — one void and one new authorization per row — five times, until the gateway’s duplicate protection refused the last one. Then the bundle recorded the refused transaction as the order’s current authorization, and billing failed against it.

The fix, and what we added after

The fix took five minutes and no code: remove the joined column. We confirmed nothing else was flagged, released the stuck orders by hand, and watched the next split shipment get exactly one authorization.

Then we made the next failure louder than this one had been:

• Idempotent flagging. The script that flags an order skips it if it is already flagged for the same amount, so a double save can’t queue two reauthorizations.
• Read-back after save. The script reads the flag back after setting it, instead of assuming the save did what it was told.
• A stale-pointer check. A small scheduled script runs just after the reauthorization job and before each billing run, compares each order’s stored transaction against its live authorization, and corrects the pointer before billing ever sees it.

Why this is a consulting problem, not a coding problem

The code change here was deleting a column. The work was everything before it: refusing to blame the vendor without evidence, using the data NetSuite already keeps to build a timeline, and testing the theory against every case rather than the one that was easiest to explain. Most of the expensive problems in an ERP look like this. Somebody built something reasonable, somebody else changed something small, and the two interact in a way nobody owns.

The fastest path is rarely the obvious suspect. It’s whoever is willing to count.

Related: is the error you’re chasing the cause, or just a symptom? and reading a NetSuite stack trace.