Every quarter I watch the same pattern repeat across pension and insurance shops in Istanbul. The BES pipeline runs green in dev. Unit tests pass. Sample XMLs validate against the schema. Someone signs off. Then the submission window opens, the real dataset flows through, and EGM returns a rejection code nobody has seen since the last regulation change. Now it is 22:00, the deadline is tomorrow, and three people are reverse-engineering a validation rule from a PDF.
This is not bad luck. It is a pipeline that was never designed to fail before the deadline made failure expensive.
What "tested" usually means in a BES pipeline
When a team tells me their regulatory reporting is tested, what they almost always mean is:
- Unit tests on transformation logic (contribution allocation, fund switch handling, vesting)
- Schema validation against a handful of sample XMLs
- Reconciliation totals matching the source ledger
- A manual eyeball on a sample participant
All useful. None of them simulate a submission. A submission is not a file — it is a conversation with a regulator's system that has its own state, its own idea of what you sent last month, and its own rejection semantics. HAYMER does not care that your unit tests pass. It cares whether the TCKN you are reporting today matches the identity it already has on file, whether the fund code exists in the version of the reference table it loaded this morning, and whether your cumulative contribution figures reconcile against what you told it 30 days ago.
None of that is testable with sample data.
Why sample data lies
Sample data is curated. It represents the cases the developer thought about. Production data represents the cases nobody thought about:
- Participants who switched employers mid-cycle and now appear under two sponsor codes
- Fund transfers initiated on day T but settled on T+2, straddling the reporting boundary
- Manual corrections entered by operations that bypass the standard event stream
- Participants whose identity was updated via a MERNIS refresh three days before submission
- Contributions posted in a currency the reference table does not know about because someone loaded a new fund without updating the map
A sample of 200 records will not surface any of this. A production snapshot of 400,000 records will surface all of it, every single cycle, in different combinations. If your pipeline has never processed the actual production volume against the actual regulator response format, you do not have a tested pipeline. You have a hypothesis.
The dry run as a first-class deliverable
The firms that submit cleanly are not smarter. They treat the dry run as a mandatory stage, not an optional QA activity. Concretely:
- Production-equivalent snapshot: A masked but structurally complete copy of the actual submission dataset, refreshed on a known cadence. Not a sample. Not synthetic. The real shape, the real volume, the real edge cases.
- A simulator or sandbox endpoint: EGM and similar bodies publish test endpoints for a reason. Use them. If none exists for a specific report, build an internal mock that enforces the same rejection codes based on the published rulebook.
- End-to-end execution against the sandbox at least 5 business days before the window opens: Not the schema validator. The actual submit-receive-acknowledge loop, including the rejection parsing path.
- A rejection replay harness: Every historical rejection code should live in a test suite that runs against every pipeline build. If HAYMER rejected you with code X in March, code X should be a permanent fixture in your regression pack.
The organizational reason this does not happen
The technical work is not hard. The reason most teams skip it is that dry runs surface problems, and surfaced problems require someone to own them. If the dry run reveals 1,200 participants with reference data mismatches, someone in operations has to fix 1,200 records. If it reveals a logic gap, someone in engineering has to patch it. Both cost time that was not in the plan.
So the dry run gets deprioritized, the pipeline ships, and the problem migrates from "we have five days to fix this" to "we have five hours to fix this." The cost does not disappear. It gets concentrated into the worst possible window, with the worst possible visibility, in front of the worst possible audience.
What to actually build
If you are responsible for a BES reporting pipeline and you cannot answer yes to all of these, you have a dry run problem:
- Can you, on demand, run the full submission pipeline against last night's production snapshot?
- Does that run terminate in an actual acknowledgement from a sandbox or high-fidelity mock, not just a file on disk?
- Are all historical rejection codes encoded as regression tests?
- Is the dry run scheduled to run automatically N days before every submission window, with results reviewed by a named owner?
- When the dry run fails, is there a defined SLA for remediation before the real window opens?
The pipelines that survive regulatory reporting are not the ones with the cleanest code. They are the ones with the shortest feedback loop between "something is wrong" and "we know about it." A pipeline that cannot fail safely five days before the deadline will fail expensively five hours before it. That is a design choice, whether the team realizes they made it or not.