Every BES pipeline I've worked on eventually hits the same wall. The valuation engine is clean. The unit allocation logic is tested. The reconciliation with the custodian ties out to the kuruş. And then a fund manager restructures a share class, or the pricing vendor silently backfills a NAV two days late, or a corporate action gets applied on T+1 instead of T, and suddenly you're staring at a member statement that is technically correct according to your pipeline and materially wrong according to the regulator.
The uncomfortable truth: the pipeline owned the calculation, but it never owned the provenance of the inputs.
What "commodity data" actually hides
Fund NAVs from an external pricing vendor look like the most boring row in your warehouse. Fund code, date, price, currency. Four columns. How wrong can four columns go?
A lot, because every one of those columns carries undocumented assumptions from the provider:
- Which cut-off time does the price represent? Some vendors publish the fund company's official NAV. Others publish an indicative price snapped at their own cut-off. On a normal Tuesday these are identical. On a day the Borsa closes early, they are not.
- What happens on a non-business day? Some feeds carry-forward. Some publish null. Some publish the previous close but stamp it with today's date. Your pipeline treats all three the same way and produces three different member balances.
- How are corporate actions reflected? A fund merger, a share class split, a distribution — each vendor has a house convention. Sometimes the price on ex-date is adjusted, sometimes it isn't, sometimes it is adjusted but the adjustment factor is delivered in a separate file that nobody in your team is subscribed to.
- When was this price actually written? The row says 2024-03-15. The vendor loaded it at 2024-03-15 18:40. Then they revised it at 2024-03-16 09:12 after a fund company correction. Your pipeline picked up the 18:40 version, ran overnight, and allocated units against a price the vendor no longer considers valid.
None of this is in the vendor contract. None of it is in the data dictionary. You learn it by running a fund that gets restructured and spending three days reconstructing what your pipeline consumed versus what the provider now claims it published.
The failure mode is always the same
After enough production incidents, the pattern is monotonous. It goes like this:
- A member or auditor questions a balance.
- You go to the pipeline output. The calculation is correct given the inputs.
- You go to the input table. The price is there.
- You go to the vendor portal. The price is different.
- You ask the vendor when the price changed. They cannot tell you, because their own system overwrites in place.
At this point you own the problem. Not the vendor. You, because your pipeline settled a valuation on a price you can no longer prove existed.
The operations team wants an answer today. Legal wants to know if the member statement needs to be reissued. The regulator does not care that the vendor changed the number under you.
What actually works
There is no clever architecture that removes the dependency on external pricing. What works is treating the vendor feed as untrusted input, not as source of truth.
Immutable ingestion. Every file from every vendor lands in an append-only store, keyed by ingestion timestamp, not business date. If the vendor sends the same fund and same date twice with two different prices, both rows exist forever with the timestamps at which they arrived. Your valuation query joins to whichever version was current at the time the pipeline ran. This is the single most important discipline and the one that gets skipped because "we already have the price in the warehouse."
Bitemporal on the price table. Business date and knowledge date. The price of Fund X for 2024-03-15 as known on 2024-03-15 might be 12.3400. As known on 2024-03-16 it might be 12.3421. You need to be able to answer both questions six months later without archaeology.
Explicit provider metadata. Which vendor. Which file. Which line in that file. Which cut-off. Carry-forward flag. Corporate action adjustment flag. If the vendor doesn't send it, mark it null and know that it's null — do not let the pipeline pretend the assumption doesn't exist.
Cross-vendor reconciliation for anything material. For funds where the AUM exposure is significant, subscribe to two independent price sources and reconcile daily. When they disagree, the exception goes to a human before allocation runs, not after. Yes, this costs more. It costs less than reissuing member statements.
A price-of-record decision that is a business decision, not a technical one. Which vendor wins when they disagree? On which fund families? Under which conditions is a same-day override permitted, and who signs? If this is not written down, the answer at 08:15 on a Monday will be whoever picks up the phone.
The part nobody wants to hear
Most BES pipelines I have reviewed cannot answer the question "what price did we use, when did we receive it, and what version of the vendor's understanding did it reflect?" for a specific fund on a specific historical date. They can show the price that is currently in the warehouse. That is not the same question.
The gap between those two answers is where regulatory findings live. It is where member complaints that escalate live. It is where the operations team's overtime lives.
Owning the calculation is the easy part. Owning the provenance of every external input the calculation depended on — with the vendor's revisions, their undocumented conventions, and their silent overwrites all captured — is the part that separates a pipeline that works from a pipeline that survives an audit.
Treat the vendor feed as evidence, not as truth. Store it like evidence. Then the calculation you already trust becomes defensible, which is a different and more useful property than being correct.