Real-time pipelines are expensive. Most institutional research use cases do not need them.
When a team says "we need real-time data", the question worth asking is: what decision is being made between 9am and 4pm that cannot wait until a scheduled refresh? For most research, risk reporting and investment-committee use cases, the answer is "none". A daily scheduled job that lands, cleans and validates the previous day's market data by 8am is enough. Real-time adds cost, complexity and failure modes, and it often does not change any decision.
Real-time is justified when a decision genuinely depends on intra-day information — for example, risk limits that must be monitored during the trading day, or operations teams that need to know a feed has failed before market open. Those are legitimate use cases, and they should be built deliberately. The error we see is teams starting from "real-time" as a default and then justifying it after the fact.
Start with a well-built batch pipeline. If, six months in, a specific intra-day decision turns out to be blocked by the daily refresh, add real-time for that specific path. This gives you most of the reliability of batch with the option to move to real-time only where it is needed.