Data Engineering for Investment Management: Pipelines for Market Data, IBOR, and Alternative Data

Investment management data engineering navigates source systems and data structures most enterprise data engineers haven't seen. Market data from Bloomberg, Refinitiv (LSEG), FactSet, S&P arrives as feeds with vendor-specific identifiers, time conventions, and corporate action handling that requires careful normalization. The OMS (Charles River, Aladdin, Eze) holds positions and orders in models specific to the platform. The fund accountant produces NAV files in formats that vary by administrator. Custodians provide holdings and transaction data through SWIFT messages, SFTP files, and APIs that all use different conventions. Corporate actions — splits, dividends, mergers, spin-offs — change historical position data in ways that require point-in-time correctness for valid backtesting. And research teams need point-in-time accurate data because lookahead bias destroys research validity. Generic enterprise data engineering doesn't address any of this.

Investment data engineering that works follows industry-specific patterns. Master security data with vendor identifier mapping (CUSIP, ISIN, Bloomberg Ticker, RIC, FIGI). Corporate action processing that maintains point-in-time correctness for backtesting. Position and transaction reconciliation across the OMS, custodian, and fund accountant. Market data normalization across vendors. Alternative data ingestion with quality monitoring and the schema evolution handling that vendor changes require. Bronze-silver-gold medallion with point-in-time gold tables for research. IBOR reconciliation as the trust foundation. Done with this discipline, the data platform supports trading, research, and operations. Done generically, it produces a system the investment teams don't trust.