Why your data pipelines collapse at scale (and nobody warns you)

Last quarter, I was on a call with a data lead at a Series E fintech. His team had 4,000+ data pipelines running in Airflow. I asked him how many of those were still serving active dashboards. Long pause. "At least half of them," he said. "

How Netflix's Data Bridge orchestrates 300,000 data pipeline jobs per week

Netflix runs one of the most complex data ecosystems on the planet. Hundreds of internal teams, dozens of purpose-built datastores, petabytes of data flowing between systems daily. Somewhere in that ecosystem, an engineer needs to move data from system A to system B. Should be straightforward, right? It wasn&

How Zepto's DataPortal cut Databricks costs by routing 1,000+ jobs to the right compute

Zepto is an Indian quick-commerce company that promises 10-minute grocery delivery. That promise depends on data. Demand forecasting, inventory optimization, rider routing, and dozens of daily dashboards for business teams all run on top of their data infrastructure. At the scale they operate (millions of orders, 200+ TB

Why cost-aware compute routing is the next feature your data platform needs

There's a pattern showing up across the data platform teams I pay attention to. Zepto built an internal DataPortal that routes Databricks jobs between Spark clusters and SQL warehouses based on workload metadata. Netflix built Data Bridge, a unified control plane that abstracts execution engines away from users

Why data pipeline fragmentation is killing your team (and what to do about it)

Last week I was talking to a Head of Data at a Series C fintech. She had nine people on her data team. Three of them, she told me, spent most of their week just keeping integrations alive between Fivetran, Airflow, dbt, and Looker. Nobody was building anything new. The

Why your data team needs more than one compute engine

Here's a pattern I keep seeing. A team runs Snowflake for everything: batch transforms, ad-hoc analyst queries, dashboard refreshes, and that one product analytics job that scans 2 billion clickstream rows every hour. Their monthly bill keeps climbing. Query performance gets worse as workloads compete for the

Why your "one platform" data strategy is costing you more than you think

3 Months back I was catching up with a CDO at a NASDAQ listed consumer tech company. Their cloud data platform bill had tripled in eight months. Not because data volume tripled. Because every team, analytics, data science, product, marketing, was running every query they could think of through the