Last quarter, I was on a call with a data lead at a Series E fintech. His team had 4,000+ data pipelines running in Airflow. I asked him how many of those were still serving active dashboards. Long pause. "At least half of them," he said. "
Netflix runs one of the most complex data ecosystems on the planet. Hundreds of internal teams, dozens of purpose-built datastores, petabytes of data flowing between systems daily. Somewhere in that ecosystem, an engineer needs to move data from system A to system B. Should be straightforward, right? It wasn&
Zepto is an Indian quick-commerce company that promises 10-minute grocery delivery. That promise depends on data. Demand forecasting, inventory optimization, rider routing, and dozens of daily dashboards for business teams all run on top of their data infrastructure. At the scale they operate (millions of orders, 200+ TB
There's a pattern showing up across the data platform teams I pay attention to. Zepto built an internal DataPortal that routes Databricks jobs between Spark clusters and SQL warehouses based on workload metadata. Netflix built Data Bridge, a unified control plane that abstracts execution engines away from users
Last week I was talking to a Head of Data at a Series C fintech. She had nine people on her data team. Three of them, she told me, spent most of their week just keeping integrations alive between Fivetran, Airflow, dbt, and Looker. Nobody was building anything new. The
Here's a pattern I keep seeing. A team runs Snowflake for everything: batch transforms, ad-hoc analyst queries, dashboard refreshes, and that one product analytics job that scans 2 billion clickstream rows every hour. Their monthly bill keeps climbing. Query performance gets worse as workloads compete for the
3 Months back I was catching up with a CDO at a NASDAQ listed consumer tech company. Their cloud data platform bill had tripled in eight months. Not because data volume tripled. Because every team, analytics, data science, product, marketing, was running every query they could think of through the