Suggestions
Sandy Ryza
Lead Engineer - Dagster
Sandy Ryza is a highly accomplished individual with a strong background in computer science and data engineering.
Sandy is well-known for his contributions to the open-source community as the Lead Engineer on the Dagster project.
He is also recognized for co-writing the book 'Advanced Analytics with Spark' and for his active involvement as a committer in both the Apache Spark and Apache Hadoop projects.
His education includes a Bachelor of Science in Computer Science from Brown University, where he graduated Cum Laude with Honors. Prior to that, he completed his High School Diploma at Crossroads School.
Throughout his career, Sandy has held various key positions in reputable organizations. He was the Lead Engineer at Dagster Labs, ML Engineering Manager at KeepTruckin, Senior Product Manager at Remix, and held senior roles at Clover Health and Cloudera.
Additionally, Sandy has a diverse professional background having experience as a Data Scientist, Software Engineer, and Engineering Manager.
He has also gained valuable insights from his internships at Cloudera and Microsoft, showcasing his dedication to continuous learning and growth.
Sandy Ryza continues to be a prominent figure in the tech industry, leveraging his expertise in data science, engineering, and product management to drive innovation and excellence.
Highlights
"The best way to improve your machine learning model is to get better data for that model"
Had a great chat with Jeff Mains on the SaaS Fuel podcast about data pipelines, AI, and orchestration.
Earlier this week I advocated for monolithic data platforms, but they come with a big challenge: they mean many teams and use cases in close contact. Close contact means it's easy for these teams to trample each other.
Trampling takes many forms:
-
Starvation: a data pipeline uses up resources needed by another pipeline. E.g. a low-priority experimental pipeline delays a business-critical report by occupying all available slots on the cluster.
-
Monolithic dependencies: library version choices in one pipeline restrict options elsewhere. E.g. an ancient pipeline depends on an old version of Pandas, so nobody else is able to use more recent Pandas versions and the new features that come with them.
-
Coupled failures: an error in one pipeline takes other pipelines down with it. E.g. a data engineer accidentally pushes a new pipeline with an invalid cron-schedule, and then every other pipeline in the platform can't load.
-
Clutter: another team's assets or pipelines make it hard to find what you're looking for. E.g. a massive dbt project is onboarded onto the platform, and whenever you search in the catalog the results you're looking for are buried under scores of "int_" tables.
The takeaway for data infra tools is that good fences make good neighbors. I.e. it requires designing for multi-tenancy & isolation all the way through.
In Dagster, this is extra challenging, because we're in the business of running user code, which is generally what's responsible for doing the trampling. And we don't just run pipelines, we also run user-defined event-based logic that decides when those pipelines should run.
To credibly support monolithic data platforms while dealing with this, Dagster has needed to develop deep isolation abstractions, like code locations, which isolate both dependencies and scheduling logic evaluation.
