Sign In

Eric Sun

Data Expert with System Insights

Eric Sun is a Senior Manager of Data Platform at Coinbase, based in the San Francisco Bay Area. He has a strong background in data engineering and analytics, with a focus on leveraging data to drive business decisions and product improvements. His role at Coinbase involves overseeing data infrastructure and ensuring the efficient processing and utilization of data across the organization.

In addition to his position at Coinbase, Eric has a diverse professional history that includes various roles in data management and engineering. His experience spans multiple companies and projects, emphasizing his expertise in the tech industry. He is also active on LinkedIn, where he has built a substantial network of connections.

For more detailed insights into his professional journey, you can view his LinkedIn profile .1

Highlights

Jul 3 · linkedin

https://lnkd.in/gjURFX8K

Secondary Indices and Materialized Views are the async by-products of the tables in lakehouse. ClickHouse, Materialize, RisingWave, Lance, and StarRocks have serious support for various types of indices to accelerate queries already.

The faster lakehouse requires optimized storage layout + unique index to accelerate Merge-On-Write and query acceleration.

Not aware of any cloud database is using in-memory (or local block storage) to deduplicate and Merge-On-Read with committed Iceberg/Delta files. If you know any company or project in this kind, please kindly leave a comment here.

#lakehouse #partition #index #lsm #bucket #merge #deduplicate #paimon #starrocks #puffin #hyperspace #compact #kafka

Jul 2 · linkedin

https://lnkd.in/gQ6RzjiJ

Exciting news! Jianlong Zhong (Jay) will be showcasing "AirAgent" at 12PM PT on Thursday, 9/12, 2024, during the "Airflow Summit 2024" in San Francisco.

Airflow depends on continuous scanning files on disk and parsing DAG definitions in Python codes, this design pattern does not go well with the typical Blue/Green deployment mode for microservices (especially, the disruption to the long-running stateful jobs/tasks). To effectively make Airflow deployment "continuously", Jay came up with the design pattern of a deployer agent which takes care of 3 major functions:

[1] fetch a specific DAG tarball (usually the latest) commit from github as well as the new Docker image (of runtime Python dependencies) if image is modified [2] point the symbolic link to the new DAG directory on the EFS/NFS [3] restart the worker and scheduler one-by-one accordingly

AirAgent is currently deploying updates hundreds of times a day on both staging and production environments. It has been a game changer in terms of Airflow development lifecycle and developer velocity at Coinbase.

Missed Jay's captivating talk last year in Toronto? Check it out here: https://lnkd.in/gBC_udTg.

#airflow #deployment #airagent #cicd #velocity

Related Questions

What is Eric Sun's role at Coinbase?
How long has Eric Sun been at Coinbase?
What are Eric Sun's main responsibilities at Coinbase?
What projects has Eric Sun led at Coinbase?
What is Eric Sun's background before joining Coinbase?
Eric Sun
Eric Sun, photo 1
Eric Sun, photo 2
Add to my network

Location

San Francisco Bay Area