Meet the Speakers: Bennett Meares
The Wonderful World of Incremental Time-Series ETL
Bennett Meares
Data Engineer at Stratifyd, Inc
Social: [ @BennettMeares, meerschaum.io, LinkedIn, Sessionize ]
Bennett Meares is a senior data engineer at Stratifyd, Inc. He began his data engineering career by building Clemson University's ETL system for facilities data (CEVAC) and graduated with an MS in Computer Science in 2021, for which he pioneered time-series synchronization strategies in his thesis.
Today, he is the author and maintainer of the open source ETL framework Meerschaum.
Intermediate // The Wonderful World of Incremental Time-Series ETL
Category: [ Development, Other ]
Languages: [ Python, SQL ]
Framework: [ Meerschaum, PostgreSQL, TimescaleDB ]
ETL is a simple process until it isn't. When you're dealing with billions of rows, how can you choose the synchronization strategy that best fits your needs?
In this talk, we dive down the rabbit hole of incremental time-series ETL. We consider the trade-offs with each strategy and how to choose the right one for you. Are your priorities perfect accuracy? Bandwidth? Run-time? Or something in between?
To put these strategies to the test, we will build several time-series ETL streams using the open source framework Meerschaum. We'll dive into a new Meerschaum Compose project and write several plugins to ingest outside data sources. Lastly, we'll wrap things up with a secure Docker image for easy deployment that Cloud Ops will love.
From beginners to experts, there's something here for everyone!