Home
Out-of-the-box ETL, easy to learn, and a pleasure to use!
What is Meerschaum?¶
Meerschaum turns queries and Python functions into named data streams called Pipes. Register a source, sync it, and query the results from your database, Python, or the web console.
Why Meerschaum?¶
Incremental updates. Fetch the new and changed rows. Meerschaum handles deduplication, upserts, schema changes, and scheduling.
Group related Pipes into a Pipeline to run transformations in dependency order and keep their history.
Genuine CLI recording from 5.0.0rc16. Playback pauses are extended for reading.
Features¶
Organize ETL into Pipes¶
Tag streams, inspect their data, and reuse them in transformations.
Ingest from any source¶
A Python plugin can return rows from a file, service, or device. A Rust plugin adds native actions, sources, routes, and more.
Keep projects reproducible¶
Meerschaum Compose keeps Pipe definitions, jobs, and configuration beside your code.
Transform in SQL¶
When source and destination share a SQL instance, transformations can run in place. Larger transfers use chunks and concurrent workers.
Use it from your code¶
import meerschaum as mrsm
pipe = mrsm.Pipe(
'plugin:manual', 'readings',
instance='sql:local',
columns={'id': 'id'},
)
result = pipe.sync([{'id': 1, 'temperature': 21.5}])
print(result.success, result.inserted)
print(pipe.get_data(as_docs=True))
Start with the Python or Rust guides, then follow the full API references.
Run and observe¶
Give work a schedule, manage it in the console, and chart sync metrics in Grafana. Teams, permissions, and locks control shared access.
Video Tutorials¶
Support the Project¶
Meerschaum is open source. Help with an issue or contribution, or support development through GitHub Sponsors.