🎼 Meerschaum Compose¶
compose plugin does the same for Meerschaum as Docker Compose does for Docker: with Meerschaum Compose, you can consolidate everything into a single YAML file ― that includes all of the pipes and configuration needed for your project!
mrsm compose up, you can stand up syncing jobs for your pipes defined in the Compose project ― one job per instance. Because the configuration is contained in the YAML file (e.g. custom connectors), Compose projects are useful for prototyping, collaboration, and consistency.
Multiple Compose Files
For complicated projects, a common pattern is to include multiple Compose files and run them with
1 2 3
This pattern allows multiple projects to cleanly share root and plugins directories.
Example Compose File
This compose project demonstrates how to sync two pipes to a new database
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
Want to skip the setup and work in a pre-configured environment? Create a new repository from the Meerschaum Compose Project Template.
compose plugin from the public repository
From a new directory, create a file
mrsm-compose.yaml. You can paste the example file above to get started.
1 2 3
You may set multiple paths for
plugins_dir. This is very useful if you want to group plugins together. A value of
null will include the environment's plugins in your project.
1 2 3
If you've used
docker-compose, you'll catch on to Meerschaum Compose pretty quickly. Here's a quick summary:
||Update and sync the pipes defined in the compose file.||
||Bring up the syncing jobs (process per instance)||
||Take down the syncing jobs.||
||Follow the jobs' logs.||
||Show the running status of background jobs.|
For our example project
awesome-sauce, let's bring up the syncing jobs:
All other commands are executed as regular actions from within the project environment.
1 2 3 4 5 6 7 8 9 10 11 12 13
compose plugin adds a few new custom flags. You can quickly view the available flags with
mrsm -h or
mrsm show help.
||Follow the logs when running
||Delete pipes when running
||Specify an alternate compose file (default: mrsm-compose.yaml).||
||Specify an alternate environment file (default: .env).||
The supported top-level keys in a Compose file are the following:
Govern the behavior of the syncing process. See The
syncKey section below for further details.
The tag given to all pipes in this project. Defaults to the current directory. If you're using multiple compose files, make sure each file has a unique project name.
A path to the root directory; see
Either a path or list of paths to the plugins directories. A value of
nullwill include the current environment plugins directories in the project. See
A list of plugins expected to be in the plugins directory. Missing plugins will be installed from
To install from a custom repository, append
@api:<label>to the plugins' names or set the configuration variable
Configuration keys to be patched on top of your host configuration, see
MRSM_CONFIG. Custom connectors should be defined here.
Additional environment variables to pass to subprocesses.
Accessing the host configuration
The Meerschaum Compose YAML file also supports Meerschaum symlinks. For example, to alias a new connector
sql:foo to your host's
1 2 3 4
Keys under the root key
sync are the following:
pipeskey contains a list of keyword arguments to build
instance(optional, defaults to
Define a regular interval for syncing processes by setting a schedule.
This corresponds to the flag
If a schedule is not set, pipes will be constantly synced and sleep
This corresponds to the flags
If this value is set, long-running syncing processes which exceed this value will be terminated.
This corresponds to the flag
This value may be a string or list of command-line arguments to append to the syncing command.
This option is available for specific edge case scenarios, such as when working with custom flags or specific intervals (i.e.