🌳 Environment Variables¶

You can manage the behavior of Meerschaum and emulate multiple installations with environment variables.

The command mrsm show environment will print out your current set MRSM_ variables. For example, running this command in the Web Console reveals how the configuration is managed to subprocesses by the web server.

Summary¶

Nearly all Meerschaum environment variables begin with the prefix MRSM_. The most commonly used variables are summarized below; each is described in detail later on this page.

MRSM_ROOT_DIR¶

By default, your root Meerschaum directory is located in ~/.config/meerschaum (Windows: %APPDATA%\Meerschaum). This is similar to how your base pip environment behaves: with pip, you isolate packages in virtual environments, and with Meerschaum, you can create an isolated environment by specifying an alternative root directory. This will recreate your configuration files and virtual environments as if you were running a clean installation.

mkdir foo
MRSM_ROOT_DIR=foo \
  mrsm show environment

MRSM_CONFIG_DIR¶

Not to be confused with MRSM_CONFIG, setting MRSM_CONFIG_DIR allows you to designate a separate directory outside of the root to isolate your secrets and segment your configuration. If unset, the default directory is $MRSM_ROOT_DIR/config.

MRSM_PLUGINS_DIR¶

Like MRSM_ROOT_DIR, MRSM_PLUGINS_DIR lets you isolate your plugins, e.g. if you wanted to manage your plugins in a version control system like git.

mkdir plugins
touch plugins/example.py

MRSM_PLUGINS_DIR=plugins \
  mrsm show plugins

Multiple Plugins Directories¶

To allow you to group plugins together, Meerschaum supports loading multiple plugins directories at once. Simply separate the paths with a colon like you would for $PATH:

export MRSM_PLUGINS_DIR='./plugins:/another/plugins/path'

You could also set MRSM_PLUGINS_DIR to a JSON-encoded list of paths:

export MRSM_PLUGINS_DIR='[
    "./plugins",
    "/another/plugins/path"
]'

MRSM_VENVS_DIR¶

Like MRSM_PLUGINS_DIR, you can designate a separate directory outside of the Meerschaum root to contain virtual environments. This is useful for sharing virtual environments between deployments as well as separating package data from user data (e.g. Kubernetes deployments).

MRSM_VENVS_DIR='venvs/' mrsm show plugins

MRSM_WORK_DIR¶

MRSM_WORK_DIR designates the base working directory under which the root directory is located when MRSM_ROOT_DIR is not explicitly set. This is primarily useful for containerized deployments where the process is launched from a known location.

MRSM_<TYPE>_<LABEL>¶

You can temporarily register new connectors in a variable in the form MRSM_<TYPE>_<LABEL>, where <TYPE> is the connector type, and <LABEL> is the label for the connector (converted to lower case). Check here for more information about environment connectors, but in a nutshell, set the variable to the URI of your connector.

MRSM_SQL_FOO=sqlite:////tmp/temp.db \
MRSM_API_BAR=http://user:pass@localhost:8000 \
  mrsm show connectors

Common examples are the default connectors sql:main and api:main:

MRSM_SQL_MAIN=postgresql://user:pass@localhost:5432/db \
MRSM_API_MAIN=https://user:pass@api.example.com \
  mrsm show connectors

You may also set the variable to a JSON object of connector attributes instead of a URI:

MRSM_SQL_FOO='{"flavor": "sqlite", "database": "/tmp/temp.db"}' \
  mrsm show connectors

MRSM_CONNECT_TIMEOUT¶

A SQL, API or Valkey connector fails fast when its server cannot be reached. The first connection a command makes reports a refused connection, an unknown host or rejected credentials at once (sql:main: cannot connect to localhost:5432 (connection refused). Is the database running?), and gives up on a host that does not answer after connect_timeout seconds (5 by default). Set the connect_timeout attribute on a connector to change it for that connector, or MRSM_CONNECT_TIMEOUT to change it for all of them:

MRSM_CONNECT_TIMEOUT=2 mrsm show pipes -i sql:main

A long-running job that loses a connection it had already established keeps retrying as before.

MRSM_CONFIG¶

You may patch your existing configuration with MRSM_CONFIG. Simply set the variable to JSON or a simplified dictionary. Remember you may symlink to other keys in your configuration (see MRSM{} config symlinks below).

MRSM_CONFIG='{"foo": {"bar": 123}}' \
  mrsm show config foo

MRSM_CONFIG='foo:123' \
  mrsm show config foo

MRSM_CONFIG='foo:MRSM{meerschaum:connectors:sql:main:password}' \
  mrsm show config foo

MRSM_PATCH¶

The variable MRSM_PATCH behaves the same way as MRSM_CONFIG. The difference is that MRSM_PATCH is applied to your configuration after MRSM_CONFIG, so you may symlink to keys defined in MRSM_CONFIG.

MRSM_CONFIG='{"foo": {"bar": 123}}' \
MRSM_PATCH='baz:MRSM{foo:bar}' \
  mrsm show config baz

MRSM_RUNTIME¶

MRSM_RUNTIME marks a special runtime context. For example, MRSM_RUNTIME=docker is set inside the official Docker image. You typically do not need to set this yourself.

MRSM_NOASK / MRSM_NONINTERACTIVE¶

Set either of these to disable interactive prompts. When set, Meerschaum assumes default answers (equivalent to passing --noask / --yes), which is useful for automation and CI pipelines.

MRSM_NOASK=1 mrsm bootstrap pipes

MRSM_ALLOW_PROTECTED¶

Connectors marked protected: true refuse every change (see Protected Connectors). Set MRSM_ALLOW_PROTECTED=1 to allow writes for a process and the processes it starts, for example a server whose instance is protected:

MRSM_ALLOW_PROTECTED=1 mrsm start api

For one command, --allow-protected does the same. -y / --yes never does.

MRSM_DATAFRAME¶

Every Python API that returns a DataFrame (pipe.get_data(), conn.read(), pipe.fetch(), Plan.to_dataframe(), the frames handed to plugin code) returns the library of dataframe in meerschaum.yaml: polars for a new root, pandas for a root created before 5.0.0rc15 (see DataFrame library). MRSM_DATAFRAME overrides it for a process, the jobs it starts and the Python workers it runs plugins in:

MRSM_DATAFRAME=pandas python my_report.py

as_polars=True / as_pandas=True on a call override both. Any other value is an error that names the variable.

MRSM_NO_AUTO_INSTALL¶

Meerschaum normally installs a missing optional dependency when a feature first imports it. Set MRSM_NO_AUTO_INSTALL=1 for locked-down, offline, or reproducible environments. Missing packages will be reported instead of downloaded; otherwise Meerschaum warns before the first automatic download in each process. Install the dependency group before starting Meerschaum.

The variable covers every implicit installation: attempt_import(), the dependency version checks (which never install a version parser), creating a virtual environment (no uv venv --seed), the API host, and the DuckDB library download (see MRSM_DUCKDB_LIB), including DuckDB's own extension auto-install. Installs you ask for, such as mrsm install packages or pip_install() from your own code, still run, but never bootstrap uv, pip or wheel for themselves.

MRSM_NO_AUTO_INSTALL=1 mrsm show pipes

MRSM_DUCKDB_LIB¶

Python hosts with meerschaum[sql-duckdb] installed and this variable unset share the public Python DuckDB driver between Rust Pipe operations and SQLAlchemy. An explicit MRSM_DUCKDB_LIB selects the native C API instead. SQLAlchemy access is refused in that process to prevent separate database instances from losing writes. Unset the variable and restart the interpreter to use both interfaces. Restart after installing the extra if the process already selected the native driver.

The standalone binary and Python hosts using the native C API load DuckDB's own shared library (libduckdb.so, libduckdb.dylib or duckdb.dll, version 1.4 or newer) at run time. Meerschaum looks for it in this order:

1. MRSM_DUCKDB_LIB, a path to the file (for air-gapped hosts; a library that fails to load is an error);
2. the cache under <root>/drivers/;
3. the system library path;
4. a checksum-verified download of the official release, unless MRSM_NO_AUTO_INSTALL is set.

MRSM_DUCKDB_LIB=/opt/duckdb/libduckdb.so mrsm show drivers

Run mrsm show drivers to see where the library comes from, and mrsm install driver duckdb / mrsm uninstall driver duckdb to manage the cached copy.

The native C API path refuses per-connection DuckDB options that it cannot honor, including access_mode=READ_ONLY. Use the Python sql-duckdb extra for connections with these settings.

MRSM_BIN¶

By default the mrsm and meerschaum console scripts of the Python package run the CLI inside the installed extension. Set MRSM_BIN to the path of a standalone mrsm binary (for example one built by cargo install meerschaum) to run that program instead. The binary must report the same --version as the installed package. Everything Meerschaum re-launches itself for (jobs, compose, pipeline steps) uses the same program.

MRSM_PYTHON¶

The standalone mrsm binary runs Python plugins in a worker process. MRSM_PYTHON names the Python interpreter for those workers, for package installs, and for mrsm upgrade meerschaum (which otherwise refuses to guess which environment to upgrade). Inside the Python package the interpreter that is already running is used and you do not need to set it.

Before the first worker starts, the binary checks the interpreter once (python -P -c ..., 10 s limit): it must import a meerschaum of the same major version. A missing, stale (v4) or mismatched package, an interpreter that cannot start, and a worker that never connects each fail at once with one message that names the interpreter, what it found and the fix (pip install meerschaum==<version> or MRSM_PYTHON); the refusal is remembered for a few seconds, so many pipes do not each wait. A worker has MRSM_WORKER_STARTUP_TIMEOUT seconds (default 60, for connecting and the hello together) before it is killed; raise it for a legitimately slow cold start.

Internal Variables¶

The following variables are set automatically by Meerschaum (e.g. for background jobs, the systemd executor, and the API server) and are documented here for completeness. You generally should not set them by hand.

MRSM{} Config Symlinks¶

Anywhere a configuration value (or pipe parameter) is read, you may reference another key in your configuration with the MRSM{key1:key2:key3} syntax. The reference is resolved to the target config value at access time, so a single source of truth (for example, a database password) can be reused across many keys.

The path inside the braces is a colon-separated sequence of config keys, equivalent to the arguments you would pass to get_config(). For example, MRSM{meerschaum:connectors:sql:main:password} resolves to config.meerschaum.connectors.sql.main.password.

Symlinks are resolved by search_and_substitute_config() in meerschaum/config/_read_config.py (exported from meerschaum.config).

In configuration¶

# Reuse the SQL main password as a top-level config value.
MRSM_CONFIG='foo:MRSM{meerschaum:connectors:sql:main:password}' \
  mrsm show config foo

In a connector¶

# config/connectors.yaml
sql:
  reporting:
    flavor: postgresql
    host: localhost
    database: reporting
    username: MRSM{meerschaum:connectors:sql:main:username}
    password: MRSM{meerschaum:connectors:sql:main:password}

Now sql:reporting shares credentials with sql:main, change them in one place.

In a pipe parameter¶

import meerschaum as mrsm

pipe = mrsm.Pipe('demo', 'cfg', parameters={
    'username': 'MRSM{meerschaum:connectors:sql:main:username}',
})
print(pipe.parameters['username'])  # resolved to the configured username at access time