🔌 Connectors

Meerschaum coordinates everything via connectors. Under the hood, Connectors are collections of attributes (e.g. username, host, uri, etc.) that allow Meerschaum to send and retrieve data by implementing Meerschaum's interface to another protocol. Use cases for Connectors include fetching and inserting data when syncing pipes.

A connector is identified by two keys: its type and label, separated by a colon (:). For example, the connector sql:main refers to a connector with the type sql and the label main.

The command mrsm show connectors will print out your defined connectors and their attributes.

🗃️ Instances and Repositories

The terms instance and repository connectors refer to subclasses of standard Meerschaum connectors. When connecting to a Meerschaum instance, you use a standard sql or api connector, which expects to be able to access internal Meerschaum methods, such as retrieving users' and pipes' metadata.

Info

Not all sql connections are instance connectors, but all api connectors are.

Repository connectors are a subset of instance connectors and may only be api connectors. Consider the Venn diagram to the right to vizualize the different classes of connectors.

🌳 Environment Connectors

One handy way to temporarily register a connector is by setting an environment variable MRSM_<TYPE>_<LABEL> to the connector's URI. For example, the following environment variable would define the connector sql:foo:

MRSM_SQL_FOO=sqlite:////tmp/foo.db

Create your own custom connectors with the @make_connector decorator:

from meerschaum.connectors import make_connector, Connector

@make_connector
class DogConnector(Connector):
    IS_INSTANCE = False
    REQUIRED_ATTRIBUTES = ['username', 'password']

In the connector's environment variable, define the attributes as JSON:

export MRSM_DOG_SPOT='{"username": "foo", "password": "bar"}'

Did you know?

You can reference your Meerschaum configuration in environment connectors, like you can do with MRSM_CONFIG:

MRSM_SQL_FOO=postgresql://user:MRSM{meerschaum:connectors:sql:main:password}@localhost:5432/db

✅ Creating a Connector

To create a new connector (or redefine an existing one), run the command bootstrap connector and answer the following prompts. The new connector will be added to your configuration file (which may be accessed with edit config).

🎦 Watch an example

❌ Deleting a Connector

To delete a connector, run the command delete connectors with the -c connector keys flag:

delete connectors -c sql:myremoteconnector -y

🤔 Types

Connectors give you options

A connector's type determines the protocol it uses and its required attributes. Different types of connectors are capable of different tasks and have varying levels of flexibility and performance.

Type	Pros	Cons	Use Cases
sql	- Fast transfer rates - May be instance connector	- Typically network restricted for security	- Internal use behind a firewall - Connect API instance to a database - Large data transfers (>100,000 rows)
api	- Low resource requirements - APIs may be chained together - May be instance / repository connector	- Slower than direct database connection	- Endpoint for or deploy on IoT devices - Expose SQLite databases - Connect to a database instance that's behind firewall - Chaining together API instances
plugin	- Allows developers to ingest any data source	- Usually for one specific data source - May not be an instance connector	- Ingesting data from other APIs - Integrating legacy systems into Meerschaum
mqtt	- Subscribe to MQTT topics	- Meerschaum shell must be running to receive data - May not be an instance connector	- Ingesting data from existing IoT devices