🔌 Connectors¶
Meerschaum coordinates everything via connectors. Under the hood, Connectors are collections of attributes (e.g. username, host, uri, etc.) that allow Meerschaum to send and retrieve data by implementing Meerschaum's interface to another protocol. Use cases for Connectors include fetching and inserting data when syncing pipes.
A connector is identified by two keys: its type and label, separated by a colon (:
). For example, the connector sql:main
refers to a connector with the type sql
and the label main
.
The command mrsm show connectors
will print out your defined connectors and their attributes.
🗃️ Instances and Repositories¶
The terms instance and repository connectors refer to subclasses of standard Meerschaum connectors. When connecting to a Meerschaum instance, you use a standard sql
or api
connector, which expects to be able to access internal Meerschaum methods, such as retrieving users' and pipes' metadata.
Info
Not all sql
connections are instance connectors, but all api
connectors are.
Repository connectors are a subset of instance connectors and may only be api
connectors. Consider the Venn diagram to the right to vizualize the different classes of connectors.
🌳 Environment Connectors¶
One handy way to temporarily register a connector is by setting an environment variable MRSM_<TYPE>_<LABEL>
to the connector's URI. For example, the following environment variable would define the connector sql:foo
:
1 |
|
Create your own custom connectors with the @make_connector
decorator:
1 2 3 4 5 6 |
|
In the connector's environment variable, define the attributes as JSON:
1 |
|
Did you know?
You can reference your Meerschaum configuration in environment connectors, like you can do with MRSM_CONFIG
:
1 |
|
✅ Creating a Connector¶
To create a new connector (or redefine an existing one), run the command bootstrap connector
and answer the following prompts. The new connector will be added to your configuration file (which may be accessed with edit config
).
🎦 Watch an example
❌ Deleting a Connector¶
To delete a connector, run the command delete connectors
with the -c
connector keys flag:
1 |
|
🤔 Types¶
Connectors give you options
A connector's type determines the protocol it uses and its required attributes. Different types of connectors are capable of different tasks and have varying levels of flexibility and performance.
Type | Pros | Cons | Use Cases |
---|---|---|---|
sql |
- Fast transfer rates - May be instance connector |
- Typically network restricted for security | - Internal use behind a firewall - Connect API instance to a database - Large data transfers (>100,000 rows) |
api |
- Low resource requirements - APIs may be chained together - May be instance / repository connector |
- Slower than direct database connection | - Endpoint for or deploy on IoT devices - Expose SQLite databases - Connect to a database instance that's behind firewall - Chaining together API instances |
plugin |
- Allows developers to ingest any data source | - Usually for one specific data source - May not be an instance connector |
- Ingesting data from other APIs - Integrating legacy systems into Meerschaum |
mqtt |
- Subscribe to MQTT topics | - Meerschaum shell must be running to receive data - May not be an instance connector |
- Ingesting data from existing IoT devices |