OpenTelemetry Support

Overview

The io.Manager Server can be configured to export traces via OpenTelemetry.

By default, the io.Manager Server supports exporting traces to an OpenTelemetry Collector. It's also possible to provide your own custom trace exporter, trace span processor, and trace sampler pointing to a different backend service. Trace sampling is supported and is configurable.

Trace Instrumentation

io.Manager uses instrumentation libraries to enrich the generated traces with more details:

HTTP Server and Express instrumentation is enabled via the @opentelemetry/instrumentation-http and @opentelemetry/instrumentation-express libraries.
HTTP Client instrumentation is enabled via the @opentelemetry/instrumentation-http and @opentelemetry/instrumentation-undici libraries.
When using MongoDB databases, driver instrumentation is enabled via the @opentelemetry/instrumentation-mongodb library.
When using PostgreSQL or Microsoft SQL Server databases, driver instrumentation is enabled for Knex via the @opentelemetry/instrumentation-knex library.

Enabling Traces Exports

Publishing traces is disabled by default. To enable publishing traces, you must enable OpenTelemetry support in the io.Manager Server and explicitly enable the traces feature via the configuration object for initializing the io.Manager Server, or via environment variables, depending on your deployment approach.

Environment Variables

To enable and configure publishing traces, register the following environment variables with the proper values. The API_OTEL_TRACES_ENABLED environment variable must be set to true:

Environment Variable	Description
`API_OTEL_TRACES_DEFAULT_SAMPLE`	Specifies the rate to be used by the default trace sampler for sampling all traces that don't match the rules specified in the `sampling` property of the `traces` object in the configuration object for initializing the io.Manager Server. Accepts as a value a number greater than `0` and less than or equal to `1` denoting the probability percentage for sampling trace spans (e.g., a value of `0.15` means that all trace spans not matched by any rules will have a 15% probability of being sampled). Ignored when a custom trace sampler is used. Defaults to `1`.
`API_OTEL_TRACES_ENABLED`	If `true`, will enable publishing OpenTelemetry traces. Defaults to `false`.
`API_OTEL_TRACES_PUBLISH_INTERVAL`	Interval in milliseconds between two consecutive traces exports. Passed as a value to the `scheduledDelayMillis` property of the `BufferConfig` constructor parameter when creating a `BatchSpanProcessor` instance. This is the default span processor used by the io.Manager Server. Ignored when a custom trace span processor is used. Defaults to `5000`.
`API_OTEL_TRACES_URL`	URL pointing to an OpenTelemetry Collector where the generated traces will be sent via HTTP. Passed as a value to the `url` property of the `OTLPExporterNodeConfigBase` constructor parameter when creating an `OTLPTraceExporter` instance. This is the default trace exporter used by the io.Manager Server. Required if using the default trace exporter and trace span processor. Ignored when either a custom trace exporter or a custom trace span processor is used.

The following example demonstrates how to enable OpenTelemetry support, how to enable publishing traces, and how to configure the interval for publishing traces:

# Enabling OpenTelemetry support.
API_OTEL_ENABLED=true
API_OTEL_RESOURCE_SERVICE_NAME=io-manager

# Enabling and configuring traces.
API_OTEL_TRACES_ENABLED=true
API_OTEL_TRACES_URL=http://localhost:4318/v1/traces
API_OTEL_TRACES_PUBLISH_INTERVAL=10000

Configuration Object

To enable publishing traces, use the traces property under the otel top-level key of the optional Config object for initializing the io.Manager Server.

The following example demonstrates how to enable OpenTelemetry support, how to enable publishing traces, and how to configure the interval for publishing traces:

import { start } from "@interopio/manager";

const config = {
    // Enabling OpenTelemetry support.
    otel: {
        enabled: true,
        resource: {
            serviceName: "io-manager"
        },
        // Enabling and configuring traces.
        traces: {
            enabled: true,
            url: "http://localhost:4318/v1/traces",
            publishInterval: 10000
        }
    }
};

const server = await start(config);

The traces object has the following properties:

Property	Type	Description
`customExporter`	`object`	`SpanExporter` instance of a custom trace exporter to be used instead of the default trace exporter. Ignored when a custom trace span processor is used.
`customProcessor`	`object`	`SpanProcessor` instance of a custom trace span processor to be used instead of the default span processor.
`customSampler`	`object`	`Sampler` instance of a custom trace sampler to be used instead of the default trace sampler. When using a custom trace sampler, the settings specified in the `sampling` and `default` properties will be ignored.
`default`	`object`	Default trace settings to be applied to all traces not matched by any of the rules specified in the `sampling` array.
`enabled`	`boolean`	If `true`, will enable publishing OpenTelemetry traces. Defaults to `false`.
`publishInterval`	`number`	Interval in milliseconds between two consecutive trace exports. Passed as a value to the `scheduledDelayMillis` property of the `BufferConfig` constructor parameter when creating a `BatchSpanProcessor` instance. This is the default span processor used by the io.Manager Server. Ignored when a custom trace span processor is used. Defaults to `5000`.
`sampling`	`object[]`	List of objects each describing rules for matching trace spans. Each object also specifies a rate to be used for sampling the matched trace spans. Ignored when a custom trace sampler is used.
`url`	`string`	URL pointing to an OpenTelemetry Collector where the generated traces will be sent via HTTP. Passed as a value to the `url` property of the `OTLPExporterNodeConfigBase` constructor parameter when creating an `OTLPTraceExporter` instance. This is the default trace exporter used by the io.Manager Server. Required if using the default trace exporter and trace span processor. Ignored when either a custom trace exporter or a custom trace span processor is used.

The default object has the following properties:

Property	Type	Description
`sample`	`number`	Specifies the rate to be used by the default trace sampler for sampling all trace spans that don't match the rules specified in the `sampling` array. Accepts as a value a number greater than `0` and less than or equal to `1` denoting the probability percentage for sampling traces (e.g., a value of `0.15` means that all trace spans not matched by any rules will have a 15% probability of being sampled). Ignored when a custom trace sampler is used. Defaults to `1`.

Each object in the sampling array has the following properties:

Property	Type	Description
`attributes`	`object`	Collection of type `Record<string, string \| number>` containing attribute names and values to be used for matching trace spans.
`name`	`string`	String value for matching one or more spans by name. If the string value starts with `#` it will be treated as a case-insensitive regular expression.
`sample`	`number`	Specifies the rate to be used by the default trace sampler for sampling the matched trace spans. Accepts as a value a number greater than `0` and less than or equal to `1` denoting the probability percentage for sampling a trace (e.g., a value of `0.15` means that all trace spans matched by any of the rules will have a 15% probability of being sampled). Ignored when a custom trace sampler is used.

Trace Sampling

The io.Manager Server provides configurable mechanisms for sampling traces. It's possible to configure the default sampling rate, as well as to provide rules for matching the root spans of traces by name and attributes. This enables you to assign a custom sampling rate to all traces matched by a certain rule.

⚠️ Note that the matching rules are applied only to the root trace spans. If a root span is matched and sampled, the entire trace (i.e., all child spans belonging to the trace) will effectively be sampled.

The default sampling rate is 1 (meaning that 100% of all traces will be sampled) and is applied to all traces not matched by any rules. It can be configured via the API_OTEL_TRACES_DEFAULT_SAMPLE environment variable or via the sample property of the default object under the traces key in the configuration object for initializing the io.Manager Server.

The following example demonstrates defining a default sampling rate of 0.5:

import { start } from "@interopio/manager";

const config = {
    otel: {
        enabled: true,
        resource: {
            serviceName: "io-manager"
        },
        traces: {
            enabled: true,
            url: "http://localhost:4318/v1/traces",
            // Default sampling rate that will be applied
            // to all traces not matched by any rules.
            default: {
                sample: 0.5
            }
        }
    }
};

const server = await start(config);

To specify rules for matching the root spans of traces, use the sampling property of the traces object in the configuration object for initializing the io.Manager Server.

⚠️ Note that it isn't possible to provide sampling rules via environment variables.

The following example demonstrates assigning a custom sample rate of 0.1 to all root spans for HTTP requests sent to a specific route. The root spans are matched by using the supported "http.target" span attribute:

import { start } from "@interopio/manager";

const config = {
    otel: {
        enabled: true,
        resource: {
            serviceName: "io-manager"
        },
        traces: {
            enabled: true,
            url: "http://localhost:4318/v1/traces",
            // Trace sampling rules.
            sampling: [
                {
                    attributes: {
                        // Matching a root span by using a supported attribute as a criteria.
                        "http.target": "/api/apps"
                    },
                    // Assigning a custom sampling rate.
                    sample: 0.1
                }
            ],
            // All traces not matched by the specified rules
            // will be sampled at the default rate.
            default: {
                sample: 0.5
            }
        }
    }
};

const server = await start(config);

The io.Manager Server supports a set of span attributes that can be used to match root spans for HTTP requests. It also emits several other root spans whose well-known names you can use as matching criteria.

Supported Span Attributes

The following attributes can be used for matching root trace spans for HTTP requests:

Attribute	Description
`"http.flavor"`	Version of the HTTP protocol used (e.g., `"1.1."`).
`"http.host"`	The value of the HTTP host header (e.g., `"localhost:4356"`).
`"http.method"`	HTTP request method (e.g., `"GET"`).
`"http.scheme"`	The URI scheme identifying the used protocol (e.g., `"http"`).
`"http.target"`	The full request target as passed in an HTTP request line or equivalent (e.g, `"/api/apps"`).
`"http.url"`	Full HTTP request URL in the form `scheme://host[:port]/path?query[#fragment]` (e.g., `"http://localhost:4356/api/apps"`).
`"http.user_agent"`	Value of the HTTP `User-Agent` header sent by the client (e.g., `"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"`).
`"net.host.name"`	Local hostname or similar (e.g., `"localhost"`).
`"net.transport"`	Transport protocol used (e.g.,`"ip_tcp"`).

The following example demonstrates how to configure the io.Manager Server to use the "http.target" attribute to match and sample root spans for requests to route /api/apps at a rate of 0.1 (10%). All other root spans will be sampled at a rate of 1 (100%):

import { start } from "@interopio/manager";

const config = {
    otel: {
        enabled: true,
        resource: {
            serviceName: "io-manager"
        },
        traces: {
            enabled: true,
            url: "http://localhost:4318/v1/traces",
            sampling: [
                {
                    attributes: {
                      "http.target": "/api/apps"
                    },
                    sample: 0.1
                }
            ],
            default: {
                sample: 1
            }
        }
    }
};

const server = await start(config);

Supported Span Names

The io.Manager Server emits the following root spans whose names you can use in the rules for matching root trace spans:

Span Name	Description
`store.init`	Emitted when the io.Manager Server starts.
`store.stop`	Emitted when the io.Manager Server stops.
`cron-task.purge`	Emitted when the periodic purge task runs.
`cron-task.otel-active-sessions-export`	Emitted when the periodic task for exporting the `"io_manager.active_sessions"` metric runs.

The following example demonstrates how to configure the io.Manager Server to use the "store.init" span name to match and sample database initialization spans at a rate of 1 (100%). All other root spans will be sampled at a rate of 0.1 (10%):

import { start } from "@interopio/manager";

const config = {
    otel: {
        enabled: true,
        resource: {
            serviceName: "io-manager"
        },
        traces: {
            enabled: true,
            url: "http://localhost:4318/v1/traces",
            sampling: [
                {
                    name: "store.init",
                    sample: 1
                }
            ],
            default: {
                sample: 0.1
            }
        }
    }
};

const server = await start(config);

Customization

Trace Exporter

The io.Manager Server provides an out-of-the-box implementation of an OTLPTraceExporter for sending traces to an OpenTelemetry Collector via HTTP.

For more advanced scenarios, it's possible to provide an instance of a custom trace exporter by using the customExporter property of the traces object in the configuration object for initializing the io.Manager Server.

⚠️ Note that it isn't possible to provide a custom trace exporter via environment variables.

For a complete example, see the OpenTelemetry Custom Trace Exporter example on GitHub.

Trace Span Processor

The io.Manager Server provides an out-of-the-box implementation of a BatchSpanProcessor for exporting traces periodically.

For more advanced scenarios, it's possible to provide an instance of a custom trace span processor by using the customProcessor property of the traces object in the configuration object for initializing the io.Manager Server.

⚠️ Note that it isn't possible to provide a custom trace span processor via environment variables.

For a complete example, see the OpenTelemetry Custom Trace Processor example on GitHub.

Trace Sampler

The io.Manager Server provides an out-of-the-box implementation of a default trace sampler that uses the rules defined in the sampling array and the default sampling rate defined in the sample property of the default object (or via the API_OTEL_TRACES_DEFAULT_SAMPLE environment variable) to sample trace spans.

For more advanced scenarios, it's possible to provide an instance of a custom trace sampler by using the customSampler property of the traces object in the configuration object for initializing the io.Manager Server.

⚠️ Note that it isn't possible to provide a custom trace sampler via environment variables.

For a complete example, see the OpenTelemetry Custom Trace Sampler example on GitHub.

io.Manager Documentation

OpenTelemetry Support