# Traces

Source: https://docs.interop.io/manager/opentelemetry-support/traces/index.html

## Overview

The **io.Manager** Server can be configured to export [traces](https://opentelemetry.io/docs/concepts/signals/traces/) via OpenTelemetry.

By default, the **io.Manager** Server supports exporting traces to an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/). It's also possible to provide your own [custom](#customization) trace exporter, trace span processor, and trace sampler pointing to a different backend service. [Trace sampling](#trace_sampling) is supported and is configurable.

### Trace Instrumentation

**io.Manager** uses instrumentation libraries to enrich the generated traces with more details:

- HTTP Server and [Express](https://expressjs.com/) instrumentation is enabled via the [`@opentelemetry/instrumentation-http`](https://www.npmjs.com/package/@opentelemetry/instrumentation-http) and [`@opentelemetry/instrumentation-express`](https://www.npmjs.com/package/@opentelemetry/instrumentation-express) libraries.

- HTTP Client instrumentation is enabled via the [`@opentelemetry/instrumentation-http`](https://www.npmjs.com/package/@opentelemetry/instrumentation-http) and [`@opentelemetry/instrumentation-undici`](https://www.npmjs.com/package/@opentelemetry/instrumentation-undici) libraries.

- When using [MongoDB](https://docs.interop.io/manager/databases/mongo/index.md) databases, driver instrumentation is enabled via the [`@opentelemetry/instrumentation-mongodb`](https://www.npmjs.com/package/@opentelemetry/instrumentation-mongodb) library.

- When using [PostgreSQL](https://docs.interop.io/manager/databases/postgresql/index.md) or [Microsoft SQL Server](https://docs.interop.io/manager/databases/microsoft-sql/index.md) databases, driver instrumentation is enabled for [Knex](https://knexjs.org/) via the [`@opentelemetry/instrumentation-knex`](https://www.npmjs.com/package/@opentelemetry/instrumentation-knex) library.

## Enabling Traces Exports

Publishing traces is disabled by default. To enable publishing traces, you must [enable OpenTelemetry support](https://docs.interop.io/manager/opentelemetry-support/overview/index.md#enabling_opentelemetry_support) in the **io.Manager** Server and explicitly enable the traces feature via the configuration object for initializing the **io.Manager** Server, or via environment variables, depending on your [deployment](https://docs.interop.io/manager/deployment/index.md) approach.

### Environment Variables

To enable and configure publishing traces, register the following environment variables with the proper values. The `API_OTEL_TRACES_ENABLED` environment variable must be set to `true`:

| Environment Variable | Description |
|----------------------|-------------|
| `API_OTEL_TRACES_DEFAULT_SAMPLE` | Specifies the rate to be used by the default trace sampler for sampling all traces that don't match the rules specified in the `sampling` property of the `traces` object in the [configuration object](#enabling_traces_exports-configuration_object) for initializing the **io.Manager** Server. Accepts as a value a number greater than `0` and less than or equal to `1` denoting the [probability](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling-experimental/) percentage for sampling trace spans (e.g., a value of `0.15` means that all trace spans not matched by any rules will have a 15% probability of being sampled). Ignored when a [custom trace sampler](#customization-trace_sampler) is used. Defaults to `1`. |
| `API_OTEL_TRACES_ENABLED` | If `true`, will enable publishing OpenTelemetry traces. Defaults to `false`. |
| `API_OTEL_TRACES_PUBLISH_INTERVAL` | Interval in milliseconds between two consecutive traces exports. Passed to the [`BatchSpanProcessor`](https://open-telemetry.github.io/opentelemetry-js/classes/_opentelemetry_sdk-trace-base.BatchSpanProcessor.html) constructor. This is the default span processor used by the **io.Manager** Server. Ignored when a [custom trace span processor](#customization-trace_span_processor) is used. Defaults to `5000`. |
| `API_OTEL_TRACES_URL` | URL pointing to an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) where the generated traces will be sent via HTTP. Passed to the [`OTLPTraceExporter`](https://open-telemetry.github.io/opentelemetry-js/classes/_opentelemetry_exporter-trace-otlp-http.OTLPTraceExporter.html) constructor. This is the default trace exporter used by the **io.Manager** Server. Required if using the default trace exporter and trace span processor. Ignored when either a [custom trace exporter](#customization-trace_exporter) or a [custom trace span processor](#customization-trace_span_processor) is used. |

The following example demonstrates how to [enable OpenTelemetry support](https://docs.interop.io/manager/opentelemetry-support/overview/index.md#enabling_opentelemetry_support), how to enable publishing traces, and how to configure the interval for publishing traces:

```cmd
# Enabling OpenTelemetry support.
API_OTEL_ENABLED=true
API_OTEL_RESOURCE_SERVICE_NAME=io-manager

# Enabling and configuring traces.
API_OTEL_TRACES_ENABLED=true
API_OTEL_TRACES_URL=http://localhost:4318/v1/traces
API_OTEL_TRACES_PUBLISH_INTERVAL=10000
```

### Configuration Object

To enable publishing traces, use the `traces` property under the `otel` top-level key of the optional `Config` object for initializing the **io.Manager** Server.

The following example demonstrates how to [enable OpenTelemetry support](https://docs.interop.io/manager/opentelemetry-support/overview/index.md#enabling_opentelemetry_support), how to enable publishing traces, and how to configure the interval for publishing traces:

```javascript
import { start } from "@interopio/manager";

const config = {
    // Enabling OpenTelemetry support.
    otel: {
        enabled: true,
        resource: {
    	    serviceName: "io-manager"
    	},
        // Enabling and configuring traces.
    	traces: {
    	    enabled: true,
    	    url: "http://localhost:4318/v1/traces",
            publishInterval: 10000
    	}
    }
};

const server = await start(config);
```

The `traces` object has the following properties:

| Property | Type | Description |
|----------|------|-------------|
| `customExporter` | `object` | [`SpanExporter`](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_sdk-trace-base.SpanExporter.html) instance of a [custom trace exporter](#customization-trace_exporter) to be used instead of the default trace exporter. Ignored when a [custom trace span processor](#customization-trace_span_processor) is used. |
| `customContextManager` | `object` | [`ContextManager`](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_api._opentelemetry_api.ContextManager.html) instance of a custom context manager. *Available since **io.Manager** Server 2.0.* |
| `customProcessor` | `object` | [`SpanProcessor`](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_sdk-trace-base.SpanProcessor.html) instance of a [custom trace span processor](#customization-trace_span_processor) to be used instead of the default span processor. |
| `customPropagator` | `object` | [`TextMapPropagator`](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_api._opentelemetry_api.TextMapPropagator.html) instance of a custom context propagator. *Available since **io.Manager** Server 2.0.* |
| `customSampler` | `object` | [`Sampler`](https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_sdk-trace-base.Sampler.html) instance of a [custom trace sampler](#customization-trace_sampler) to be used instead of the default trace sampler. When using a custom trace sampler, the settings specified in the `sampling` and `default` properties will be ignored. |
| `default` | `object` | Default trace settings to be applied to all traces not matched by any of the rules specified in the `sampling` array. |
| `enabled` | `boolean` | If `true`, will enable publishing OpenTelemetry traces. Defaults to `false`. |
| `publishInterval` | `number` | Interval in milliseconds between two consecutive trace exports. Passed to the [`BatchSpanProcessor`](https://open-telemetry.github.io/opentelemetry-js/classes/_opentelemetry_sdk-trace-base.BatchSpanProcessor.html) constructor. This is the default span processor used by the **io.Manager** Server. Ignored when a [custom trace span processor](#customization-trace_span_processor) is used. Defaults to `5000`. |
| `sampling` | `object[]` | List of objects each describing rules for matching trace spans. Each object also specifies a rate to be used for sampling the matched trace spans. Ignored when a [custom trace sampler](#customization-trace_sampler) is used. |
| `url` | `string` | URL pointing to an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) where the generated traces will be sent via HTTP. Passed to the [`OTLPTraceExporter`](https://open-telemetry.github.io/opentelemetry-js/classes/_opentelemetry_exporter-trace-otlp-http.OTLPTraceExporter.html) constructor. This is the default trace exporter used by the **io.Manager** Server. Required if using the default trace exporter and trace span processor. Ignored when either a [custom trace exporter](#customization-trace_exporter) or a [custom trace span processor](#customization-trace_span_processor) is used. |

The `default` object has the following properties:

| Property | Type | Description |
|----------|------|-------------|
| `sample` | `number` | Specifies the rate to be used by the default trace sampler for sampling all trace spans that don't match the rules specified in the `sampling` array. Accepts as a value a number greater than `0` and less than or equal to `1` denoting the [probability](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling-experimental/) percentage for sampling traces (e.g., a value of `0.15` means that all trace spans not matched by any rules will have a 15% probability of being sampled). Ignored when a [custom trace sampler](#customization-trace_sampler) is used. Defaults to `1`. |

Each object in the `sampling` array has the following properties:

| Property | Type | Description |
|----------|------|-------------|
| `attributes` | `object` | Collection of type `{ [key: string]: string \| number \| boolean }` containing attribute names and values to be used for matching trace spans. If a string value starts with `#`, it will be treated as a case-insensitive regular expression. |
| `name` | `string` | String value for matching one or more spans by name. If the string value starts with `#`, it will be treated as a case-insensitive regular expression. |
| `sample` | `number` | Specifies the rate to be used by the default trace sampler for sampling the matched trace spans. Accepts as a value a number greater than `0` and less than or equal to `1` denoting the [probability](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling-experimental/) percentage for sampling a trace (e.g., a value of `0.15` means that all trace spans matched by any of the rules will have a 15% probability of being sampled). Ignored when a [custom trace sampler](#customization-trace_sampler) is used. |

## Trace Sampling

The **io.Manager** Server provides configurable mechanisms for [sampling traces](https://opentelemetry.io/docs/concepts/sampling/). It's possible to configure the default sampling rate, as well as to provide rules for matching the root spans of traces by name and attributes. This enables you to assign a custom sampling rate to all traces matched by a certain rule.

> ⚠️ *Note that the matching rules are applied only to the root trace spans. If a root span is matched and sampled, the entire trace (i.e., all child spans belonging to the trace) will effectively be sampled.*

The default sampling rate is `1` (meaning that 100% of all traces will be sampled) and is applied to all traces not matched by any rules. It can be configured via the `API_OTEL_TRACES_DEFAULT_SAMPLE` [environment variable](#enabling_traces_exports-environment_variables) or via the `sample` property of the `default` object under the `traces` key in the [configuration object](#enabling_traces_exports-configuration_object) for initializing the **io.Manager** Server.

The following example demonstrates defining a default sampling rate of `0.5`:

```javascript
import { start } from "@interopio/manager";

const config = {
    otel: {
        enabled: true,
        resource: {
    	    serviceName: "io-manager"
    	},
    	traces: {
    	    enabled: true,
    	    url: "http://localhost:4318/v1/traces",
            // Default sampling rate that will be applied
            // to all traces not matched by any rules.
            default: {
                sample: 0.5
            }
    	}
    }
};

const server = await start(config);
```

To specify rules for matching the root spans of traces, use the `sampling` property of the `traces` object in the configuration object for initializing the **io.Manager** Server.

> ⚠️ *Note that it isn't possible to provide sampling rules via environment variables.*

The following example demonstrates assigning a custom sample rate of `0.1` to all root spans for HTTP requests sent to a specific route. The root spans are matched by using the supported `"http.target"` span attribute:

```javascript
import { start } from "@interopio/manager";

const config = {
    otel: {
        enabled: true,
        resource: {
    	    serviceName: "io-manager"
    	},
    	traces: {
    	    enabled: true,
    	    url: "http://localhost:4318/v1/traces",
            // Trace sampling rules.
            sampling: [
                {
                    attributes: {
                        // Matching a root span by using a supported attribute as a criteria.
                        "http.target": "/api/apps"
                    },
                    // Assigning a custom sampling rate.
                    sample: 0.1
                }
            ],
            // All traces not matched by the specified rules
            // will be sampled at the default rate.
            default: {
                sample: 0.5
            }
    	}
    }
};

const server = await start(config);
```

The **io.Manager** Server supports a set of span attributes that can be used to match root spans for HTTP requests. It also emits several other root spans whose well-known [names](#supported_span_names) you can use as matching criteria.

### Supported Span Attributes

The following attributes can be used for matching root trace spans for HTTP requests:

| Attribute | Description |
|-----------|-------------|
| `"http.flavor"` | Version of the HTTP protocol used (e.g., `"1.1."`). |
| `"http.host"` | The value of the HTTP host header (e.g., `"localhost:4356"`). |
| `"http.method"` | HTTP request method (e.g., `"GET"`). |
| `"http.scheme"` | The URI scheme identifying the used protocol (e.g., `"http"`). |
| `"http.target"` | The full request target as passed in an HTTP request line or equivalent (e.g, `"/api/apps"`). |
| `"http.url"` | Full HTTP request URL in the form `scheme://host[:port]/path?query[#fragment]` (e.g., `"http://localhost:4356/api/apps"`). |
| `"http.user_agent"` | Value of the HTTP `User-Agent` header sent by the client (e.g., `"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"`). |
| `"net.host.name"` | Local hostname or similar (e.g., `"localhost"`). |
| `"net.transport"` | Transport protocol used (e.g.,`"ip_tcp"`). |

The following example demonstrates how to configure the **io.Manager** Server to use the `"http.target"` attribute to match and sample root spans for requests to route `/api/apps` at a rate of `0.1` (10%). All other root spans will be sampled at a rate of `1` (100%):

```javascript
import { start } from "@interopio/manager";

const config = {
    otel: {
    	enabled: true,
    	resource: {
    	    serviceName: "io-manager"
    	},
    	traces: {
    	    enabled: true,
    		url: "http://localhost:4318/v1/traces",
    		sampling: [
    		    {
    		        attributes: {
    		          "http.target": "/api/apps"
    		        },
    		        sample: 0.1
    		    }
    		],
    		default: {
    		    sample: 1
    		}
    	}
    }
};

const server = await start(config);
```

### Supported Span Names

The **io.Manager** Server emits the following root spans whose names you can use in the rules for matching root trace spans:

| Span Name | Description |
|-----------|-------------|
| `store.init` | Emitted when the **io.Manager** Server starts. |
| `store.stop` | Emitted when the **io.Manager** Server stops. |
| `Create Nest App` | Emitted when the **io.Manager** Server starts. *Available since **io.Manager** Server 2.0.* |
| `cron-task.purge` | Emitted when the periodic [purge operation](https://docs.interop.io/manager/databases/overview/index.md#database_size_management) is executed. |
| `cron-task.otel-active-sessions-export` | Emitted when the periodic task for exporting the [`"io_manager.active_sessions"`](https://docs.interop.io/manager/opentelemetry-support/metrics/index.md#available_metrics-iomanageractivesessions) metric is executed. |

The following example demonstrates how to configure the **io.Manager** Server to use the `"store.init"` span name to match and sample database initialization spans at a rate of `1` (100%). All other root spans will be sampled at a rate of `0.1` (10%):

```javascript
import { start } from "@interopio/manager";

const config = {
    otel: {
    	enabled: true,
    	resource: {
    	    serviceName: "io-manager"
    	},
    	traces: {
    	    enabled: true,
    		url: "http://localhost:4318/v1/traces",
    		sampling: [
    		    {
    		        name: "store.init",
    		        sample: 1
    		    }
    		],
    		default: {
    		    sample: 0.1
    		}
    	}
    }
};

const server = await start(config);
```

## Emitted Spans

In addition to the spans produced by the instrumentation libraries, the **io.Manager** Server emits its own spans for notable operations. The ones listed below are child spans - unlike the [root spans](#supported_span_names), they can't be matched by a sampling rule and are sampled together with the trace they belong to.

### db.transaction

Emitted for each database transaction the **io.Manager** Server opens, covering every attempt of that transaction. Operations that run without a database transaction - for example, when running against a read-only database host, or against MongoDB with transactions disabled - don't emit this span. *Available since **io.Manager** 4.0.*

The `"db.transaction"` span carries the following attributes:

| Attribute | Description |
|-----------|-------------|
| `"db.system.name"` | The database the server is running against - `postgresql`, `mongodb`, or `microsoft.sql_server`. |
| `"io_manager.db.transaction.attempts"` | How many attempts the transaction took, counting the first one. Always `1` when running against MongoDB, where transactions are retried by the database driver. |
| `"io_manager.db.transaction.isolation"` | The `transactionIsolation` value configured for the database. Not present when running against MongoDB, which has no such setting. |

Each transient-failure retry of the transaction is recorded on the span as a `"db.transaction.retry"` event carrying an `error.type` attribute with the database error code that triggered the retry.

## Customization

### Trace Exporter

The **io.Manager** Server provides an out-of-the-box implementation of an [`OTLPTraceExporter`](https://open-telemetry.github.io/opentelemetry-js/classes/_opentelemetry_exporter-trace-otlp-http.OTLPTraceExporter.html) for sending traces to an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) via HTTP.

For more advanced scenarios, it's possible to provide an instance of a custom trace exporter by using the `customExporter` property of the `traces` object in the [configuration object](#enabling_traces_exports-configuration_object) for initializing the **io.Manager** Server.

> ⚠️ *Note that it isn't possible to provide a custom trace exporter via environment variables.*

> ℹ️ *For a complete example of creating a custom trace exporter, see the [OpenTelemetry Custom Trace Exporter](https://github.com/InteropIO/manager-examples/tree/main/otel-custom-trace-exporter) example on GitHub.*

### Trace Span Processor

The **io.Manager** Server provides an out-of-the-box implementation of a [`BatchSpanProcessor`](https://open-telemetry.github.io/opentelemetry-js/classes/_opentelemetry_sdk-trace-base.BatchSpanProcessor.html) for exporting traces periodically.

For more advanced scenarios, it's possible to provide an instance of a custom trace span processor by using the `customProcessor` property of the `traces` object in the [configuration object](#enabling_traces_exports-configuration_object) for initializing the **io.Manager** Server.

> ⚠️ *Note that it isn't possible to provide a custom trace span processor via environment variables.*

> ℹ️ *For a complete example of creating a custom trace span processor, see the [OpenTelemetry Custom Trace Processor](https://github.com/InteropIO/manager-examples/tree/main/otel-custom-trace-processor) example on GitHub.*

### Trace Sampler

The **io.Manager** Server provides an out-of-the-box implementation of a default trace sampler that uses the rules defined in the `sampling` array and the default sampling rate defined in the `sample` property of the `default` object (or via the `API_OTEL_TRACES_DEFAULT_SAMPLE` environment variable) to sample trace spans.

For more advanced scenarios, it's possible to provide an instance of a custom trace sampler by using the `customSampler` property of the `traces` object in the [configuration object](#enabling_traces_exports-configuration_object) for initializing the **io.Manager** Server.

> ⚠️ *Note that it isn't possible to provide a custom trace sampler via environment variables.*

> ℹ️ *For a complete example of creating a custom trace sampler, see the [OpenTelemetry Custom Trace Sampler](https://github.com/InteropIO/manager-examples/tree/main/otel-custom-trace-sampler) example on GitHub.*