Ukraine flag We stand with our friends and colleagues in Ukraine. To support Ukraine in their time of need visit this page.

Deployment

Version  next-release-v2 Click here for the latest version

See also:


The main Jaeger backend components are released as Docker images on Docker Hubexternal link and Quayexternal link :

!!!UPDATE HERE

ComponentDocker HubQuay
jaeger-all-in-onehub.docker.com/r/jaegertracing/all-in-one/external linkquay.io/repository/jaegertracing/all-in-oneexternal link
jaeger-agenthub.docker.com/r/jaegertracing/jaeger-agent/external linkquay.io/repository/jaegertracing/jaeger-agentexternal link
jaeger-collectorhub.docker.com/r/jaegertracing/jaeger-collector/external linkquay.io/repository/jaegertracing/jaeger-collectorexternal link
jaeger-queryhub.docker.com/r/jaegertracing/jaeger-query/external linkquay.io/repository/jaegertracing/jaeger-queryexternal link
jaeger-ingesterhub.docker.com/r/jaegertracing/jaeger-ingester/external linkquay.io/repository/jaegertracing/jaeger-ingesterexternal link
jaeger-remote-storagehub.docker.com/r/jaegertracing/jaeger-remote-storage/external linkquay.io/repository/jaegertracing/jaeger-remote-storageexternal link

The images listed above are the primary release versions. Most components have additional images published:

  • ${component}-debug images include Delve debugger
  • ${component}-snapshot images are published from the tip of the main branch for every commit, allowing testing of unreleased versions
  • ${component}-debug-snapshot snapshot images that include the Delve debugger

There are orchestration templates for running Jaeger with:

Configuration Options

Jaeger binaries are configured via configuration YAML file.

!!!NEED TO WORK BELOW HERE

All-in-one

Jaeger all-in-one is a special distribution that combines Jaeger components, agent , collector , and query service/UI , in a single binary or container image. It is useful for single-node deployments where your trace volume is light enough to be handled by a single instance. By default, all-in-one starts with memory storage, meaning it will lose all data upon restart. All other span storage backends can also be used with all-in-one, but memory and badger are exclusive to all-in-one because they cannot be shared between instances.

All-in-one listens to the same ports as the components it contains (described below), with the exception of the admin port.

PortProtocolFunction
14269HTTPadmin port: health check at / and metrics at /metrics
## make sure to expose only the ports you use in your deployment scenario!
docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 9411:9411 \
  jaegertracing/all-in-one:next-release-v2.0

You can navigate to http://localhost:16686 to access the Jaeger UI.

Agent

jaeger-agent is deprecatedexternal link . The OpenTelemetry data can be sent from the OpenTelemetry SDKs (equipped with OTLP exporters) directly to jaeger-collector. See the Architecture page for alternative deployment options.

jaeger-agent is designed to receive tracing data in Thrift format over UDP and run locally on each host, either as a host agent / daemon or as an application sidecar. jaeger-agent exposes the following ports:

PortProtocolFunction
6831UDPAccepts jaeger.thriftexternal link in compact Thrift protocol used by most current Jaeger clients.
6832UDPAccepts jaeger.thriftexternal link in binary Thrift protocol used by Node.js Jaeger client (because thriftrwexternal link npm package does not support compact protocol).
5778HTTPServes SDK configs, namely sampling strategies at /sampling (see Remote Sampling ).
5775UDPAccepts zipkin.thriftexternal link in compact Thrift protocol (deprecated; only used by very old Jaeger clients, circa 2016).
14271HTTPAdmin port: health check at / and metrics at /metrics.

It can be executed directly on the host or via Docker, as follows:

## make sure to expose only the ports you use in your deployment scenario!
docker run \
  --rm \
  -p6831:6831/udp \
  -p6832:6832/udp \
  -p5778:5778/tcp \
  -p5775:5775/udp \
  jaegertracing/jaeger-agent:next-release-v2.0

Discovery System Integration

jaeger-agents can connect point-to-point to a single jaeger-collector address, which could be load balanced by another infrastructure component (e.g. DNS) across multiple jaeger-collectors. jaeger-agent can also be configured with a static list of jaeger-collector addresses.

On Docker, a command like the following can be used:

docker run \
  --rm \
  -p5775:5775/udp \
  -p6831:6831/udp \
  -p6832:6832/udp \
  -p5778:5778/tcp \
  jaegertracing/jaeger-agent:next-release-v2.0 \
  --reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250

When using gRPC, you have several options for load balancing and name resolution:

  • Single connection and no load balancing. This is the default if you specify a single host:port. (example: --reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250)
  • Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250)
  • Dynamic DNS resolution and round-robin load balancing. To get this behavior, prefix the address with dns:/// and gRPC will attempt to resolve the hostname using SRV records (for external load balancingexternal link ), TXT records (for service configsexternal link ), and A records. Refer to the gRPC Name Resolution docsexternal link and the dns_resolver.go implementationexternal link for more info. (example: --reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250)

Agent level tags

Jaeger supports agent level tags, that can be added to the process tags of all spans passing through jaeger-agent. This is supported through the command line flag --agent.tags=key1=value1,key2=value2,...,keyn=valuen. Tags can also be set through an environment flag like so - --agent.tags=key=${envFlag:defaultValue} - The tag value will be set to the value of the envFlag environment key and defaultValue if not set.

Collector

jaeger-collectors are stateless and thus many instances of jaeger-collector can be run in parallel. jaeger-collectors require almost no configuration, except for storage location, such as --cassandra.keyspace and --cassandra.servers options, or the location of Elasticsearch cluster, via --es.server-urls, depending on which storage is specified. See the CLI Flags for all command line options.

At default settings jaeger-collector exposes the following ports:

PortProtocolEndpointFunction
4317gRPCn/aAccepts traces in OpenTelemetry OTLP formatexternal link (Protobuf).
4318HTTP/v1/tracesAccepts traces in OpenTelemetry OTLP formatexternal link (Protobuf and JSON).
14268HTTP/api/samplingServes sampling policies (see Remote Sampling ).
/api/tracesAccepts spans in jaeger.thriftexternal link format with binary thrift protocol (POST).
14269HTTP/Admin port: health check (GET).
/metricsPrometheus-style metrics (GET).
9411HTTP/api/v1/spans and /api/v2/spansAccepts Zipkin spans in Thrift, JSON and Proto (disabled by default).
14250gRPCn/aUsed by jaeger-agent to send spans in model.protoexternal link Protobuf format.

Ingester

jaeger-ingester is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).

PortProtocolFunction
14270HTTPadmin port: health check at / and metrics at /metrics

To view all exposed configuration options run the following command:

docker run \
  -e SPAN_STORAGE_TYPE=cassandra \
  jaegertracing/jaeger-ingester:next-release-v2.0
  --help

Query Service & UI

jaeger-query serves the API endpoints and a React/Javascript UI. The service is stateless and is typically run behind a load balancer, such as NGINXexternal link .

At default settings the jaeger-query service exposes the following port(s):

PortProtocolFunction
16685gRPCProtobuf/gRPC QueryServiceexternal link
16686HTTP/api/* endpoints and Jaeger UI at /
16687HTTPadmin port: health check at / and metrics at /metrics

Minimal deployment example (Elasticsearch backend):

docker run -d --rm \
  -p 16685:16685 \
  -p 16686:16686 \
  -p 16687:16687 \
  -e SPAN_STORAGE_TYPE=elasticsearch \
  -e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
  jaegertracing/jaeger-query:next-release-v2.0

Clock Skew Adjustment

Jaeger backend combines trace data from applications that are usually running on different hosts. The hardware clocks on the hosts often experience relative drift, known as the clock skew effectexternal link . Clock skew can make it difficult to reason about traces, for example, when a server span may appear to start earlier than the client span, which should not be possible. jaeger-query service implements a clock skew adjustment algorithm ( codeexternal link ) to correct for clock drift, using the knowledge about causal relationships between spans. All adjusted spans have a warning displayed in the UI that provides the exact clock skew delta applied to their timestamps.

Sometimes these adjustments themselves make the trace hard to understand. For example, when repositioning the server span within the bounds of its parent span, Jaeger does not know the exact relationship between the request and response latencies, so it assumes they are equal and places the child span in the middle of the parent span (see issue #961external link ).

jaeger-query service supports a configuration flag --query.max-clock-skew-adjustment that controls how much clock skew adjustment should be allowed. Setting this parameter to zero (0s) disables clock skew adjustment completely. This setting applies to all traces retrieved from the given query service. There is an open ticket #197external link to support toggling the adjustment on and off directly in the UI.

UI Base Path

The base path for all jaeger-query HTTP routes can be set to a non-root value, e.g. /jaeger would cause all UI URLs to start with /jaeger. This can be useful when running jaeger-query behind a reverse proxy.

The base path can be configured via the --query.base-path command line parameter or the QUERY_BASE_PATH environment variable.

UI Customization and Embedding

Please refer to the dedicated Frontend/UI page .

Aggregation Jobs for Service Dependencies

Production deployments need an external process that aggregates data and creates dependency links between services. Project spark-dependenciesexternal link is a Spark job which derives dependency links and writes them directly to the storage.