Ukraine flag We stand with our friends and colleagues in Ukraine. To support Ukraine in their time of need visit this page.

Client Libraries

Version  2.0 Latest Go to the latest 1.x version

See also:

Jaeger clients have been retired. Please use the OpenTelemetry SDKs.

Deprecating Jaeger clients

The Jaeger clients have faithfully served our community for several years. We pioneered many new features, such as remotely controlled samplers and per-operation / adaptive sampling, which were critical to the success of distributed tracing deployments at large organizations. However, now that the larger community in OpenTelemetry has caught up with the Jaeger clients in terms of feature parity and there is full support for exporting data to Jaeger, we believe it is time to decommission Jaeger’s native clients and refocus the efforts on the OpenTelemetry SDKs.

For new applications, we recommend using the OpenTelemetry external link APIs, SDKs, and instrumentation. Since v1.35, the Jaeger backend can receive trace data from the OpenTelemetry SDKs in their native OpenTelemetry Protocol (OTLP) external link.

For existing applications that are already instrumented with the OpenTracing API, we recommend replacing the Jaeger clients with the corresponding OpenTelemetry SDKs and the OpenTracing shim/bridge available in most languages supported by Jaeger.

Timeline

We plan to continue accepting pull requests and making new releases of Jaeger clients through the end of 2021. In January 2022 we will enter a code freeze period for 6 months, during which time we will no longer accept pull requests with new features, with the exception of security-related fixes. After that we will archive the client library repositories and will no longer accept new changes.

Migration to OpenTelemetry

The OpenTelemetry project published a migration guide external link from the OpenTracing APIs to the OpenTelemetry SDKs via the OpenTracing bridges/shims. There may be different levels of maturity and features in different OpenTelemetry SDKs. We will keep updating the information below as more of it becomes available.

Baggage support

OpenTelemetry implements baggage propagation differently from OpenTracing and they are not completely equivalent. In OpenTelemetry the context layer sits below the tracing API and relies on immutable context objects, whereas baggage in OpenTracing is stored in a span which is mutable (and may occasionally lead to tricky race conditions when starting children spans).

We need your help!

If you find inaccuracies or have information that can be added, please open an issue or a PR to the documentation repo external link. If some features are missing and you need them, please open tickets in the respective OpenTelemetry repos or contribute. For example, Jaeger’s remote samplers are not yet implemented in every OpenTelemetry SDK, but porting them from the Jaeger codebase is a fairly straightforward task.

Copying Jaeger code

We encourage OpenTelemetry SDK authors to copy relevant pieces of the Jaeger clients instead of depending on Jaeger modules directly. This is why we use a liberal APL2 license. When copying code, the correct way to respect the license requirements is to keep the copyright notices. For example, Jaeger authors did the same with the code originally written at Uber:

// Copyright (c) 2019 The Jaeger Authors.
// Copyright (c) 2017 Uber Technologies, Inc.
// ... <rest of Apache notice> ...

Java

Python

Node.js

Go

C# / .NET

C++

Intro

All Jaeger client libraries support the OpenTracing APIs external link. The following resources provide more information about instrumenting your application with OpenTracing:

The rest of this page contains information about configuring and instantiating a Jaeger tracer in an application that is already instrumented with OpenTracing API.

Terminology

We use the terms client library, instrumentation library, and tracer interchangeably in this document.

Supported Libraries

The following client libraries are officially supported:

LanguageGitHub Repo
Gojaegertracing/jaeger-client-go
Javajaegertracing/jaeger-client-java
Node.jsjaegertracing/jaeger-client-node
Pythonjaegertracing/jaeger-client-python
C++jaegertracing/jaeger-client-cpp
C#jaegertracing/jaeger-client-csharp

Libraries in other languages are currently under development, please see issue #366 external link.

Initializing Jaeger Tracer

The initialization syntax is slightly different in each languages, please refer to the README’s in the respective repositories. The general pattern is to not create the Tracer explicitly, but use a Configuration class to do that. Configuration allows simpler parameterization of the Tracer, such as changing the default sampler or the location of Jaeger agent.

Tracer Internals

Sampling

See Architecture | Sampling.

Reporters

Jaeger tracers use reporters to process finished spans. Typically Jaeger libraries ship with the following reporters:

  • NullReporter does nothing with the span. It can be useful in unit tests.
  • LoggingReporter simply logs the fact that a span was finished, usually by printing the trace and span ID and the operation name.
  • CompositeReporter takes a list of other reporters and invokes them one by one.
  • RemoteReporter (default) buffers a certain number of finished spans in memory and uses a sender to submit a batch of spans out of process to Jaeger backend. The sender is responsible for serializing the span to the wire format (e.g. Thrift or JSON) and communicating with the backend components (e.g. over UDP or HTTP).

EMSGSIZE and UDP buffer limits

By default Jaeger libraries use a UDP sender to report finished spans to the jaeger-agent daemon. The default max packet size is 65,000 bytes, which can be transmitted without segmentation when connecting to the agent via loopback interface. However, some OSs (in particular, MacOS), limit the max buffer size for UDP packets, as raised in this GitHub issue external link. If you run into issue with EMSGSIZE errors, consider raising the limits in your kernel (see the issue for examples). You can also configure the client libraries to use a smaller max packet size, but that may cause issues if you have large spans, e.g. if you log big chunks of data. Spans that exceed max packet size are dropped by the clients (with metrics emitted to indicate that). Another alternative is to use non-UDP transports, such as HttpSender in Java external link (not currently available for all languages).

Metrics

Jaeger tracers emit various metrics about how many spans or traces they have started and finished, how many of them were sampled or not sampled, if there were any errors in decoding trace context from inbound requests or reporting spans to the backend.

TODO standardize and describe the metric names and labels (issues #572 external link, #611 external link).

Propagation Format

When SpanContext is encoded on the wire as part of the request to another service, Jaeger client libraries default to the Jaeger native propagation format specified below. In addition, Jaeger clients support Zipkin B3 format external link and W3C Trace-Context external link.

Trace/Span Identity

Key

uber-trace-id

  • Case-insensitive in HTTP
  • Lower-case in protocols that preserve header case

Value

{trace-id}:{span-id}:{parent-span-id}:{flags}

  • {trace-id}
    • 64-bit or 128-bit random number in base16 format
    • Can be variable length, shorter values are 0-padded on the left
      • Receivers MUST accept hex-strings shorter than 32 characters and 0-pad them on the left
      • Senders SHOULD generate hex strings of exactly 16 or 32 characters in length
    • Clients in some languages support 128-bit, migration pending
    • Value of 0 is not valid
  • {span-id}
    • 64-bit random number in base16 format
    • Can be variable length, shorter values are 0-padded on the left
      • Receivers MUST accept hex-strings shorter than 16 characters and 0-pad them on the left
      • Senders SHOULD generate hex strings of exactly 16 characters in length
    • Value of 0 is not valid
  • {parent-span-id}
    • 64-bit value in base16 format representing parent span id
    • Deprecated, most Jaeger clients ignore on the receiving side, but still include it on the sending side
    • 0 value is valid and means “root span” (when not ignored)
  • {flags}
    • One byte bitmap, as one or two hex digits (leading zero may be omitted)
    • Bit 1 (right-most, least significant, bit mask 0x01) is “sampled” flag
      • 1 means the trace is sampled and all downstream services are advised to respect that
      • 0 means the trace is not sampled and all downstream services are advised to respect that
        • We’re considering a new feature that allows downstream services to upsample if they find their tracing level is too low
    • Bit 2 (bit mask 0x02 ) is “debug” flag
      • Debug flag should only be set when the sampled flag is set
      • Instructs the backend to try really hard not to drop this trace
    • Bit 3 (bit mask 0x04 ) is not used
    • Bit 4 (bit mask 0x08 ) is “firehose” flag
      • Spans tagged as “firehose” are excluded from being indexed in the storage
      • The traces can only be retrieved by trace ID (usually available from other sources, like logs)
    • Other bits are unused

Baggage

  • Key: uberctx-{baggage-key}
  • Value: {baggage-value} as a string (see Value Encoding below)
  • Limitation: since HTTP headers don’t preserve the case, Jaeger recommends baggage keys to be lowercase-kebab-case, e.g. my-baggage-key-1.

Example: the following code sequence:

span.SetBaggageItem("key1", "value1")
span.SetBaggageItem("key2", "value2")

will result in the following HTTP headers:

uberctx-key1: value1
uberctx-key2: value2

Value Encoding

OpenTracing defines two formats for plain text headers: HTTP_HEADERS and TEXT_MAP. The former was introduced to deal with restrictions imposed by the HTTP protocol on the context of the headers, whereas the latter does not impose any restrictions, e.g. it can be used with Kafka Record Headers. The main difference between these two formats in the Jaeger SDKs is that the baggage values are URL-encoded when using the HTTP_HEADERS propagation format.

Example: when using the HTTP_HEADERS propagation format, the following code sequence:

span.SetBaggageItem("key1", "value 1 / blah")

will result in the following HTTP header:

uberctx-key1: value%201%20%2F%20blah