OpenTelemetry with Java Guide

Credits : https://www.aspecto.io/blog/getting-started-with-opentelemetry-java/

What is OpenTelemetry?

OpenTelemetry is a collection of SDKs and APIs – an open-source project –  that allows us to collect, generate, and export logs, metrics, and traces (also known as the three pillars of observability).

OpenTelemetry enables us to instrument our distributed services. Instrumenting means capturing telemetry data from events and operations in our distributed system.

Ultimately, we use this data to understand and investigate our system’s behavior and troubleshoot and debug performance issues and errors.

Led by the CNCF (Cloud Native Computing Foundation, the foundation responsible for Kubernetes), it serves as a single library that gathers data under a single specification.

It allows us to be vendor-agnostic and not tied down to a single tool and ship our data to any dedicated location (e.g., backend, OpenTelemetry collector, supporting open sources, and more). 

Since its implementation for most modern programming languages, it has been growing in popularity and usage by developers everywhere. It combines all three aspects needed for proper monitoring (leading with tracing, followed by metrics).

There is much more to learn about OpenTelemetry and if you want to dive deeper into its structure, follow this guide. For this guide, here are the key terms you need to know:

• Span: A span represents an action/operation that occurred in our system. An HTTP request or a database operation that spans over time. A span would usually be the parent and/or the child of another span.

• Trace: Represents a tree of spans connected in a child/parent relationship. Traces describe the progression of requests across different services and components in our system (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.

• Instrumentation – instrumentation libraries are what allow us to gather the data and create spans based on the different libraries in our system such as Kafka, MySQL, Spring, etc. There are 2 ways to instrument our app – manually or automatically:
  • Auto instrumentation: Automatically create spans from the application libraries we use with ready-to-use OpenTelemetry libraries.
  • Manual instrumentation: Manually create spans by adding code to your application to define the beginning and end of each span.

• Exporter: Once we create a span, we need to send it to a dedicated backend. It may be in memory, console output, vendor, or an open-source such as Jaeger Tracing. The exporter handles sending the data to our backend.

For more OpenTelemetry terminology, visit the official documentation.

Hello World: OpenTelemetry Java

Step 1: Download the latest OpenTelemetry Java agent

Download the latest Java agent ‘JAR’ from the official repo on GitHub and copy the opentelemetry-javaagent-all.jar file to your project. 

Note: Make sure you cloned the Spring Pet Clinic project we mentioned before. 

Step 2: Logging OpenTelemetry spans to console

1) Run the following commands in your terminal 

1. We are setting up our service name my-service
2. Set the exporter to log spans to our console  OTEL_TRACES_EXPORTER=logging

OTEL_SERVICE_NAME=my-service OTEL_TRACES_EXPORTER=logging java -javaagent:./opentelemetry-javaagent.jar -jar target/*.jar

We mentioned instrumentations earlier, the libraries that actually allow us to gather data from our apps. The above command will automatically instrument most common libraries. You can view the complete list here.

In terms of implementing OpenTelemetry Java instrumentation in our project to collect traces, we’re pretty much done (keep reading, there’s more to it).

Now, the final output should look something like the one below.

//Console output
 
INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - '/owners/{ownerId}' :
99af87eeaa19b83d014463b046884e56 632e2d2a5931bd17 SERVER [tracer: io.opentelemetry.tomcat-7.0:1.13.1-
alpha] AttributesMap{data={net.transport=ip_tcp, http.target=/owners/11, thread.id=120, http.flavor=1.1,
http.status_code=200, net.peer.ip=0:0:0:0:0:0:0:1, thread.name=http-nio-8080-exec-10,
http.host=localhost:8080, http.route=/owners/{ownerId}, http.user_agent=Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36,
http.method=GET, net.peer.port=50778, http.scheme=http}, capacity=128, totalAddedValues=13}

Note: This is not the best-looking console output 😬 We thought about making it prettier, however, this is the way it looks in the console so we decided to stick to the source to prevent any confusion.

If you look at the JSON, you can see that, for example, http.route=/owners/{ownerId} represents an https call to fetch an owner by ID.

At this point, we created spans and log them to our console. But we’re not here only to have beautiful spans in our console – it’s all about visualization.

Our ability to visualize trace data is where the true troubleshooting power of this technology comes into play. 

For visualization, we’ll be using the open-source Jaeger Tracing and Aspecto.

Getting Started with OpenTelemetry Java and Jaeger Tracing

Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces. 

This is how we send traces to Jaeger:

Export to Jaeger

1) Run Jaeger locally with the following docker command

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.32

2) Previously, we set the exporter to log to our console. Now, we need to make the following changes and run this command:

1. Change the exporter to Jaeger’s OTEL_TRACES_EXPORTER=jaeger
2. Specify the Jaeger’s endpoint OTEL_EXPORTER_JAEGER_ENDPOINT

OTEL_SERVICE_NAME=my-service OTEL_TRACES_EXPORTER=jaeger OTEL_EXPORTER_JAEGER_ENDPOINT=http://localhost:14250 java -javaagent:./opentelemetry-javaagent.jar -jar target/*.jar

3) Run your Pet Clinic project and execute a few actions. For example, we created a new owner, then searched the owner by the last name.

4) Use your browser to view Jaeger UI at  http://localhost:16686/

5) We can now view our trace in the Jaeger UI. Select ‘my-service’ from the search pane on the right and click on Find Traces.

You can drill down into a specific trace. For example, a request to search the owner by owner ID.

Advanced Visualization with OpenTelemetry Java and Aspecto

So now you know the basics of instrumentations, spans, traces, and how we can use OpenTelemetry to generate traces for our Java app.

You can take your tracing visualization to the next level with Aspecto.

Sending traces to Aspecto is as easy as what we were doing so far. Before jumping in, you can try it yourself with the free-forever plan that has no limited features.

Give this Live Playground a try to get a better idea.

Export to Aspecto

Here’s how it’s done:

1) Create a free account at www.aspecto.io or log in to your existing account

2) Previously, we set the exporter to send traces to Jaeger. Now, we need to make the following changes and run these commands:

1. Change the exporter to OTEL_EXPORTER_OTLP_HEADERS
2. Specify the Aspecto endpoint OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
3. Replace the {ASPECTO_AUTH}  with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)

OTEL_SERVICE_NAME=my-service 
OTEL_EXPORTER_OTLP_HEADERS=Authorization={ASPECTO_AUTH}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otelcol.aspecto.io:4317 java -javaagent:./opentelemetry-javaagent.jar -jar target/*.jar 

3) Execute a few actions in your project. In your Aspecto account, the main Trace Search view should look something like this:

Using the left Filters pane, we can quickly find any error we have in our system (this error was embedded in the Pet Clinic project by default):

Then, drill down into this trace to investigate and troubleshoot the root cause:

We can also filter traces out by HTTP route name. For example, here we’re filtering out only the http.route=/owners/{ownerId} we mentioned before:

Our ability to quickly solve issues and fix errors in production relies heavily on how we visualize traces and the ease with which we can filter out and deep dive into our data.

That’s about it for this OpenTelemetry Java guide, folks. If you have any questions or issues with any of these steps, feel free to reach out to us via chat or join our OpenTelemetry Slack channel (part of the CNCF Slack).