Tracing Specifics – Know your System with OpenTelmetry

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Spotify | TuneIn | RSS

Get a behind the scenes intro to some of the interesting conversations we have before we even get into the content. We’ll be jumping into the meat of this episode and looking at the specifics of tracing using OpenTelemetry. Before we do that though, we should probably find out what special 2-liter containers Outlaw uses that can somehow trap the bubbles for more than 24 hours after opening, find out if Joe is alone in liking flat carbonated drinks, and maybe Allen has fallen off his rocker for suggesting that the ONE THING that is metric in the USA should be converted to empirical measurements. Maybe leave a comment on the episode to join in the fun. To see the full show notes and/or leave a comment, head over to…
https://www.codingblocks.net/episode217

OpenTelemetry Diving In

TracerProvider

This is a factory for Tracers

• In most apps, the TracerProvider is initialized once and typically has the same lifecycle as the application
• Also includes a Resource and Exporter initialization
• Typically, the first step in setting up OpenTelemetry and in some language implementations a global provider is set up for you

Tracer

These are created by a trace provider and creates spans with more information about what’s happening with the request

Trace Exporter

These send traces to a consumer

• Can be sent to standard out if you’re just debugging
• Can be an OpenTelemetry Collector
• Can be any other open source or other consumer

Context Propagation

• The heart of distributed tracing as it takes and correlates multiple spans

Context

• The context contains information that allows spans to be correlated
  • Example is Service A calling Service B
  • Service A will have a trace id as well as its own span id
  • Service B will reuse that same trace id so the entire trace can be correlated, and Service B will also have its own unique span id but the parent id will point to the span id from Service A, so again this correlates the parent / child hierarchy of the spans

Propagation

• This is what moves the context between services and processes
• Serializes / deserializes the context object and provides the relevant trace information to be carried from one service or process to the next
  • This is usually handled by instrumentation libraries but can be done manually via propagation APIs
• There are a number of formats that OpenTelemetry supports, but the default is the W3C’s TraceContext 
  https://www.w3.org/TR/trace-context/
  • The context objects are stored within a span

Spans

• These represent a unit of work or an operation
• Spans are the building blocks of traces
• Information included in a span
  • Name
  • Parent span id (empty for a root span)
  • Span context
  • Attributes
  • Span events
  • Span links
  • Span status
• Spans can be nested (parent/child) – any child span should be a sub-operation

Span Context

• IMMUTABLE
• Contains the following
  • Trace id
  • Unique span id
  • Trace flags – binary encoding containing information about the trace
  • Trace state – list of key-value pairs that can carry vendor-specific trace information
  • This data sits alongside distributed context and baggage

Attributes

• Key/value pairs used to carry information about the operation it’s tracking
  • Example used – shopping cart may store the user id, item id added to cart and a cart id
• Keys must be non-null
• Values must be non-null strings, a boolean, floating point value, integer or an array of any of those
• There exists semantic attributes for well-known attributes that should be followed to help standardize across systems
  https://opentelemetry.io/docs/specs/otel/trace/semantic_conventions/
  • Some examples of “general attributes”
    • server.address
    • server.port
  • Some “database” examples
    • db.system
    • db.connection_string
      

Span Events

• A structured log message or annotation on a span, usually used for a meaningful point in time in the span’s duration
  • Example – two web browser scenarios
    • Tracking a page load – what you’d use a span for because it has a start and end time
    • Denoting when a page becomes interactive – this is a singular point in time in the life of that span above

Span Links

• Allows you to associate a span with one or more other spans – indicating a causal relationship
• An example is when you have a system that queues actions based off other actions in an asynchronous manner – you don’t know when the queued action will start so you create a span link that can correlate these spans over time when the asynchronous event occurs
• Span links are optional but can be a good way to associate trace spans with one another

Span Status

• Status is attached to a span – one of three values
  • Unset – this is what you usually want to do
  • Ok – the back-end that processes the spans should set this for you as a final status
  • Error – usually set whenever you handle an exception/error

Span Kind

• When a span is created, it is created as one of the following types
  • Client – represents synchronous, outgoing remote calls – this doesn’t mean that it’s not asynchronous technically, it just means it’s not being queued for later processing
  • Server – represents an incoming synchronous call such as an HTTP request
  • Internal – operations that do not cross a process boundary – things like instrumenting a function
  • Producer – represent the creation of a job that may be asynchronously processed later – things like messages sent to queues or handling of events
  • Consumer – represents the processing of a job that was created by a producer and potentially starts well after the producer span has ended
• These types provide a hint to the backend span processor as to how these spans should be assembled
• Based on the OpenTelemetry specification
  • The parent span of a server span is usually a remote client span
  • The parent of a consumer span is always a producer span
• If no type is provided, it is assumed to be an internal span

Resources we Like

https://opentelemetry.io 
https://opentelemetry.io/docs/demo/architecture/ 
https://opentelemetry.io/docs/demo/screenshots/

Tip of the Week

• Interested in tldr.sh but want a faster, single binary version? Check out “tealdeer” – same info, faster implimentation in Rust. Thanks for the tip, Aleksander Andrzejewski! 
  https://github.com/dbrgn/tealdeer
• Don’t waste your time hitting ctrl-alt-delete and then selecting the task manager. You gotta go fast, just use ctrl-shift-escape next time and it’ll pop right up! Thanks for the tip in the comments Mark Crowley!
• Sea of Stars is a new, retro, video game that hearkens back to the golden era of ol’ Super Nintendo/Famicom games like Chronotrigger and Secret of Mana. No grindy battlepasses, weird mobile ads, or zany gpu requirements…just a good game. Bonus, it features new songs from Uasunori Mitsuda, who is well known for their music work with a bunch of classic RPG series including Final Fantasy, Chronotrigger, Xenogears. If any of those names make your heart jump out of your chest then you owe it to yourself to give it a look. 
  https://seaofstarsgame.co/
• Fuzzing tool for a kubernetes cluster 
  https://google.github.io/clusterfuzz/
• Foundational C# Certification – Partnership with freeCodeCamp / Microsoft 
  https://devblogs.microsoft.com/dotnet/announcing-foundational-csharp-certification/
• Leader board for Chat Bots 
  https://chat.lmsys.org/
• There’s an AI for that 
  https://theresanaiforthat.com/
• Use kubectl get deployments -o wide to see which image tag your containers are using. 
  https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get
• Did you know you can exec into a pod without looking up the generated pod name? Yep! You can just exec into a deployment and it will pick a pod for you, works for logs too!
  • kubectl exec -it deploy/your-pod-name — bash
  • kubectl logs deploy/your-pod-name