Table of Contents
Note

This guide applies to Steeltoe v3. This component has been removed from v4.

Using Wavefront for app container metrics, distributed tracing, and observability

This tutorial takes you creating a simple Steeltoe app with actuators, logging, and distributed tracing. With that app running you then export the data to a Wavefront account.

Note

For more detailed examples, please refer to the Management solution in the Steeltoe Samples Repository.

Prereq's

You'll need a Wavefront account to complete this guide successfully. Create a 30 day trial, if you don't already have access.

First, clone to accompanying repo that contains all the needed assets

  1. git clone https://github.com/steeltoeoss-incubator/observability.git
    
    cd observability/wavefront
    
  2. Have a look at what things are provided

    ls
    
    Name                    Description
    ----                    ----
    dashboard-template.json Wavefront dashboard configuration
    docker-compose.yml      Docker file to start all containers
    telegraf.conf           Telegraf inputs and output configuration
    
  3. Replace the placeholder YOUR_API_TOKEN in docker-compose.yml with your Wavefront API token. Lean how to retrieve that token here.

Then create a .NET Core WebAPI with the correct Steeltoe dependencies

  1. Create a new ASP.NET Core WebAPI app with the Steeltoe Initializr

  2. Name the project "WavefrontObservability"

  3. Add the "Actuators" dependency

  4. Add the "Dynamic Logger" dependency

  5. Add the "Docker" dependency

  6. Click Generate to download a zip containing the new project

  7. Extract the zipped project and open in your IDE of choice

  8. Add the other needed actuators in startup.cs

    using Steeltoe.Management.Endpoint.Metrics;
    using Steeltoe.Management.Tracing;
    using Steeltoe.Management.Exporter.Tracing;
    
    public class Startup {
      public void ConfigureServices(IServiceCollection services) {
          services.AddPrometheusActuator(Configuration);
          services.AddMetricsActuator(Configuration);
          services.AddDistributedTracing(Configuration);
          services.AddZipkinExporter(Configuration);
      }
    
      public void Configure(IApplicationBuilder app, IWebHostEnvironment env) {
        app.UsePrometheusActuator();
        app.UseMetricsActuator();
          app.UseTracingExporter();
      }
    }
    
  9. Set the actuator path, exposure, and zipkin server address in appsettings.json

    {
      "Logging": {
        "LogLevel": {
          "Default": "Debug",
          "System": "Information",
          "Microsoft": "Information"
        }
      },
      "spring": {
        "application": {
          "name": "Wavefront_demo_app"
        }
      },
      "management": {
        "endpoints": {
          "actuator": {
            "exposure": {
              "include": ["*"]
            }
          },
          "path": "/",
          "cloudfoundry": {
            "validateCertificates": false
          }
        },
        "metrics": {
          "exporter": {
            "cloudfoundry": {
              "validateCertificates": false
            }
          }
        },
        "tracing": {
          "alwaysSample": true,
          "useShortTraceIds ": true,
          "exporter": {
            "zipkin": {
              "endpoint": "http://wavefront-proxy:9411/api/v2/spans",
              "validateCertificates": false
            }
          }
        }
      }
    }
    
  10. Adjust docker-compose.yml to include the path to the .NET project by replacing <ABSOLUTE_PATH_TO_PROJECT> with the absolute path to the folder holding the .csproj file.

Next, deploy everything with docker compose

  1. Build the image using the provided docker-compose file

    docker-compose up -d
    
  2. Confirm everything started successfully by running docker-compose ps and checking the State. Output should look similar to this:

    Name              Command                         State    Ports
    -----------------------------------------------------------------------------------------------------------------------------
    steeltoe-app      dotnet Grafana_Observabili ...   Up      0.0.0.0:80->80/tcp
    telegraf          /entrypoint.sh --config=/e ...   Up      8092/udp, 8094/tcp, 8125/udp
    wavefront-proxy   /bin/bash /run.sh                Up      0.0.0.0:2878->2878/tcp, 3878/tcp, 4242/tcp, 0.0.0.0:9411->9411/tcp
    

Finally use Wavefront to view the metrics and traces being fed in

  1. Navigate to your Wavefront instance and the "Dashboards" area Wavefront - Dashboard

  2. Create a new dashboard by clicking "Create Dashboard" button Wavefront - Create Dashboard

  3. Now locate the "JSON" link toward the top of the window and click Wavefront - JSON Link

  4. A popup window will be shown. Find the "Tree" drop down in the blue bar and change to "Code" view. Wavefront - Tree Drop Down

  5. Clear the pre-loaded JSON in the window and copy the contents of dashboard-template.json to the window.

  6. Click the "Accept" link to close the window and let Wavefront parse the JSON.

  7. Save your new dashboard by clicking the "Save" link at the top. A popup window will ask you to name your new dashboard and finish saving. Wavefront - Save Dashboard

    > [!NOTE]
    > You must save the new dashboard for your application name to show and things start receiving data.
    
  8. To make sure the correct traces are being used, in the top bar make sure the "application" is set to "Zipkin" and the "service" is set to the name of your application. Wavefront - Application and Service

  9. Done! Everything else should be pre-loaded for you. As the application runs in Docker, the metrics are shown.

Further learning in Wavefront

  1. Notice the chart "Top Requests" in the bottom left corner. Click one of the URLs to view all of the traces coming in for your Steeltoe application. If you had other microservices they would also appear here and you can even configure your traces to deliver logs as well. Wavefront - Top Requests

  2. You can see your traces on the left and sort them, on the right you will see other metrics as well.

  3. To get even deeper, select a trace in the left and then find the "traceid" in the "Critical Path Breakdown" area. Copy that value. Wavefront - Get Trace ID

  4. Then "Add Filter" from the link at the top and paste that traceid value in the "TraceId" text box. Now click the "Add Trace Id" button. Your view will refresh to show all the essentials of that specific trace! Wavefront - Trace ID

  5. Creating filters gives you all kinds of different ways to view data. In the "Add Filter" window clear the traceid value and click the "</>" button. This switches your filter view to thre query builder. Here are a few queries to get started:

  6. Get the last 100 traces: limit(100, traces("Zipkin.wavefront_demo_app.httpclient:/api/v1/spans")) Find traces that took longer than 30 seconds: limit(100, highpass(30ms, traces("Zipkin.wavefront_demo_app.httpclient:/api/v1/spans")))

If you would like to learn more about the query builder have a look at Wavefront docs.