Distributed Tracing System (Spring Cloud Sleuth + OpenZipkin)

When we build a microservice architecture and the number of services keeps growing, we face the problem of debugging and tracing requests through our entire system. What happened when a user got a 500 error on his request? What service incorrectly processed his request? All these questions can be solved by the Distributed Tracing System. Let's take Spring Cloud Sleuth as an example.

How Spring Cloud Sleuth Works

To trace a request through a distributed system, the concepts TraceID and SpanID are introduced. TraceID is generated when a request enters our system and remains unchanged throughout its path. SpanID changes as the request passes from one service to another. If necessary, it is possible to generate new spans within one service to distinguish business processes.

Traces and spans are passed between systems in HTTP headers. One standard is B3 Propagation, which describes the header format for Distributed Tracing Systems. Context is passed from one service to another via headers of the form X-B3-*. Having received a request with such headers, we can restore the tracing context and continue the execution flow.

The beauty of Sleuth is that it automatically injects the necessary headers to all requests generated by, for example, RestTemplate or Feign, using interceptors. For this to work automatically, you need to declare HTTP clients as Spring beans.

Let's consider the following architecture consisting of 3 microservices:

1. A service for user registration
2. A service responsible for the loyalty program
3. A service for sending notifications to users
   

In this example, let's see how to connect Spring Cloud Sleuth to Spring Boot applications. We will use Feign as the HTTP client. After creating a Gradle project, we will add dependencies to each service: Sleuth, the library for sending data to Zipkin and Feign clients that will allow us to make requests to other services.

User Service

build.gradle:

ext {
    set('springCloudVersion', "2021.0.0")
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.cloud:spring-cloud-starter-sleuth'
    implementation 'org.springframework.cloud:spring-cloud-sleuth-zipkin'
    implementation 'org.springframework.cloud:spring-cloud-starter-openfeign'

    compileOnly 'org.projectlombok:lombok:1.18.22'
    annotationProcessor 'org.projectlombok:lombok:1.18.22'
    testCompileOnly 'org.projectlombok:lombok:1.18.22'
    testAnnotationProcessor 'org.projectlombok:lombok:1.18.22'
}

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
    }
}

We need to enable Feign functionality:

@SpringBootApplication
@EnableFeignClients
public class UserApplication {
    public static void main(String[] args) {
        SpringApplication.run(UserApplication.class, args);
    }
}

User service when a user registers will form HTTP requests to bonus service, send a notification to the user, and return the identifier of the user and the bonus program. Let's write a controller:

@RestController
@RequestMapping("/user")
@RequiredArgsConstructor
public class UserController {

    private final BonusClient bonusClient;
    private final NotificationClient notificationClient;

    @PostMapping("/register")
    RegisterResponse register(@RequestBody RegisterRequest registerRequest) {
        log.info("User registration request received");
        String userId = UUID.randomUUID().toString(); // generate uniq identifier of user
        String bonusId = bonusClient.register(); // register in loyalty program

        notificationClient.send(new NotificationRequest(userId, "email")); // send notification to user

        return new RegisterResponse(userId, bonusId); //return response to user
    }
}

Using Feign, let's describe BonusClient and NotificationClient:

@FeignClient(name = "bonus", url = "http://localhost:8081/bonus")
public interface BonusClient {

    @RequestMapping(value = "/register", method = RequestMethod.POST)
    String register();
}

@FeignClient(name = "notification", url = "http://localhost:8082/notification")
public interface NotificationClient {

    @PostMapping("/send")
    String send(@RequestBody NotificationRequest notificationRequest);
}

Add the identifier of the application to be used by Sleuth:

spring:
  application:
    name: user-service

Let's implement the remaining services in the same way. The project dependencies will be the same.

Bonus Service

For simplicity, write only one controller:

@RestController
@RequestMapping("/bonus")
@Slf4j
public class BonusController {

    @PostMapping("/register")
    @SneakyThrows
    String register() {
        log.info("Bonus program registration request received");
        //e.g. load from DB
        Thread.sleep(new Random().nextInt(100, 1000));
        return UUID.randomUUID().toString();
    }
}

Here we will receive the request, simulate a delay and respond with the generated identifier. Similarly, add the application name to the application.yml:

spring:
  application:
    name: bonus-service

server:
  port: 8081

We also specified a different port so our applications would not conflict with each other when we run them on the same instance.

Notification Service

Let's make the notification service a bit more complicated. It will receive the message and call the external service by simulating a call to the email gateway.

@RestController
@RequiredArgsConstructor
@RequestMapping("/notification")
@Slf4j
public class NotificationController {

    private final ExternalEmailGateway gateway;

    @PostMapping("/send")
    @SneakyThrows
    String send(@RequestBody NotificationRequest notificationRequest) {
        log.info("Notification request received");
        //e.g. load from DB & logic
        Thread.sleep(new Random().nextInt(100, 1000));

        gateway.sendEmail(1);

        return "ok";
    }
}

@FeignClient(name = "externalEmailGateway", url = "https://httpbin.org")
public interface ExternalEmailGateway {
    @GetMapping("/delay/{delay}")
    String sendEmail(@PathVariable int delay);
}

Here we will call https://httpbin.orgto simulate a delayed response.

application.yml:

spring:
  application:
    name: notification-service

server:
  port: 8082

OpenZipkin

By default, our services will try to send trace messages to localhost:9411. If OpenZipkin runs at a different address, you need to specify it in the settings of each service:

spring:
  zipkin:
    base-url: http://<host>:<port>

Run Zipkin using Docker:

docker run -d -p 9411:9411 openzipkin/zipkin

Let's launch all our applications and execute a user registration request:

./gradlew user-service:bootRun 
./gradlew bonus-service:bootRun 
./gradlew notification-service:bootRun

curl -L -X POST 'http://localhost:8080/user/register' \
-H 'Content-Type: application/json' \
--data-raw '{}'

{"userId":"b5d04d9a-0447-4cf5-9002-b10841181e9f","bonusProgramId":"bda336f6-f5ab-4fc9-97b9-58df741d1de2"}

We can now analyze the results by going to localhost:9411

Here you can see the total time of the request and how much time was spent on each of them. If one of the services returns an error, we will see the following:

Also, if you look in the logs, you will see records with the service name, TraceID, and SpanID in square brackets. As you can see within one request, TraceID remains constant, and SpanID changes from service to service:

2022-02-16 08:48:58.617  INFO [user-service,15f8d0cc9c8c9e4a,15f8d0cc9c8c9e4a] 7028 --- [nio-8080-exec-4] c.e.user.controller.UserController       : User registration request received
...
2022-02-16 08:48:58.619  INFO [bonus-service,15f8d0cc9c8c9e4a,d8d79fe594bc4522] 12664 --- [nio-8081-exec-3] c.e.bonus.controller.BonusController     : Bonus program registration request received
...
2022-02-16 08:48:59.366  INFO [notification-service,15f8d0cc9c8c9e4a,3d31b3eb8062cbe0] 25796 --- [nio-8082-exec-3] c.e.n.controller.NotificationController  : Notification request received

Conclusion

Distributed Tracing System is essential for microservice architecture and allows you to analyze problem areas in the system. Spring Cloud Sleuth makes this as easy as possible with almost no added code, and Zipkin is a very easy-to-use analysis tool.

The project code is available on GitHub.