Skip to content

jonaskahn/nodejs-open-telemetry

BranchesTags

Repository files navigation

Service and Cronjob Demo

Vietnamese

Generated by Claude AI

This project demonstrates a JavaScript application with 5 services and 2 cronjobs using CommonJS modules. The services call each other and the cronjobs utilize these services to perform scheduled tasks. It also includes OpenTelemetry instrumentation for monitoring function calls.

Project Structure

├── src/
│   ├── services/           # Service modules
│   │   ├── userService.js  # User management
│   │   ├── dataService.js  # Data operations
│   │   ├── notificationService.js  # Notifications
│   │   ├── loggingService.js  # Logging
│   │   └── authService.js  # Authentication
│   ├── jobs/               # Cronjob modules
│   │   ├── dataBackupJob.js  # Backup job (with OpenTelemetry)
│   │   └── reportGenerationJob.js  # Report generation job
│   ├── middleware/
│   │   └── telemetry.js    # OpenTelemetry configuration
│   └── index.js            # Application entry point
├── docker-compose.yml      # Docker configuration for Jaeger
└── package.json

Services

  1. User Service - Manages user-related operations
  2. Data Service - Handles data operations and storage
  3. Notification Service - Sends notifications to users
  4. Logging Service - Provides logging functionality
  5. Auth Service - Handles authentication and authorization

Cronjobs

  1. Data Backup Job - Runs daily at 2:00 AM to create data backups (Instrumented with OpenTelemetry)
  2. Report Generation Job - Runs every Monday at 8:00 AM to generate weekly reports

Getting Started

Starting Jaeger for Telemetry Visualization

Before running the application, start the Jaeger container:

docker-compose up -d

This will start Jaeger at http://localhost:16686

Installation

npm install

Running the Application

# Run with telemetry enabled (default)
npm start

# Run with telemetry explicitly enabled
npm run start:with-telemetry

# Run with telemetry disabled
npm run start:no-telemetry

After running the application, you can see the OpenTelemetry traces in the Jaeger UI at http://localhost:16686.

Integrating OpenTelemetry Into Your NodeJS Project

This section provides guidelines for implementing tracing in your existing Node.js application.

Setting Up OpenTelemetry

  1. Install the required packages:

    npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources @opentelemetry/semantic-conventions

  2. Configure OpenTelemetry in your project:

    • Use the telemetry.js middleware as a reference
    • Initialize the SDK with your service name and version
    • Configure the exporter to send traces to your backend (Jaeger, Zipkin, etc.)

Enabling or Disabling Telemetry

OpenTelemetry can be dynamically enabled or disabled in your application. When disabled, the tracing functions still work but simply pass through without creating spans, ensuring your application code works without modification regardless of telemetry state.

Using Environment Variables with dotenv

The application uses dotenv to load configuration from .env files:

# Install dotenv if not already installed
npm install dotenv

Create a .env file in your project root:

# OpenTelemetry Configuration
TELEMETRY_ENABLED=true

# Service Configuration
SERVICE_NAME=my-service-name
SERVICE_VERSION=1.0.0

# OpenTelemetry Exporter Configuration
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces

To disable telemetry, you can:

  1. Set TELEMETRY_ENABLED=false in your .env file
  2. Use the environment variable when running the application:
# Linux/Mac
TELEMETRY_ENABLED=false npm start

# Windows
set TELEMETRY_ENABLED=false && npm start
  1. Use the provided convenience scripts:
# Using npm scripts
npm run start:no-telemetry  # Telemetry disabled
npm run start:with-telemetry  # Telemetry enabled

# Or shell scripts
./run-without-telemetry.sh  # Linux/Mac
run-without-telemetry.cmd  # Windows

Testing Your Telemetry Configuration

You can verify your telemetry configuration with the included test script:

npm run test:config

This will show:

  • Current telemetry settings from environment variables
  • Whether telemetry is enabled or disabled
  • Initialization status
  • Instructions for running with telemetry disabled

Runtime API for Telemetry Control

In addition to environment variables, you can control telemetry at runtime:

// Disable telemetry globally
telemetry.setEnabled(false);

// Check if telemetry is enabled
if (telemetry.isEnabled()) {
  console.log('Telemetry is currently enabled');
} else {
  console.log('Telemetry is currently disabled');
}

Creating a Telemetry Instance with Initial State

// When creating a new telemetry instance:
const serviceName = 'my-service';
const serviceVersion = '1.0.0';
const telemetryEnabled = process.env.TELEMETRY_ENABLED !== 'false';

const myTelemetry = new Telemetry(serviceName, serviceVersion, telemetryEnabled);

Behavior When Disabled

When telemetry is disabled:

  1. No spans will be created or sent to the backend
  2. All tracing methods (wrapWithSpan, startActiveSpan) continue to work but simply execute the wrapped function
  3. Dummy span objects are provided to callbacks to maintain API compatibility
  4. No performance overhead from tracing is incurred
  5. Your application code continues to work without modification

This approach allows you to:

  • Disable telemetry in production environments if needed
  • Create environment-specific telemetry configurations
  • Toggle telemetry at runtime for debugging purposes
  • Implement feature flags for telemetry

Converting Regular Methods to Traced Methods

To add tracing to existing methods, follow these patterns:

Basic Method Tracing

// Before: Regular method
function myFunction(param1, param2) {
  // Function implementation
  return result;
}

// After: Method with tracing
function myFunction(param1, param2) {
  return telemetry.wrapWithSpan('myFunction', { param1, param2 }, async () => {
    // Function implementation
    return result;
  });
}

Maintaining Trace Context with Nested Functions

To maintain the parent-child relationship in traces when using nested function calls:

// Parent function with tracing
function parentFunction(params) {
  return telemetry.wrapWithSpan('parentFunction', { params }, async () => {
    // Call preparation logic
    const result1 = await _childFunction1();
    const result2 = await _childFunction2(result1);
    return result2;
  });
}

// Child function 1 with tracing - maintains the same trace
function _childFunction1() {
  return telemetry.wrapWithSpan('_childFunction1', {}, async () => {
    // Implementation
    return result;
  });
}

// Child function 2 with tracing - maintains the same trace
function _childFunction2(input) {
  return telemetry.wrapWithSpan('_childFunction2', { input }, async () => {
    // Implementation
    return result;
  });
}

Best Practices for Hierarchical Tracing

  1. Use Consistent Naming Conventions:

    • Use descriptive span names that reflect the operation being performed
    • Prefix internal/private functions with underscore (_) to identify them in traces

  2. Add Relevant Attributes to Spans:

    • Include function parameters as span attributes for better context
    • Exclude sensitive information like passwords or tokens
    • Add operation-specific attributes (e.g., user IDs, task IDs)

  3. Error Handling in Spans:

    function myFunction(params) {
  return telemetry.wrapWithSpan('myFunction', { params }, async span => {
    try {
      // Implementation
      return result;
    } catch (error) {
      // Record error in the span
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error.message,
      });
      // You can also add error details as attributes
      span.setAttribute('error.type', error.name);
      span.setAttribute('error.stack', error.stack);
      throw error; // Re-throw the error
    }
  });
}

  4. Execution IDs for Correlation:

    • Generate a unique ID for each top-level operation
    • Pass this ID through the entire call chain
    • Add it as an attribute to all spans for easy correlation

Real-World Example: Job with Nested Traces

See the implementation in src/jobs/reportGenerationJob.js and src/jobs/dataBackupJob.js for complete examples of hierarchical tracing with multiple levels of nested function calls.

The notificationService.js file demonstrates a complex service with 5 levels of nested calls, all properly traced with OpenTelemetry.

OpenTelemetry Instrumentation

This project demonstrates OpenTelemetry usage for the following:

  1. Function call tracing - The Data Backup Job is instrumented to trace all function calls
  2. Span attributes - Various attributes are added to spans to provide context
  3. Error tracking - Errors are properly recorded in spans

How it Works

The services are designed to work together:

  • User Service uses Auth Service to verify user access and Logging Service to log operations
  • Data Service uses Notification Service to send updates and Logging Service to log operations
  • Notification Service uses Logging Service to track notifications
  • Auth Service uses Logging Service to log authentication events

The cronjobs demonstrate scheduled tasks:

  • Data Backup Job uses Data Service to create backups and Notification Service to send alerts
  • Report Generation Job uses User Service and Data Service to gather data, then Notification Service to send the report

Testing

For testing purposes, you can modify the cronjob schedules in their respective configuration objects:

  • In src/jobs/dataBackupJob.js, update the CONFIG.schedule value to run more frequently
  • In src/jobs/reportGenerationJob.js, update the CONFIG.schedule value to run more frequently

Design Principles

This project demonstrates:

  1. Modular design with CommonJS exports
  2. Function-based implementation (no classes)
  3. Service dependencies and cross-service calls
  4. Scheduled tasks via cronjobs
  5. OpenTelemetry instrumentation for observability

Converting Existing Nested Methods While Maintaining Span Context

When integrating OpenTelemetry into an existing codebase with deeply nested method calls, follow these steps to maintain proper span context:

Before: Original Nested Methods

// Original code with nested method calls
function processOrder(order) {
  validateOrder(order);
  const items = prepareItems(order.items);
  const payment = processPayment(order.payment);
  const shipment = arrangeShipping(items, order.address);
  sendConfirmation(order.customer, shipment);
  return { orderId: order.id, status: 'completed' };
}

After: Traced Nested Methods

// Primary function with tracing
function processOrder(order) {
  const executionId = uuid.v4(); // Generate unique execution ID

  return telemetry.startActiveSpan(
    'processOrder',
    {
      attributes: {
        'order.id': order.id,
        'execution.id': executionId,
      },
    },
    async span => {
      try {
        // All nested function calls will inherit this span context
        await _validateOrder(order);
        const items = await _prepareItems(order.items);
        const payment = await _processPayment(order.payment);
        const shipment = await _arrangeShipping(items, order.address);
        await _sendConfirmation(order.customer, shipment);

        return { orderId: order.id, status: 'completed' };
      } catch (error) {
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: error.message,
        });
        throw error;
      }
    }
  );
}

// Nested functions with their own spans, maintaining the parent context
function _validateOrder(order) {
  return telemetry.startActiveSpan(
    '_validateOrder',
    { attributes: { 'order.id': order.id } },
    async () => {
      // Implementation
    }
  );
}

function _prepareItems(items) {
  return telemetry.startActiveSpan(
    '_prepareItems',
    { attributes: { 'items.count': items.length } },
    async () => {
      // Implementation
      return processedItems;
    }
  );
}

// And so on for other methods...

Key Points When Refactoring Existing Code:

  1. Start with the Outermost Function:

    • Begin by wrapping the top-level function with startActiveSpan
    • This creates the parent span that all nested calls will inherit from

  2. Convert Direct Function Calls to Private Methods:

    • Refactor inline function calls into separate private methods (prefixed with _)
    • This makes it easier to add tracing to each step

  3. Use Consistent Context Propagation:

    • Ensure each nested function uses startActiveSpan rather than just wrapWithSpan
    • This properly maintains the parent-child relationship in the trace

  4. Preserve the Original API:

    • Keep the original function signatures when adding tracing
    • This ensures backward compatibility with existing code

  5. Handle Asynchronous Operations:

    • Convert synchronous functions to asynchronous (Promise-based) when adding tracing
    • Use async/await for cleaner code flow

For examples of converted code, see how we've refactored the reportGenerationJob.js and implemented the nested spans in notificationService.js.

Examples From This Project

This section provides concrete before-and-after examples from the actual codebase, demonstrating how we've applied OpenTelemetry tracing to real-world scenarios.

Example 1: Data Backup Job

Before Adding Telemetry:

// Original version without telemetry
function performBackup() {
  try {
    // Generate a simple ID for logging
    const backupId = generateSimpleId();

    loggingService.info(`Starting data backup: ${backupId}`);

    // Create the backup directly
    const backup = dataService.createDataBackup();

    // Notify admins about the backup
    notificationService.sendNotification('admin', `Data backup completed. Backup ID: ${backup.id}`);

    loggingService.info(`Backup ${backupId} completed successfully`);
    return backup;
  } catch (error) {
    loggingService.error(`Backup failed: ${error.message}`);

    // Send error notification
    notificationService.sendNotification('admin', `Data backup failed: ${error.message}`, 'email');

    throw error;
  }
}

// Schedule the job
function initBackupJob() {
  const schedule = '0 2 * * *'; // 2:00 AM daily

  cron.schedule(schedule, () => {
    performBackup();
  });

  loggingService.info(`Scheduled backup job: ${schedule}`);
}

After Adding Telemetry:

// ------------------------------
// PUBLIC API (Entry Points)
// ------------------------------

/**
 * Initialize and schedule the backup job
 * This is the main entry point for setting up the job
 */
const initBackupJob = telemetry.wrapWithSpan(_initBackupJob, 'initBackupJob', {
  'job.name': 'dataBackupJob',
  'job.type': 'cron',
});

/**
 * Perform a backup operation
 * This can be called manually or by the scheduled job
 */
function performBackup(executionId) {
  const execId = executionId || uuidv4();
  return telemetry.wrapWithSpan(() => _performBackup(execId), `performBackup.${execId}`, {
    'backup.type': 'scheduled',
    'backup.job': 'dataBackupJob',
    'backup.execution_id': execId,
  })();
}

// ------------------------------
// IMPLEMENTATION DETAILS
// ------------------------------

/**
 * Private implementation of job initialization
 */
function _initBackupJob() {
  if (!CONFIG.enabled) {
    loggingService.info('Data backup job is disabled');
    return false;
  }

  loggingService.info(`Scheduling data backup job with schedule: ${CONFIG.schedule}`);

  const job = cron.schedule(CONFIG.schedule, () => {
    const executionId = uuidv4();
    const executionTracer = telemetry.getTracer(`dataBackupJob.${executionId}`);

    executionTracer.startActiveSpan('backupJob.execution', span => {
      try {
        span.setAttribute('backup.scheduled_time', new Date().toISOString());
        span.setAttribute('backup.cron_pattern', CONFIG.schedule);
        span.setAttribute('backup.execution_id', executionId);

        const result = performBackup(executionId);

        span.setAttribute('backup.success', result.success);
        if (result.backupId) {
          span.setAttribute('backup.id', result.backupId);
        }
        if (result.duration) {
          span.setAttribute('backup.duration_seconds', result.duration);
        }

        span.end();
        return result;
      } catch (error) {
        span.recordException(error);
        span.setStatus({ code: SpanStatusCode.ERROR });
        span.end();
        throw error;
      }
    });
  });

  return job;
}

/**
 * Private implementation of backup operation
 */
function _performBackup(executionId) {
  try {
    loggingService.info(`Starting scheduled data backup [execution: ${executionId}]...`);

    const startTime = new Date();
    const backup = dataService.createDataBackup();
    const endTime = new Date();
    const duration = (endTime - startTime) / 1000;

    loggingService.info(
      `Backup completed successfully in ${duration} seconds. Backup ID: ${backup.id}`
    );

    notificationService.sendNotification(
      CONFIG.adminUser,
      `Data backup completed successfully. Backup ID: ${backup.id}`
    );

    return {
      success: true,
      backupId: backup.id,
      timestamp: backup.timestamp,
      duration,
      executionId,
    };
  } catch (error) {
    loggingService.error(`Backup failed [execution: ${executionId}]: ${error.message}`, {
      error,
    });

    notificationService.sendNotification(
      CONFIG.adminUser,
      `Data backup failed: ${error.message}`,
      'email'
    );

    return {
      success: false,
      error: error.message,
      timestamp: new Date(),
      executionId,
    };
  }
}

Key Improvements:

  1. Execution Traceability: Each backup operation now has a unique executionId that flows through the entire execution chain
  2. Structured Data: The return values include detailed metadata about the operation
  3. Performance Metrics: We now track and record the duration of each backup operation
  4. Contextual Span Attributes: Each span includes relevant attributes that make troubleshooting easier
  5. Proper Error Handling: Errors are properly captured and recorded in the telemetry system

Example 2: Notification Service

The notification service demonstrates a complex flow with 5 levels of nested operations.

Before Adding Telemetry:

// Original notification sending process
function sendNotification(userId, message, channel = 'email') {
  try {
    const notificationId = generateId();

    // Step 1: Prepare content
    loggingService.info(`Preparing content for notification to ${userId}`);
    const preferences = getUserPreferences(userId);
    const formattedMessage = formatMessage(message, preferences);

    // Step 2: Create notification record
    const notification = {
      id: notificationId,
      userId,
      message: formattedMessage,
      channel,
      status: 'processing',
      timestamp: new Date(),
    };

    // Step 3: Get user device information
    const deviceInfo = getUserDeviceInfo(userId);

    // Step 4: Deliver the notification
    loggingService.info(`Delivering notification ${notificationId} via ${channel}`);

    if (channel === 'push') {
      sendPushNotification(userId, notification, deviceInfo);
    } else if (channel === 'email') {
      sendEmailNotification(userId, notification);
    } else {
      sendDefaultNotification(userId, notification);
    }

    // Step 5: Update status
    notification.status = 'sent';
    notification.deliveredAt = new Date();

    loggingService.info(`Notification ${notificationId} sent successfully`);
    return notification;
  } catch (error) {
    loggingService.error(`Failed to send notification: ${error.message}`);
    throw error;
  }
}

// Supporting functions
function getUserPreferences(userId) {
  // Implementation
}

function formatMessage(message, preferences) {
  // Implementation
}

function getUserDeviceInfo(userId) {
  // Implementation
}

function sendPushNotification(userId, notification, deviceInfo) {
  // Implementation
}

After Adding Telemetry:

// ------------------------------
// PUBLIC API (Entry Points)
// ------------------------------

/**
 * Send a notification to a user
 * This is the main entry point for notifications
 */
const sendNotification = telemetry.wrapWithSpan(
  _sendNotification,
  'notificationService.sendNotification',
  { 'notification.operation': 'send' }
);

// ------------------------------
// IMPLEMENTATION DETAILS (Top-down hierarchy)
// ------------------------------

/**
 * Level 1 - Main notification function implementation
 */
function _sendNotification(userId, message, channel = 'email') {
  try {
    const notificationId = uuidv4();
    const timestamp = new Date();
    const content = await _prepareNotificationContent(userId, message, channel);

    notifications[notificationId] = {
      id: notificationId,
      userId,
      message: content.formattedMessage,
      originalMessage: message,
      channel,
      timestamp,
      status: 'processing',
    };

    loggingService.info(`Notification created for user ${userId} via ${channel}: ${notificationId}`);

    const deliveryResult = await _deliverNotification(
      userId,
      notifications[notificationId],
      channel
    );

    notifications[notificationId].status = 'sent';
    notifications[notificationId].deliveredAt = deliveryResult.timestamp;

    loggingService.info(`Notification sent to user ${userId} via ${channel}: ${notificationId}`);

    return notifications[notificationId];
  } catch (error) {
    loggingService.error(`Failed to send notification: ${error.message}`);
    throw error;
  }
}

/**
 * Level 2 - Prepare notification content
 */
function _prepareNotificationContent(userId, message, channel) {
  return new Promise(resolve => {
    setTimeout(async () => {
      loggingService.info(`Preparing notification content for user ${userId}`);
      const preferences = await _getUserNotificationPreferences(userId);
      const formattedMessage = preferences.useHtml ? `<div>${message}</div>` : message;

      resolve({
        userId,
        originalMessage: message,
        formattedMessage,
        channel,
        timestamp: new Date(),
        templateId: preferences.templateId,
        includeFooter: preferences.includeFooter,
      });
    }, simulatedDelay());
  });
}

/**
 * Level 2 - Deliver notification
 */
function _deliverNotification(userId, notification, channel) {
  return new Promise((resolve, reject) => {
    loggingService.info(`Delivering notification via ${channel} to user ${userId}`);

    firebaseService.storeNotification(userId, notification.id)
      .then(() => {
        if (channel === 'push') {
          return firebaseService.sendPushNotification(userId, notification);
        }
        return Promise.resolve();
      })
      .then(() => firebaseService.trackNotificationStatus(notification.id, 'delivered'))
      .then(() => resolve(notification))
      .catch(error => {
        loggingService.error(`Failed to deliver notification: ${error.message}`);
        reject(error);
      });
  });
}

/**
 * Level 3 - Get user preferences
 */
function _getUserNotificationPreferences(userId) {
  return new Promise(resolve => {
    setTimeout(async () => {
      loggingService.info(`Retrieving notification preferences for user ${userId}`);
      const deviceInfo = await _getUserDeviceInfo(userId);

      resolve({
        userId,
        templateId: `template-${Math.floor(Math.random() * 5) + 1}`,
        useHtml: deviceInfo.supportHtml,
        includeFooter: true,
        deliveryPriority: 'high',
        timestamp: new Date(),
      });
    }, simulatedDelay());
  });
}

/**
 * Level 4 - Get device info
 */
function _getUserDeviceInfo(userId) {
  return new Promise(resolve => {
    setTimeout(async () => {
      loggingService.info(`Retrieving device information for user ${userId}`);
      const tokensResult = await firebaseService.getUserDeviceTokens(userId);

      resolve({
        userId,
        deviceType: tokensResult.platforms[0],
        supportHtml: tokensResult.platforms.includes('ios'),
        lastActive: new Date(Date.now() - 24 * 60 * 60 * 1000),
        tokens: tokensResult.tokens,
        timestamp: new Date(),
      });
    }, simulatedDelay());
  });
}

Key Improvements:

  1. Hierarchical Tracing: Each level of the notification process is now visible as a separate span
  2. Asynchronous Operations: All operations are properly handled with Promises and async/await
  3. Clearly Defined Steps: Each step in the process is now a separate function with its own span
  4. Better Visibility: The process flow is much clearer and easier to understand
  5. Error Boundaries: Errors are properly captured at each level and bubble up appropriately
  6. Detailed Logging: Each step logs appropriate information for troubleshooting

Key Insights from the Examples

When comparing the before and after versions, several important patterns emerge:

  1. Function Decomposition: The code is broken down into smaller, more focused functions
  2. Explicit Dependencies: Dependencies between functions are more clearly defined
  3. Consistent Error Handling: Error handling follows a consistent pattern
  4. Traceability: Each operation has a unique identifier for correlation
  5. Performance Metrics: Time-consuming operations are measured and recorded
  6. Context Propagation: The trace context flows through the entire call chain

These examples demonstrate how to apply telemetry to existing code without completely rewriting it. The core business logic remains largely unchanged, but the observability is greatly enhanced.

Converting Traditional Code to Telemetry-Enabled Code: A Practical Guide

This section provides a straightforward approach to adding OpenTelemetry to your existing codebase with minimal disruption.

Step 1: Identify Key Functions to Trace

Start by identifying the most important functions in your application:

  • Entry points (API endpoints, event handlers)
  • Long-running operations
  • Functions that interact with external services
  • Error-prone areas of your code

Step 2: Create Private Implementation Functions

For each function you want to trace:

  1. Rename the original function by adding an underscore prefix
  2. Keep the implementation intact initially
// Original function
function processOrder(order) {
  // Implementation
}

// Step 1: Rename with underscore prefix
function _processOrder(order) {
  // Same implementation
}

Step 3: Create the Public Traced Function

Create a new function with the original name that wraps the private implementation with tracing:

// New public function with same name as original
function processOrder(order) {
  const executionId = uuidv4(); // Generate tracking ID

  return telemetry.wrapWithSpan(
    'processOrder',
    {
      'order.id': order.id,
      'execution.id': executionId,
    },
    () => _processOrder(order)
  );
}

Step 4: Add Hierarchical Tracing to Nested Functions

If your function calls other functions:

  1. Extract those calls into separate private functions
  2. Add trace spans to each extracted function
  3. Maintain context propagation
// Step 1: Original function with nested calls
function _processOrder(order) {
  // Validate order
  if (!order.items || order.items.length === 0) {
    throw new Error('Order must have items');
  }

  // Process payment
  const paymentResult = processPayment(order.payment);

  // Create shipment
  const shipment = createShipment(order.items, order.address);

  return { orderId: order.id, status: 'completed' };
}

// Step 2: Extract nested calls
function _validateOrder(order) {
  if (!order.items || order.items.length === 0) {
    throw new Error('Order must have items');
  }
  return true;
}

function _processPayment(payment) {
  // Implementation
  return paymentResult;
}

function _createShipment(items, address) {
  // Implementation
  return shipment;
}

// Step 3: Update the main function to use extracted functions
function _processOrder(order) {
  _validateOrder(order);
  const paymentResult = _processPayment(order.payment);
  const shipment = _createShipment(order.items, order.address);
  return { orderId: order.id, status: 'completed' };
}

// Step 4: Add tracing to each extracted function
function _validateOrder(order) {
  return telemetry.wrapWithSpan('_validateOrder', { 'order.id': order.id }, () => {
    if (!order.items || order.items.length === 0) {
      throw new Error('Order must have items');
    }
    return true;
  });
}

// Similar for other extracted functions

Step 5: Convert to Async/Promise Pattern

Convert synchronous functions to use promises for better tracing:

// Before: Synchronous
function _processOrder(order) {
  _validateOrder(order);
  const paymentResult = _processPayment(order.payment);
  const shipment = _createShipment(order.items, order.address);
  return { orderId: order.id, status: 'completed' };
}

// After: Async/Promise
async function _processOrder(order) {
  await _validateOrder(order);
  const paymentResult = await _processPayment(order.payment);
  const shipment = await _createShipment(order.items, order.address);
  return { orderId: order.id, status: 'completed' };
}

Step 6: Add Error Handling

Add proper error handling to capture errors in spans:

// Public function with error handling
function processOrder(order) {
  const executionId = uuidv4();

  return telemetry.wrapWithSpan(
    'processOrder',
    { 'order.id': order.id, 'execution.id': executionId },
    async span => {
      try {
        const result = await _processOrder(order);
        return result;
      } catch (error) {
        // Record error in the span
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: error.message,
        });

        // Record error details
        span.recordException(error);

        // Re-throw the error
        throw error;
      }
    }
  );
}

Step 7: Export Only Public Functions

In your module exports, only include the traced public functions:

module.exports = {
  processOrder,
  // Other public functions...
};

Practical Conversion Example

Here's a complete example showing the transformation of a simple data service:

Before: Traditional Code

// dataService.js
const db = require('./database');
const loggingService = require('./loggingService');

function getUser(userId) {
  loggingService.info(`Getting user data for ID: ${userId}`);

  try {
    const userData = db.query('SELECT * FROM users WHERE id = ?', [userId]);

    if (!userData) {
      throw new Error(`User not found: ${userId}`);
    }

    return {
      id: userData.id,
      name: userData.name,
      email: userData.email,
      // Other user data
    };
  } catch (error) {
    loggingService.error(`Failed to get user ${userId}: ${error.message}`);
    throw error;
  }
}

function updateUserProfile(userId, profileData) {
  loggingService.info(`Updating profile for user: ${userId}`);

  try {
    validateProfileData(profileData);

    db.execute('UPDATE users SET name = ?, email = ? WHERE id = ?', [
      profileData.name,
      profileData.email,
      userId,
    ]);

    const updatedUser = getUser(userId);

    loggingService.info(`Profile updated for user: ${userId}`);
    return updatedUser;
  } catch (error) {
    loggingService.error(`Failed to update profile for user ${userId}: ${error.message}`);
    throw error;
  }
}

function validateProfileData(profileData) {
  if (!profileData.name) {
    throw new Error('Name is required');
  }

  if (!profileData.email || !profileData.email.includes('@')) {
    throw new Error('Valid email is required');
  }

  return true;
}

module.exports = {
  getUser,
  updateUserProfile,
};

After: Telemetry-Enabled Code

// dataService.js
const db = require('./database');
const loggingService = require('./loggingService');
const telemetry = require('../middleware/telemetry');
const { SpanStatusCode } = require('@opentelemetry/api');
const { v4: uuidv4 } = require('uuid');

function _getUser(userId, requestId) {
  loggingService.info(`Getting user data for ID: ${userId}`);

  try {
    const userData = db.query('SELECT * FROM users WHERE id = ?', [userId]);

    if (!userData) {
      throw new Error(`User not found: ${userId}`);
    }

    return {
      id: userData.id,
      name: userData.name,
      email: userData.email,
      // Other user data
    };
  } catch (error) {
    loggingService.error(`Failed to get user ${userId}: ${error.message}`);
    throw error;
  }
}

function getUser(userId) {
  const requestId = uuidv4();

  return telemetry.wrapWithSpan(
    'dataService.getUser',
    { 'user.id': userId, 'request.id': requestId },
    () => _getUser(userId, requestId)
  );
}

async function _updateUserProfile(userId, profileData, requestId) {
  loggingService.info(`Updating profile for user: ${userId}`);

  try {
    await _validateProfileData(profileData);

    await db.execute('UPDATE users SET name = ?, email = ? WHERE id = ?', [
      profileData.name,
      profileData.email,
      userId,
    ]);

    const updatedUser = await _getUser(userId, requestId);

    loggingService.info(`Profile updated for user: ${userId}`);
    return updatedUser;
  } catch (error) {
    loggingService.error(`Failed to update profile for user ${userId}: ${error.message}`);
    throw error;
  }
}

function updateUserProfile(userId, profileData) {
  const requestId = uuidv4();

  return telemetry.wrapWithSpan(
    'dataService.updateUserProfile',
    {
      'user.id': userId,
      'request.id': requestId,
    },
    async span => {
      try {
        return await _updateUserProfile(userId, profileData, requestId);
      } catch (error) {
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: error.message,
        });
        span.recordException(error);
        throw error;
      }
    }
  );
}

function _validateProfileData(profileData) {
  return telemetry.wrapWithSpan(
    'dataService._validateProfileData',
    {
      'validation.fields': Object.keys(profileData).join(','),
    },
    () => {
      if (!profileData.name) {
        throw new Error('Name is required');
      }

      if (!profileData.email || !profileData.email.includes('@')) {
        throw new Error('Valid email is required');
      }

      return true;
    }
  );
}

module.exports = {
  getUser,
  updateUserProfile,
};

Key Patterns for Convenient Conversion

  1. Follow the Wrapper Pattern:

    • Keep original implementation in private functions
    • Add public wrapper functions with telemetry

  2. Preserve Function Signatures:

    • Public functions should maintain same arguments and return types as original

  3. Use Request/Execution IDs:

    • Generate unique IDs at entry points
    • Pass them through to child functions for correlation

  4. Incremental Conversion:

    • Start with key operations first
    • Gradually expand to more parts of the application
    • Begin with leaf functions before tackling complex ones

  5. Add Meaningful Attributes:

    • Include business-relevant attributes in spans
    • Make attributes consistent across related operations

By following this process, you can add comprehensive tracing to your application in a methodical way while minimizing disruption to your existing code structure.

Visualizing and Analyzing Traces in Jaeger

After implementing OpenTelemetry tracing in your Node.js application, you can use Jaeger to visualize and analyze the collected traces.

Accessing the Jaeger UI

  1. Start Jaeger using Docker:

    docker-compose up -d

  2. Open the Jaeger UI in your browser:

    http://localhost:16686

Finding and Analyzing Traces

  1. Select Your Service:

    • In the "Service" dropdown, select your service name (e.g., "service-cronjob-demo")

  2. Filter by Operation:

    • Use the "Operation" dropdown to filter for specific operations (e.g., "performBackup", "sendNotification")
    • You can also search for traces containing specific tags or attributes

  3. Analyze Trace Hierarchy:

    • Click on a trace to see the detailed view
    • The hierarchical view shows parent-child relationships between spans
    • Spans are color-coded based on their duration

  4. Identifying Performance Issues:

    • Look for spans with long durations (they appear wider in the timeline)
    • Check for unexpected gaps between spans
    • Examine spans with error statuses (typically shown in red)

Example: GenerateUsageReport Chain Analysis

Notification Chain Trace Example

This visualization shows:

  • The total time to send a notification
  • Where most of the time is spent (e.g., in database operations, external API calls)
  • Any errors that occurred during processing
  • The parent-child relationships between all operations

Troubleshooting with Traces

  1. Error Analysis:

    • Filter for traces with errors to identify failure patterns
    • Examine error attributes to understand the root cause

  2. Performance Optimization:

    • Identify the slowest spans in your application
    • Focus optimization efforts on the most time-consuming operations

  3. End-to-End Visibility:

    • Follow a request across multiple services
    • Understand the complete flow of operations

About

Demo nodejs application tracing method for open telemetry

Topics

nodejs opentelemetry

Resources

Readme
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages