Logging Standards

Structured logging to Loki and Datadog using CustomLoggerService. All logs include request context automatically via ContextService.

Log Message Structure

All log messages use the createLogs helper to create structured log objects with unique codes and human-readable messages.

// module-name.log.ts
import { createLogs } from '../logger/create-logs.helper';

export const userLogs = createLogs('userLogs', {
  userCreationStarting: 'User creation process initiated',
  userCreationComplete: 'User successfully created in database',
  userNotFoundInCache: 'User lookup in cache returned no results',
  userValidationFailed: 'User input validation failed',
});

The createLogs helper automatically creates structured log objects:

{
  code: 'userLogs.userCreationStarting',
  message: 'User creation process initiated'
}

When logged, this appears as:

Console: userLogs.userCreationStarting: User creation process initiated
Loki/Datadog: Structured JSON with separate logCode and message fields for precise filtering

Log Correlation: Connecting the Flow

Every log automatically includes requestId and traceId from ContextService. This is the foundation of our observability strategy - you can reconstruct the entire request flow by filtering logs with the same requestId.

How It Works

HTTP Request arrives → Middleware creates requestId and traceId
ContextService stores them → Uses AsyncLocalStorage to propagate automatically
Every log includes them → Via this.contextService.getLoggingContext()
All logs are connected → Filter by requestId to see the entire flow

Example: Full Request Flow

// Request comes in: GET /api/users/123
// requestId: req_abc123, traceId: trace_xyz789

// Controller logs:
userLogs.controllerStarting  // requestId: req_abc123, traceId: trace_xyz789

// Service logs:
userLogs.readOneStarting     // requestId: req_abc123, traceId: trace_xyz789
userLogs.readOneComplete     // requestId: req_abc123, traceId: trace_xyz789

// Controller logs:
userLogs.controllerComplete  // requestId: req_abc123, traceId: trace_xyz789

In Loki/Datadog: Filter requestId:"req_abc123" → See all logs from this request in order

Why This Matters for V1

Log correlation eliminates the need for service-level tracing in most cases.

Without manual tracing code, you get:

✅ See every step of the request (via logs with shared requestId)
✅ Know when each operation started/completed (via log timestamps)
✅ See what data was processed (via log context)
✅ Track errors with full context (via error logs)
✅ HTTP-level timing (via automatic HTTP tracing)
✅ Database query timing (via automatic database tracing)

What you don't get (and probably don't need for V1):

❌ Precise method-level timing breakdown within a service
❌ Distributed trace visualization across multiple background jobs
❌ Complex span relationships for async operations

For 95% of debugging, logs + requestId + HTTP/DB traces are sufficient.

Accessing Request Context

const context = {
  module: 'UserService',
  method: 'readOne',
  ...this.contextService.getLoggingContext(),  // Adds: requestId, userId, traceId, spanId
  id: values.id,
};

The getLoggingContext() method returns:

{
  requestId: 'req_abc123',    // Unique per HTTP request
  userId: 'user_456',          // Current user (if authenticated)
  traceId: 'trace_xyz789',     // OpenTelemetry trace ID
  spanId: 'span_qrs012',       // OpenTelemetry span ID
}

Cross-Service Correlation

When one service calls another (HTTP, RabbitMQ, etc.), the traceId propagates automatically:

HTTP: W3C Trace Context headers (traceparent)
RabbitMQ: Message metadata with trace context
Background Jobs: Trace context passed in job data

This means logs from multiple services can be correlated by filtering on the same traceId.

Basic Pattern

Every service method logs start, completion, and errors.

import { Injectable } from '@nestjs/common';
import { CustomLoggerService } from '../../common/logger/logger.service';
import { ContextService } from '../../common/context/context.service';
import { userLogs } from './user.log';

@Injectable()
export class UserService {
  constructor(
    private readonly logger: CustomLoggerService,
    private readonly contextService: ContextService,
  ) {}

  async readOne(values: UserReadOneInput): Promise<User> {
    const context = {
      module: 'UserService',
      method: 'readOne',
      ...this.contextService.getLoggingContext(),  // Gets requestId, userId, traceId, spanId
      id: values.id,
    };

    this.logger.logWithContext(userLogs.readOneStarting, context);

    try {
      const entity = await this.repository.findOne({ where: { id: values.id } });

      if (!entity) {
        this.logger.errorWithContext(userLogs.userNotFoundError, '', context);
        throw new UserNotFoundError(values.id);
      }

      this.logger.logWithContext(userLogs.readOneComplete, context);
      return this.toUser(entity);
    } catch (error) {
      // Only log if not already logged
      if (!(error instanceof UserNotFoundError)) {
        this.logger.errorWithContext(
          userLogs.readOneUnexpectedError,
          error instanceof Error ? error.stack || '' : String(error),
          context,
        );
      }
      throw error;
    }
  }
}

Enhancing Console Readability with Template Variables

While log objects provide excellent searchability and structure, you can enhance console output by using template variables in your log messages. This keeps logging to exactly one line while making logs immediately actionable.

Basic Pattern

Step 1: Define log messages with $VARIABLE placeholders

// user.log.ts
export const userLogs = createLogs('userLogs', {
  // No variables - message stays simple
  userCreationStarting: 'User creation process initiated',

  // With variables - use $UPPERCASE placeholders
  userCreationComplete: 'User created successfully with id: $ID and email: $EMAIL',
  userNotFound: 'User not found for id: $ID',
  userValidationFailed: 'User validation failed for field: $FIELD with reason: $REASON',
});

Step 2: Use .replace() to substitute variables (one line!)

// Without variables - simple one line
this.logger.logWithContext(userLogs.userCreationStarting, context);

// With variables - still one line
this.logger.logWithContext(
  userLogs.userCreationComplete.replace({ id: user.id, email: user.email }),
  context,
);

// Multiple variables - one line
this.logger.errorWithContext(
  userLogs.userValidationFailed.replace({ field: 'email', reason: 'invalid format' }),
  '',
  context,
);

How It Works

The .replace() method:

Takes an object with camelCase keys
Converts keys to UPPERCASE and adds $ prefix
Replaces $VARIABLE placeholders in the message
Returns { code, message } ready for logging

// Log definition:
userNotFound: 'User not found for id: $ID'

// Usage:
userLogs.userNotFound.replace({ id: '123' })

// Result:
{
  code: 'userLogs.userNotFound',
  message: 'User not found for id: 123'
}

Real-World Examples

// Integration loading success
integrationLogs.integrationLoadedSuccessfully.replace({
  name: 'gmail',
  version: '1.0.0',
  domain: 'gmail'
})
// Result: "Integration loaded successfully with name: gmail, version: 1.0.0, domain: gmail"

// Integration not found
integrationLogs.integrationNotFound.replace({ domain: 'unknown-integration' })
// Result: "Integration not found for domain: unknown-integration"

// Error with context
integrationLogs.integrationLoadFailed.replace({
  directory: 'gmail',
  error: error.message
})
// Result: "Integration failed to load for directory: gmail with error: Module not found"

Variable Naming Convention

In log messages:

Use $UPPERCASE format: $ID, $NAME, $DOMAIN, $ERROR
Make it descriptive: $USER_ID not $U, $ERROR_MESSAGE not $ERR

In code:

Pass camelCase object keys: { userId, errorMessage }
Auto-converts to match: userId → $USER_ID

When to Use Variables

✅ DO use variables for:

Identifiers (IDs, names, domains, emails)
Counts and metrics (total, count, size)
Error messages and reasons
Status and state values

❌ DON'T use variables for:

Sensitive data (passwords, tokens, API keys)
Large payloads (entire objects, long arrays)
Data already in the context object (unless needed for immediate visibility)

Benefits

One line logging: Every log call is exactly one line
Console: Immediately actionable - critical info visible in message
Loki/Datadog: Structured - still has logCode field for filtering
Searchability: Log codes unchanged, messages have consistent format
Simple: Everyone understands variable replacement
Easy to review: Missing variables are obvious: "domain: $DOMAIN"

Logging Methods

All logging methods accept either a log object (from createLogs) or a plain string. Always prefer log objects for consistency.

logWithContext(message, context) Info logs for method start/completion and state changes.

message: Log object from .log.ts file or string
context: LogContext with module, method, and additional data

errorWithContext(message, trace, context) Error logs with stack trace.

message: Log object from .log.ts file or string
trace: Error stack trace (pass error.stack || '')
context: LogContext with module, method, and additional data

warnWithContext(message, context) Warnings for deprecated features or potential issues.

message: Log object from .log.ts file or string
context: LogContext with module, method, and additional data

debugWithContext(message, context) Debug information for development.

message: Log object from .log.ts file or string
context: LogContext with module, method, and additional data

Context Structure

Every log needs:

module - Service class name
method - Method name
...this.contextService.getLoggingContext() - Automatic requestId, userId, traceId, spanId
Additional fields - IDs, counts, status, etc.

Creating Log Files

Each module must have a module-name.log.ts file that defines all log messages for that module.

File structure:

src/
  user/
    user.service.ts
    user.controller.ts
    user.error.ts
    user.log.ts          ← Log messages here

Creating the log file:

// user.log.ts
import { createLogs } from '../common/logger/create-logs.helper';

export const userLogs = createLogs('userLogs', {
  // readOne method
  readOneStarting: 'User retrieval by ID initiated',
  readOneComplete: 'User successfully retrieved from database',
  readOneUnexpectedError: 'Unexpected error during user retrieval',
  userNotFoundError: 'User ID not found in database',

  // createOne method
  createOneStarting: 'User creation process initiated',
  createOneComplete: 'User successfully created in database',
  createOneValidationFailed: 'User creation validation failed',
  createOneDuplicateEmail: 'User creation failed due to duplicate email',

  // updateOne method
  updateOneStarting: 'User update process initiated',
  updateOneComplete: 'User successfully updated in database',
  updateOneNotFound: 'User update failed - ID not found',

  // deleteOne method
  deleteOneStarting: 'User deletion process initiated',
  deleteOneComplete: 'User successfully deleted from database',
});

Naming conventions:

Log constant names: camelCase describing the log (e.g., userCreationStarting)
Namespace: Same as the file/module name (e.g., 'userLogs')
Messages: Human-readable, specific to the operation
Group by method using comments for organization

Anti-Patterns

❌ Don't log the same error twice Check error type before logging in catch blocks.

// Bad - logs error twice
if (!entity) {
  this.logger.errorWithContext('Not found', '', context);
  throw new NotFoundError(id);
}
// Catch logs it again!

// Good - skip known errors in catch
catch (error) {
  if (!(error instanceof NotFoundError)) {
    this.logger.errorWithContext('Error', error.stack, context);
  }
  throw error;
}

❌ Don't use console.log Always use CustomLoggerService methods.

❌ Don't use duplicate log messages Every log message should be unique across the codebase. This makes logs searchable and helps identify exactly where a log came from.

All log messages must be defined in .log.ts files using the createLogs helper. Each module must have its own .log.ts file.

// Bad - same message in multiple places
this.logger.logWithContext('Processing request', context);  // Used in 10 different services
this.logger.logWithContext('Operation completed', context);  // Used everywhere

// Bad - string literals even if unique
this.logger.logWithContext('User creation starting', context);  // Not using log objects

// Good - log objects with unique codes and messages
this.logger.logWithContext(userLogs.userCreationStarting, context);
this.logger.logWithContext(userLogs.userCreationComplete, context);

Benefits of log objects with unique codes:

Searchable in code: Search for userCreationStarting to find the definition and all usages
Searchable in logs: Filter by logCode:"userLogs.userCreationStarting" in Loki/Datadog
Structured data: Separate logCode and message fields for precise queries
Easier debugging: Instantly identify which service and method produced a log
Better alerting: Create alerts based on specific log codes
Type-safe: TypeScript autocomplete prevents typos

❌ Don't log sensitive data Never log passwords, tokens, credit cards, API keys.

❌ Don't pass requestId/userId manually Use ContextService.getLoggingContext() instead.

// Bad - manual context
const context = {
  module: 'UserService',
  method: 'readOne',
  requestId: values.requestId,  // Don't pass these manually
  userId: values.userId,
};

// Good - use ContextService
const context = {
  module: 'UserService',
  method: 'readOne',
  ...this.contextService.getLoggingContext(),  // Automatic
};

Other mistakes:

❌ Forgetting to log method start and completion
❌ Not including error stack traces
❌ Missing ContextService in constructor

Log Message Structure​

Log Correlation: Connecting the Flow​

How It Works​

Example: Full Request Flow​

Why This Matters for V1​

Accessing Request Context​

Cross-Service Correlation​

Basic Pattern​

Enhancing Console Readability with Template Variables​

Basic Pattern​

How It Works​

Real-World Examples​

Variable Naming Convention​

When to Use Variables​

Benefits​

Logging Methods​

Context Structure​

Creating Log Files​

Anti-Patterns​

Log Message Structure

Log Correlation: Connecting the Flow

How It Works

Example: Full Request Flow

Why This Matters for V1

Accessing Request Context

Cross-Service Correlation

Basic Pattern

Enhancing Console Readability with Template Variables

Basic Pattern

How It Works

Real-World Examples

Variable Naming Convention

When to Use Variables

Benefits

Logging Methods

Context Structure

Creating Log Files

Anti-Patterns