Topics Production Debugging Topic 3: Log Analysis & Troubleshooting Techniques (How Senior Engineers Find Root Causes from Logs)
Back Sign up to track progress
Production Debugging

Topic 3: Log Analysis & Troubleshooting Techniques (How Senior Engineers Find Root Causes from Logs)

Sign up free to track your views & progress

🚀 TL;DR

In production systems:

Metrics Tell You

Something Is Wrong

But:

Logs Tell You

Why It Is Wrong

This is why log analysis is one of the most important skills for:

  • Production Support Engineers
  • SREs
  • Java Backend Engineers
  • Architects

Most production incidents are solved by:

Reading Logs Correctly

not by:

Reading Source Code First

📘 Why Logs Matter

Imagine:

Order API

500 Error

Monitoring shows:

Error Rate Increased

Question:

Why?

Possible reasons:

Database Down

NullPointerException

Timeout

Connection Pool Exhausted

Kafka Failure

Metrics cannot tell which one.


Logs can.


What is a Log?

A log is a record of what happened inside an application.

Example:

2026-06-05 10:15:21

INFO

Order Created Successfully

Logs provide:

Timeline

Events

Errors

Context

Root Cause Clues


Log Levels

Very common interview question.


TRACE

Most detailed.

Example:

Entering Method

Used rarely in production.


DEBUG

Developer debugging information.

Example:

Received Request Payload

Usually disabled in production.


INFO

Normal business events.

Example:

Order Created

Most commonly used.


WARN

Potential problem.

Example:

Retry Attempt 2

Needs attention.


ERROR

Failure occurred.

Example:

Database Connection Failed

Most important during incidents.


First Rule of Log Analysis

Start with:

ERROR

Then:

WARN

Then:

INFO

Never start reading millions of log lines from the beginning.


Real Production Example

Customer complaint:

Unable To Place Order

Log:

ERROR

Connection Timeout

Potential root cause:

Database Connectivity

Investigation direction identified.


Golden Log Analysis Workflow

Incident Time

↓

Find ERROR Logs

↓

Find First Failure

↓

Follow Request Flow

↓

Identify Root Cause

The Most Important Rule

Do NOT focus on:

Last Error

Focus on:

First Error

Example

Bad logs:

Payment Failed

Order Failed

Notification Failed

Checkout Failed

Actual root cause:

Database Connection Pool Exhausted

Everything else is a symptom.


Finding the First Exception

Very important skill.


Example:

ERROR

Notification Failed

Not root cause.


Scroll upward.


Find:

ERROR

Inventory Service Timeout

Scroll further.


Find:

ERROR

Database Timeout

Root cause area found.


Exception Analysis

Most production debugging involves exceptions.


Example:

java.lang.NullPointerException

Question:

Which Line?

Stack trace answers.


Understanding Stack Trace

Example:

java.lang.NullPointerException

at OrderService.process(OrderService.java:45)

at OrderController.createOrder(OrderController.java:20)

Read from:

Top To Bottom

First line:

Exception Type

Next lines:

Call Stack

Common Java Exceptions

NullPointerException

Meaning:

Object Is Null

OutOfMemoryError

Meaning:

Heap Exhausted

StackOverflowError

Meaning:

Infinite Recursion

SQLException

Meaning:

Database Issue

ConnectException

Meaning:

Network Connection Failed

TimeoutException

Meaning:

Dependency Too Slow

Correlation ID

Most important microservices logging topic.


Without Correlation ID:

Gateway Log

Order Log

Payment Log

Inventory Log

Impossible to connect.


With Correlation ID:

REQ-12345

appears everywhere.


Example:

REQ-12345

Gateway

REQ-12345

Order Service

REQ-12345

Payment Service

Entire request becomes traceable.


Production Investigation Example

Customer reports:

Order Failed

Search:

Correlation ID

Follow journey:

Gateway

↓

Order Service

↓

Payment Service

↓

Inventory Service

Root cause becomes visible.


Searching Logs Efficiently

Never search:

Everything

Use filters:

Correlation ID

User ID

Order ID

Exception Name

Timestamp


Example:

OrderId=1001

Much faster investigation.


ELK Investigation Workflow

Typical enterprise process.


Step 1:

Search timestamp.


Step 2:

Search ERROR.


Step 3:

Search Correlation ID.


Step 4:

Follow request.


Step 5:

Identify first failure.


Common Log Patterns

Database Connection Issue

Connection Refused

Usually indicates:

Database Down

Connection Pool Exhaustion

Unable To Acquire JDBC Connection

Usually indicates:

Pool Exhausted

Timeout

Read Timeout

Usually indicates:

Slow Dependency

Memory Issue

OutOfMemoryError

Usually indicates:

Memory Leak

Heap Exhaustion

Deadlock

Deadlock Found

Usually indicates:

Concurrency Problem

Real Scenario 1

Alert:

Order API Slow

Metrics:

Latency 10 Seconds

Logs:

Payment Service Timeout

Root cause area:

Payment Dependency

Real Scenario 2

Alert:

Application Crash

Logs:

OutOfMemoryError

Root cause area:

Memory Leak Investigation

Real Scenario 3

Alert:

Database Errors

Logs:

Unable To Acquire JDBC Connection

Root cause:

Connection Pool Exhausted

Structured Logging

Bad:

Something Failed

Good:

OrderId=1001

CustomerId=500

PaymentId=200

Status=FAILED

Always log context.


What Makes a Good Log?

Should answer:

What Happened?

When?

Where?

Why?

Which Request?


Log Analysis Strategy Used by Senior Engineers

1. Incident Time

↓

2. ERROR Logs

↓

3. Correlation ID

↓

4. First Exception

↓

5. Stack Trace

↓

6. Dependency Analysis

↓

7. Root Cause

Common Mistakes

Reading Logs Randomly

Bad approach.


Ignoring Timestamp

Huge mistake.


Ignoring Correlation ID

Makes debugging harder.


Looking At Last Error

Need first failure.


Ignoring WARN Logs

Often contain clues.


Production Support Interview Scenario

Question

Order API returns 500 errors.

How would you investigate?

Strong Answer

1. Identify incident timestamp
2. Search ERROR logs
3. Find Correlation ID
4. Trace request path
5. Analyze first exception
6. Check dependency failures
7. Validate root cause

This is a senior-level answer.


🎯 Interview Q&A

Q1: Why are logs important?

Answer: Logs provide detailed information about application behavior and failures.


Q2: Which log level is most useful during incidents?

Answer: ERROR, followed by WARN.


Q3: What should you look for first in logs?

Answer: The first exception or error near the incident timestamp.


Q4: Why is Correlation ID important?

Answer: It allows tracking a request across multiple services.


Q5: Why should engineers focus on the first error?

Answer: Later errors are often symptoms of the original failure.


Q6: What does OutOfMemoryError indicate?

Answer: Heap memory exhaustion or a memory leak.


Q7: What does a TimeoutException usually indicate?

Answer: A slow or unavailable dependency.


Q8: Why is structured logging important?

Answer: It makes searching and troubleshooting significantly easier.


Q9: What information should a good log contain?

Answer: Timestamp, severity, request context, and meaningful messages.


Q10: What is the biggest mistake in log analysis?

Answer: Reading logs without using timestamps, correlation IDs, or filtering.


📋 Revision Cheat Sheet

  • Logs explain why incidents occur.
  • Start with ERROR logs.
  • Check WARN logs next.
  • Focus on the first exception.
  • Use timestamps.
  • Use Correlation IDs.
  • Follow request flow.
  • Analyze stack traces.
  • Structured logs are easier to search.
  • OutOfMemoryError indicates memory problems.
  • TimeoutException indicates dependency issues.
  • SQLException indicates database issues.
  • Never read logs randomly.
  • Filter by request identifiers.
  • Follow incident timeline.
  • Logs + Metrics + Traces = Root Cause.
  • Senior engineers follow evidence.
  • Good logs contain context.
  • First failure matters most.
  • Log analysis is a core production support skill.

Next Topic: API Performance Troubleshooting & Slow API Investigation

This is one of the most frequently encountered production issues. We will cover:

  • Slow REST APIs
  • High Response Time
  • Database Bottlenecks
  • External Service Delays
  • Thread Pool Exhaustion
  • Connection Pool Issues
  • N+1 Query Problems
  • Real-world troubleshooting workflow used by senior engineers.
Done reading this topic? Sign up free to track your progress.
Sign Up to Track