Logging in Python: Production Configuration

1. Overview

print() works fine for small scripts. But production is different: you need to filter logs by level, aggregate logs into a central system, trace a request through many log lines, and logging shouldn’t slow down the application.

This guide covers configuring Python’s standard logging module for production. After reading, you’ll be able to build a setup with per-module loggers, dictConfig, JSON logging, automatic request_id attachment, asynchronous logging with QueueHandler, and testing with pytest. All examples use Python 3.12 and only the standard library—except testing, which requires pytest.

2. Why print and basicConfig Aren’t Enough

Look at a quick-and-dirty order processing service:

import logging

logging.basicConfig(level=logging.INFO)

def place_order(order_id: str, amount: float) -> None:
    print(f"Placing order {order_id}")
    logging.info(f"Order {order_id} amount={amount}")

place_order("ORD-1001", 250_000)

Placing order ORD-1001
INFO:root:Order ORD-1001 amount=250000

This code has three problems:

• print has no level, no timestamp, and can’t be disabled via configuration.
• logging.info(...) writes to the root logger. You don’t know which module the log came from, and you can’t adjust levels per module.
• basicConfig only works on the first call, if the root logger has no handlers yet. If another library calls it first, your configuration gets silently ignored.

3. Per-Module Logger with getLogger(__name__)

The first rule: each module creates its own logger, named after __name__.

# shop/orders.py
import logging

logger = logging.getLogger(__name__)  # logger name: "shop.orders"

class OrderService:
    def place_order(self, order_id: str, quantity: int) -> None:
        logger.info("Placing order %s with quantity %d", order_id, quantity)
        if quantity > 50:
            logger.warning("Large order %s: quantity=%d", order_id, quantity)

Logger names form a hierarchy using dots. shop.orders is a child of shop, and shop is a child of root. A record propagates from a child logger to its parent (called propagation). So you only need to attach a handler at root or shop, and set the level for each branch independently. For example, you can set shop.payments to DEBUG and keep the rest at INFO.

Also notice the parameter passing: use %s with arguments instead of f-strings. When a level is disabled, the string won’t be formatted, so you don’t waste CPU on logs that won’t be printed. Additionally, log aggregation tools can group lines by the same message template.

Note: In libraries (packages you distribute to others), don’t configure handlers. Only call getLogger(__name__) and let the application decide. If needed, add logging.getLogger("your_lib").addHandler(logging.NullHandler()).

4. Centralized Configuration with dictConfig

In production, consolidate all configuration to one place, called once when the app starts. logging.config.dictConfig is perfect for this.

# shop/logging_config.py
import logging.config
import os

def setup_logging() -> None:
    level = os.getenv("LOG_LEVEL", "INFO")
    logging.config.dictConfig({
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {
            "plain": {
                "format": "%(asctime)s %(levelname)-8s %(name)s - %(message)s",
            },
        },
        "handlers": {
            "console": {
                "class": "logging.StreamHandler",
                "stream": "ext://sys.stdout",
                "formatter": "plain",
            },
        },
        "loggers": {
            "shop": {"level": level},
            "urllib3": {"level": "WARNING"},
        },
        "root": {"level": "WARNING", "handlers": ["console"]},
    })

Key points:

• "disable_existing_loggers": False is mandatory in practice. The default True disables all loggers created before dictConfig runs, including logger at the top of your imported modules. Logs vanish without error.
• Handler is only attached to root. Logger shop only sets the level; records propagate up to root to be printed.
• Root is set to WARNING to suppress noisy INFO logs from third-party libraries. Your code in the shop branch still logs at INFO.
• The level is read from an environment variable. Enable DEBUG when investigating without modifying code.

Call this at the entry point:

# main.py
from shop.logging_config import setup_logging
from shop.orders import OrderService

setup_logging()
OrderService().place_order("ORD-1001", quantity=80)

2026-10-18 09:15:02,114 INFO     shop.orders - Placing order ORD-1001 with quantity 80
2026-10-18 09:15:02,114 WARNING  shop.orders - Large order ORD-1001: quantity=80

Writing to stdout is the right choice when running in containers. Docker or Kubernetes will collect logs from this stream, so you don’t need to manage files or log rotation yourself.

5. Structured Logging with JSON

Log aggregation systems like Loki, Elasticsearch, or CloudWatch search JSON much better than free-form text. You can write a formatter yourself without external libraries:

# shop/json_formatter.py
import json
import logging
from datetime import datetime, timezone

_RESERVED = set(vars(logging.LogRecord("", 0, "", 0, "", None, None))) | {"message", "asctime"}

class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        payload = {
            "ts": datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        for key, value in record.__dict__.items():
            if key not in _RESERVED and not key.startswith("_"):
                payload[key] = value
        if record.exc_info:
            payload["exc"] = self.formatException(record.exc_info)
        return json.dumps(payload, ensure_ascii=False, default=str)

How it works:

• _RESERVED holds default LogRecord attributes like pathname, lineno, args.
• Any other attributes are treated as custom fields and added to JSON. These typically come from the extra= parameter or from a filter (see section 6).
• default=str helps json.dumps avoid errors when encountering types like Decimal or UUID.

Declare the formatter in dictConfig using a special key:

"formatters": {
    "json": {"()": "shop.json_formatter.JsonFormatter"},
},
"handlers": {
    "console": {
        "class": "logging.StreamHandler",
        "stream": "ext://sys.stdout",
        "formatter": "json",
    },
},

Add business fields with extra:

logger.info("Payment captured", extra={"order_id": "ORD-1001", "amount": 250000})

{"ts": "2026-10-18T02:15:02.114+00:00", "level": "INFO", "logger": "shop.payments", "message": "Payment captured", "order_id": "ORD-1001", "amount": 250000}

Note: Keys in extra must not collide with built-in LogRecord attributes. For example, extra={"name": "..."} or extra={"message": "..."} will raise KeyError: "Attempt to overwrite 'name' in LogRecord".

6. Automatic request_id with contextvars

Passing extra={"request_id": ...} to every log call is both verbose and error-prone. A better approach is to store the request_id in a ContextVar, then let a Filter attach it to every record.

# shop/request_context.py
import logging
from contextvars import ContextVar

request_id_var: ContextVar[str] = ContextVar("request_id", default="-")

class RequestIdFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        record.request_id = request_id_var.get()
        return True  # don't block any records, just add data

ContextVar is safe for both threads and asyncio. Each task or request keeps its own value, so there’s no cross-contamination between requests.

Attach filters to handlers, not loggers. A filter on a logger only runs for records created directly from that logger, not for propagated records.

"filters": {
    "request_id": {"()": "shop.request_context.RequestIdFilter"},
},
"handlers": {
    "console": {
        "class": "logging.StreamHandler",
        "stream": "ext://sys.stdout",
        "formatter": "json",
        "filters": ["request_id"],
    },
},

Set the value at the start of each request, for example in a middleware:

import uuid
from shop.request_context import request_id_var

def handle_request(order_id: str) -> None:
    token = request_id_var.set(uuid.uuid4().hex[:12])
    try:
        OrderService().place_order(order_id, quantity=3)
    finally:
        request_id_var.reset(token)

{"ts": "...", "level": "INFO", "logger": "shop.orders", "message": "Placing order ORD-1002 with quantity 3", "request_id": "9f2c1ab07d3e"}

Now you can filter by request_id to see the entire journey of a request.

7. Logging Exceptions Correctly

logger = logging.getLogger("shop.payments")

class PaymentService:
    def charge(self, order_id: str, amount: int) -> bool:
        try:
            raise ConnectionError("gateway timeout")
        except ConnectionError:
            logger.exception("Charge failed for order %s", order_id)
            return False

logger.exception(...) logs at ERROR level and automatically includes the stack trace. You only call it inside an except block. If you want a different level while keeping the stack trace, use logger.warning("...", exc_info=True).

Two common mistakes:

• logger.error(f"Charge failed: {e}") loses the stack trace. When investigating, you only see one line like “gateway timeout” with no idea where it happened.
• Logging and re-raising at every level. The same exception appears five times in the log. Log at the exception handling layer, where you decide not to re-raise.

8. Prevent Logging from Blocking: QueueHandler

StreamHandler writing to stdout is usually fast enough. But handlers that send over the network (syslog, HTTP, SMTP) or write to disk slowly can block the very thread handling the request. To avoid this, put records in a queue and let a background thread write to the actual destination:

import atexit
import logging
import logging.handlers
import queue

def setup_async_logging(target: logging.Handler) -> None:
    log_queue: queue.Queue = queue.Queue(-1)
    queue_handler = logging.handlers.QueueHandler(log_queue)

    listener = logging.handlers.QueueListener(
        log_queue, target, respect_handler_level=True
    )
    listener.start()
    atexit.register(listener.stop)  # flush remaining logs on exit

    root = logging.getLogger()
    root.handlers = [queue_handler]

The flow works like this:

• The application thread does one thing: puts records into a queue. This is nearly instantaneous.
• QueueListener runs on its own thread, dequeues records, and passes them to target, like a SysLogHandler.
• atexit.register(listener.stop) ensures final logs are flushed before the process exits.

Note: Filters like RequestIdFilter must be attached to QueueHandler, not to the target handler. The target handler runs on the listener’s thread, where ContextVar doesn’t carry the value from the original request.

9. Testing Logs with pytest caplog

Business warning logs, like large orders, are behavior worth testing. Add pytest as a dev dependency:

pip install pytest

# tests/test_orders.py
import logging
from shop.orders import OrderService

def test_large_order_logs_warning(caplog):
    with caplog.at_level(logging.INFO, logger="shop.orders"):
        OrderService().place_order("ORD-2001", quantity=80)

    warnings = [r for r in caplog.records if r.levelno == logging.WARNING]
    assert len(warnings) == 1
    assert warnings[0].name == "shop.orders"
    assert warnings[0].getMessage() == "Large order ORD-2001: quantity=80"

def test_small_order_does_not_warn(caplog):
    with caplog.at_level(logging.INFO, logger="shop.orders"):
        OrderService().place_order("ORD-2002", quantity=2)

    assert all(r.levelno < logging.WARNING for r in caplog.records)

tests/test_orders.py ..                                  [100%]
2 passed in 0.03s

caplog.at_level temporarily lowers the level of logger shop.orders within the with block. caplog.records contains the raw LogRecord objects, so you can check the level, logger name, and any extra fields if present.

Note: caplog attaches a handler to the root logger. If your configuration sets "propagate": False, the test won’t capture any records.

10. Common Mistakes and What Not to Do

• Configuring logging at module level (calling basicConfig or dictConfig at import). Configure only at the entry point, exactly once.
• Handlers attached twice, like attaching to both shop and root while propagate is still True. Each log line prints twice. Attach handlers to only one level.
• Logging sensitive data: passwords, tokens, card numbers. Don’t log entire request objects. If necessary, write a Filter to mask these fields.
• Using RotatingFileHandler in multi-process applications (Gunicorn with multiple workers). Multiple processes rotating the same file can lose logs. Write to stdout instead, or have each worker push through QueueHandler to a single writer process.
• Logging too much at INFO in hot loops. Storage costs and noise increase rapidly. Put such logs at DEBUG instead.
• Using JSON during local development. JSON is hard to read by eye, so choose the formatter based on an environment variable: plain for dev, json for production.

11. Conclusion

Good logging configuration for production isn’t complex, just consistent. Use getLogger(__name__) in every module. Consolidate configuration in a single dictConfig call with disable_existing_loggers: False. Write JSON to stdout, attach request_id via ContextVar and Filter, and use logger.exception to preserve the stack trace. When the target handler is slow, switch to QueueHandler and QueueListener.

Finally, treat logging as part of your system’s behavior and test it with caplog. When trouble strikes at 2 AM, these structured logs will help you find the root cause much faster.