Centralized Error Collection and Normalization Subsystem for Multi-Process Parser

A unified module without external dependencies (except for the database driver) that addresses the observability issue of a multi-process parser: dozens of scripts are launched as separate processes, and in case of failure, the cause gets lost in the general output stream. The module collects errors from all processes into a single table, normalizes them into a concise uniform format, and connects with a single line — without rewriting existing code.

Runtime error interception

The interception is built on two Python runtime mechanisms. The first is overriding the built-in print function: the wrapper saves the original console output but additionally analyzes each message and, upon encountering a project-consistent error marker, extracts and saves it. This allowed integrating collection into the codebase, where exception handling has historically been done through print with a marker, without touching the scripts themselves. The second is replacing the system's unhandled exception handler: any exception that reaches the top of the stack is intercepted, the source file, type, and location of the failure are determined from the traceback, the record is saved, and then control is passed to the standard handler so that the interpreter's behavior remains unchanged. The installation of interceptors is idempotent.

Runner for child processes

To ensure interception works not only in the main process but also in all child processes, the module also acts as a runner: it launches the target script with the interceptors already set, correctly substituting the execution context so that the script runs as a regular main module. The dispatchers of all three parser subsystems launch their child scripts precisely through this runner, so centralized collection covers the entire multi-process pipeline.

Error normalization

The core of the module is the normalization of 'raw' errors into human-readable form. The parser messages have a structured format with delimiters, and the module parses them using regular expressions: it extracts the source file, exception type, and exact location of the failure in the code (file, line, function). A special case for Selenium and ChromeDriver has been developed: the native part of the driver is removed from the mixed stack, leaving only the Python traceback, and the noisy message is cleaned of auxiliary tails. A dictionary of short Russian explanations for common browser automation exceptions has been created, so a clear formulation 'where it happened: what occurred' is stored instead of the English stack. The length of messages and stack is limited: the full text remains in the console, while a compressed summary goes to the database.

Storage and reliability

The storage is a separate table in the same database, created idempotently; in addition to the message and type, it stores the failure location, timestamp, and notification sending flag (a provision for future mailing). A fundamental requirement is that the module must not become a source of failures itself: writing to the database completely suppresses its own exceptions, and the wrappers are protected from secondary errors. As a result, this is a transparent infrastructure layer that aggregates errors from the entire pipeline in one place and formats them for both quick analysis and notifications.