Collecting Wildberries storefront prices and competitor prices through the customer site

Technically this is part of the same Wildberries analytics project, but the data is taken not from the seller dashboard but from the customer-facing site, so it is a separate circuit with its own authorization, its own browser and its own tasks.

The seller dashboard only knows our own price. What the customer eventually pays for it is not visible there: on top of our price Wildberries adds its own discount, and on top of that the wallet discount, and both are computed on the storefront side. Competitor prices are not in the dashboard at all. So prices are collected from the customer site — from the same source an ordinary customer sees.

Dropping the browser in favour of the product-card JSON

The first version opened product cards in a browser and read the price off the page. That worked, but it cost about 790 KB of traffic and 0.6 seconds per product, and the list includes both our catalogue and competitors' SKUs. The working version calls the product card's own internal JSON in batches of up to three hundred SKUs: about a kilobyte and hundredths of a second per product. The browser stayed, but exactly one is needed and only at startup.

Bypassing the anti-bot without hard-coded headers

The browser's job is to get past the anti-bot and hand over the headers without which the API answers 403. The headers and the request address are not hard-coded: before the site's scripts load, a wrapper around fetch is injected into the page via CDP, collecting the addresses and headers of the page's real requests, and the script peeks at how the site itself calls its own API. This is fundamental: a guest and an authorized customer have different endpoints and different parameters, and the frontend changes them from version to version — with its own hard-coded address the script would get an answer about the wrong session, or none at all. The requests themselves are executed from inside the page: then the cookies, headers and TLS fingerprint are native and the anti-bot cannot tell us from the site itself — the same request from outside returns 403.

A dedicated browser and a virtual display

The browser here is its own, not the project's shared one: the shared one runs headless, and the anti-bot lets neither a regular driver nor undetected through in headless mode. A real Chrome is needed, so on a server without a monitor the virtual display Xvfb is raised — the browser works as usual but draws into a framebuffer. The driver is not downloaded on the fly either: Google's storage is unreachable from Russia, and the system proxy is deliberately cut off here (Wildberries behaves differently through it), so the project keeps its own copy of the driver, which is patched for the installed Chrome version. Before startup, lock files left behind by a browser killed on timeout are removed from the profile: without this Chrome no longer starts on that profile, and the parser would silently drop into guest mode forever.

The customer Chrome profile

A separate Chrome profile is the customer one, with its own one-time SMS authorization. The seller dashboard profiles are no good here: a different site, a different session. The authorization script asks for the phone number and the code in the terminal and types them into the page itself, so a profile can be created over SSH where the browser window is not visible. Determining "logged in or browsing as a guest" is done from two pieces of evidence — the site header and the session cookies — because Wildberries first serves a guest header and only then replaces it: judging by the first look at the page, a profile with a live session was declared a guest. If the profile is busy or damaged, collection continues as a guest: the prices matter more than an extra percent of discount.

The wallet discount

The trickiest part is the wallet discount. It is not in the API response at all, the storefront computes that price on the frontend, and the percentage depends on whether the customer is logged in (2% for a guest, 3% for an authorized customer). Instead of a hard-coded constant the script measures the percentage itself: it opens several live product cards, reads the prices drawn there and reconstructs the percentage from them. The measurement must pass a check — the percentage found must reproduce every price seen to within a rouble; on a sample of three products, without such a check, 0.6% was measured instead of the guest's 2%, because the second price on one of the cards turned out to be a promotional price rather than the wallet one. If it does not add up or the result falls outside plausible bounds, the constant is used and a warning is written to the log; the measurement is compared with the previous run so as to notice that the platform has changed the percentage.

The SKU list and product statuses

The script takes the list of SKUs from the database, not from a file: ours from the product catalogue with a cutoff by the last export, so that cards long gone from the catalogue do not get into the run; competitors from a separate reference table where up to seven other sellers' SKUs are linked to each of our products. Products missing from the response are not lost but get a status: "on sale", "out of stock" (the card exists, sales do not) and "removed" (no card — taken down or blocked).

Storing prices as history

In the database prices are stored as history: each run appends rows and overwrites nothing, and the current price is the latest by time. Previously there was one row per SKU and it was overwritten, which made it impossible to see when and how often a competitor changed a price. The prices and the "our product — its competitors" link live in different tables and are joined at export time. Stored separately are the clean price, the same for every customer (this is exactly the one to compare with the market), and the price with the wallet, which depends on the customer and is therefore for reference only.

The market comparison summary sheet

The summary sheet is assembled by joining four sources. The price is shown in four steps, where each next one includes all previous discounts: without discounts, with our discount, with the Wildberries discount, with the wallet — the first two from the dashboard, the second two from the storefront. Previously all four were taken from the storefront, and what was called "the WB discount" was in fact our own — the sheet showed 73% where we ourselves had set 70. Discounts are output as fractions, the percentage is drawn by the cell format; the platform's total discount is calculated as the drop from our price to the price with the wallet — neither as a sum of the individual fractions (they have different bases) nor as a ratio of prices, both substitutes were rejected by the director. Alongside are the product's item type and category (the category is not in the product report, it comes from a separately collected WB item reference), yesterday's orders, and the deviation of our price from the minimum competitor price.