Treat scrapers like production software

The first version of a scraper is always a script: open a URL, pick a CSS selector, dump JSON. That version is useful for a spike. It is not a system you can leave running on a schedule and trust on Monday morning.

We treat Python automation the same way we treat a web app. Typed inputs, explicit failure modes, logs without personal data, and a deploy path that is not "run it on my machine." If the job matters, it gets the same CI and monitoring bar as everything else we ship.

Selectors will break. Design for that.

Sites redesign. Class names churn. Anti-bot pages appear on a Tuesday. A production scraper assumes the DOM will change and fails in a way a human can diagnose: which step died, which selector missed, which status code came back. Silent empty files are worse than a loud error.

• Retry with backoff on timeouts and 5xx. Do not retry a 404 as if the page will return.
• Record run metrics: rows written, duration, skip reasons. A dashboard you never open still beats a print statement.
• Keep a contract for the output schema. Downstream jobs should not parse whatever the HTML felt like today.

Respect the public web you depend on

Public data is not a free-for-all. Honor robots.txt, rate-limit yourself, cache when the source does not need a live hit, and store only what the job requires. If a site offers an API, use it. Scraping is a last resort for sources that have no supported interface, not a shortcut around one.

Ship it like a service

A useful scraper has a schedule, an owner, and an alert when it stops producing. It runs in an environment you can rebuild. Secrets stay in the environment, not in the repo. The output lands in a store the rest of the stack can read. That is the difference between a weekend script and automation a business can depend on.