How to pull structured data from any website via an API or CLI
You open a product page and see a price. Your code opens the same page and sees a few hundred nested divs, some inline scripts, images that load whenever they feel like it, and class names that will change next Tuesday. What you actually want out of all that is JSON: a title, a price, a couple of fields, done.
Doing this for one page is a five-minute job. Doing it for fifty sites that keep changing is where it gets annoying. The extraction is rarely the real problem. The problem is holding a steady output schema in front of sources that will not hold still, and catching it when one of them breaks before your database fills up with nulls. There are three ways to go about it, and each has its own headaches.
Approach 1: parse the HTML yourself
Grab the page, parse it, pull out what you need with CSS selectors.
import requests
from bs4 import BeautifulSoup
html = requests.get(“https://api.example.com/product/42”).text
soup = BeautifulSoup(html, “html.parser”)
data = {
“title”: soup.select_one(“h1.product-title”).get_text(strip=True),
“price”: soup.select_one(“[data-price]”)[“data-price”],
}
print(data)
Cheap, fast, no real dependencies. Fine when you own the page or it barely changes.
The problem is upkeep. Your selectors lean on someone else’s HTML, and they never agreed to keep it stable for you. Somebody renames a class or ships an A/B test, select_one starts returning None, and you find out two weeks later when you check the table and half the rows are empty. Do this across thirty sites and you have basically signed up for a second job keeping selectors alive.
Approach 2: drive a headless browser
Plenty of pages ship almost nothing in the initial HTML and build the content with JavaScript after load. For those you need a real browser. Headless Chromium runs the page like a normal one and gives you the DOM once it has settled.
import { chromium } from “playwright”;
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(“https://api.example.com/feed”);
await page.waitForSelector(“.card”);
const items = await page.$$eval(“.card”, cards =>
cards.map(c => ({
name: c.querySelector(“h2”)?.innerText,
url: c.querySelector(“a”)?.href,
}))
);
await browser.close();
console.log(items);
This gets you past the JS-rendering wall that plain fetching runs into. You pay for it, though. A browser process per job, more RAM, slower runs, and a fresh set of timing bugs around when the content actually shows up. And you are still writing the same selectors, just with a much heavier thing running them.
Approach 3: call a hosted API or CLI
Here you stop doing the extraction yourself. You hand a URL to a service, it hands back structured data. The browser, the parsing, the retries, the layout changes, all of that is on them now.
The request is about as dull as it gets, which is the whole point:
curl “https://api.example.com/extract?url=https://target.site/profile” \
-H “Authorization: Bearer $TOKEN”
{
“title”: “Jane Doe”,
“headline”: “Staff Engineer”,
“links”: [“https://janedoe.dev”],
“text”: “…”
}
Same thing from a terminal, which is handy for scripts and one-off pulls:
extract get –url “https://target.site/profile” –format json
This is where something like Anysite comes in. You give it a URL and it returns structured data over an API or CLI, with ready-made endpoints for the usual platforms (profiles, posts and comments, listings, company and filing data, maps results) and a general parser that hands back cleaned content, links and metadata as JSON when there is no specific endpoint for the site. You get a normalized object back instead of writing and nursing a custom selector for every place you scrape.
A concrete flow
Say you want the latest posts from a handful of public profiles. The DIY version means fetching each page, working out whether it needs a browser, writing selectors per site, and dealing with however each one handles pagination. Through a hosted call it turns into one loop:
import requests
def posts_for(url):
r = requests.get(
“https://api.example.com/extract”,
params={“url”: url, “type”: “posts”},
headers={“Authorization”: f”Bearer {TOKEN}”},
)
r.raise_for_status()
return r.json()[“items”]
for profile in profiles:
for post in posts_for(profile):
save(post[“id”], post[“text”], post[“timestamp”])
All the per-site weirdness sits behind the endpoint. Your code just reads a stable schema and stops caring which platform the URL points at.
Extract responsibly, whatever route you pick
None of this gets you out of the basics. Read the site’s robots.txt and terms before you start hammering it, and use the official API if there is one. Space your requests out, send a real user agent, and cache so you are not pulling the same URL over and over. Treat failures as normal rather than exceptional: retry with backoff, keep a list of URLs that keep dying, and get alerted when your success rate drops instead of finding out from an empty dashboard. A hosted service takes care of a good chunk of this, but how hard you hit someone’s site is still on you.
When a hosted API or CLI wins
Doing it yourself makes sense when the target is small, stable and yours: an internal tool, a single vendor page, a site whose markup you can keep an eye on. You keep full control and pull in nothing extra, and that counts for something.
Go with a hosted API or CLI when:
- You are pulling from a lot of sites, or sites you do not control.
- The pages need JavaScript to render, or actively fight bots.
- You want the same JSON shape across sources without maintaining a schema for each one.
- A quiet failure (stale data, missing records, a dashboard that is confidently wrong) costs you more than a per-request fee.
It comes down to control against upkeep. Rolling your own is cheap to start and expensive to keep running against targets that keep moving. A service charges per call but eats the layout changes, the browser overhead and the retry logic that otherwise piles up in your code. Once you are past a few sources, that upkeep usually outweighs everything else, and handing it off is what stops your pipeline from falling apart while you are off building other things.
So start with the simplest thing that survives your real targets. Parse by hand when the site is small and stable. Bring in a browser when the page will not render without JavaScript. Move to a hosted API or CLI when the number of sources, or the cost of things breaking, gets bigger than what you feel like maintaining yourself.