{"id":887,"date":"2026-07-10T17:34:46","date_gmt":"2026-07-10T17:34:46","guid":{"rendered":"https:\/\/quickref.me\/blog\/?p=887"},"modified":"2026-07-10T17:34:46","modified_gmt":"2026-07-10T17:34:46","slug":"how-to-pull-structured-data-from-any-website-via-an-api-or-cli","status":"publish","type":"post","link":"https:\/\/quickref.me\/blog\/how-to-pull-structured-data-from-any-website-via-an-api-or-cli\/","title":{"rendered":"How to pull structured data from any website via an API or CLI"},"content":{"rendered":"

You open a product page and see a price. Your code opens the same page and sees a few hundred nested divs, some inline scripts, images that load whenever they feel like it, and class names that will change next Tuesday. What you actually want out of all that is JSON: a title, a price, a couple of fields, done.<\/span><\/p>\n

Doing this for one page is a five-minute job. Doing it for fifty sites that keep changing is where it gets annoying. The extraction is rarely the real problem. The problem is holding a steady output schema in front of sources that will not hold still, and catching it when one of them breaks before your database fills up with nulls. There are three ways to go about it, and each has its own headaches.<\/span><\/p>\n

Approach 1: parse the HTML yourself<\/span><\/h2>\n

Grab the page, parse it, pull out what you need with CSS selectors.<\/span><\/p>\n

import requests<\/span><\/p>\n

from bs4 import BeautifulSoup<\/span><\/p>\n

 <\/p>\n

html = requests.get(“https:\/\/api.example.com\/product\/42”).text<\/span><\/p>\n

soup = BeautifulSoup(html, “html.parser”)<\/span><\/p>\n

 <\/p>\n

data = {<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0“title”: soup.select_one(“h1.product-title”).get_text(strip=True),<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0“price”: soup.select_one(“[data-price]”)[“data-price”],<\/span><\/p>\n

}<\/span><\/p>\n

print(data)<\/span><\/p>\n

 <\/p>\n

Cheap, fast, no real dependencies. Fine when you own the page or it barely changes.<\/span><\/p>\n

The problem is upkeep. Your selectors lean on someone else’s HTML, and they never agreed to keep it stable for you. Somebody renames a class or ships an A\/B test, <\/span>select_one<\/span> starts returning <\/span>None<\/span>, and you find out two weeks later when you check the table and half the rows are empty. Do this across thirty sites and you have basically signed up for a second job keeping selectors alive.<\/span><\/p>\n

Approach 2: drive a headless browser<\/span><\/h2>\n

Plenty of pages ship almost nothing in the initial HTML and build the content with JavaScript after load. For those you need a real browser. Headless Chromium runs the page like a normal one and gives you the DOM once it has settled.<\/span><\/p>\n

import { chromium } from “playwright”;<\/span><\/p>\n

 <\/p>\n

const browser = await chromium.launch();<\/span><\/p>\n

const page = await browser.newPage();<\/span><\/p>\n

await page.goto(“https:\/\/api.example.com\/feed”);<\/span><\/p>\n

await page.waitForSelector(“.card”);<\/span><\/p>\n

 <\/p>\n

const items = await page.$$eval(“.card”, cards =><\/span><\/p>\n

\u00a0\u00a0cards.map(c => ({<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0name: c.querySelector(“h2”)?.innerText,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0url: c.querySelector(“a”)?.href,<\/span><\/p>\n

\u00a0\u00a0}))<\/span><\/p>\n

);<\/span><\/p>\n

 <\/p>\n

await browser.close();<\/span><\/p>\n

console.log(items);<\/span><\/p>\n

 <\/p>\n

This gets you past the JS-rendering wall that plain fetching runs into. You pay for it, though. A browser process per job, more RAM, slower runs, and a fresh set of timing bugs around when the content actually shows up. And you are still writing the same selectors, just with a much heavier thing running them.<\/span><\/p>\n

Approach 3: call a hosted API or CLI<\/span><\/h2>\n

Here you stop doing the extraction yourself. You hand a URL to a service, it hands back structured data. The browser, the parsing, the retries, the layout changes, all of that is on them now.<\/span><\/p>\n

The request is about as dull as it gets, which is the whole point:<\/span><\/p>\n

curl “https:\/\/api.example.com\/extract?url=https:\/\/target.site\/profile” \\<\/span><\/p>\n

\u00a0\u00a0-H “Authorization: Bearer $TOKEN”<\/span><\/p>\n

 <\/p>\n

{<\/span><\/p>\n

\u00a0\u00a0“title”: “Jane Doe”,<\/span><\/p>\n

\u00a0\u00a0“headline”: “Staff Engineer”,<\/span><\/p>\n

\u00a0\u00a0“links”: [“https:\/\/janedoe.dev”],<\/span><\/p>\n

\u00a0\u00a0“text”: “…”<\/span><\/p>\n

}<\/span><\/p>\n

 <\/p>\n

Same thing from a terminal, which is handy for scripts and one-off pulls:<\/span><\/p>\n

extract get –url “https:\/\/target.site\/profile” –format json<\/span><\/p>\n

 <\/p>\n

This is where something like<\/span> Anysite<\/span><\/a> comes in. You give it a URL and it returns structured data over an API or CLI, with ready-made endpoints for the usual platforms (profiles, posts and comments, listings, company and filing data, maps results) and a general parser that hands back cleaned content, links and metadata as JSON when there is no specific endpoint for the site. You get a normalized object back instead of writing and nursing a custom selector for every place you scrape.<\/span><\/p>\n

A concrete flow<\/span><\/h2>\n

Say you want the latest posts from a handful of public profiles. The DIY version means fetching each page, working out whether it needs a browser, writing selectors per site, and dealing with however each one handles pagination. Through a hosted call it turns into one loop:<\/span><\/p>\n

import requests<\/span><\/p>\n

 <\/p>\n

def posts_for(url):<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0r = requests.get(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“https:\/\/api.example.com\/extract”,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0params={“url”: url, “type”: “posts”},<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0headers={“Authorization”: f”Bearer {TOKEN}”},<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0r.raise_for_status()<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0return r.json()[“items”]<\/span><\/p>\n

 <\/p>\n

for profile in profiles:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0for post in posts_for(profile):<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0save(post[“id”], post[“text”], post[“timestamp”])<\/span><\/p>\n

 <\/p>\n

All the per-site weirdness sits behind the endpoint. Your code just reads a stable schema and stops caring which platform the URL points at.<\/span><\/p>\n

Extract responsibly, whatever route you pick<\/span><\/h2>\n

None of this gets you out of the basics. Read the site’s <\/span>robots.txt<\/span> and terms before you start hammering it, and use the official API if there is one. Space your requests out, send a real user agent, and cache so you are not pulling the same URL over and over. Treat failures as normal rather than exceptional: retry with backoff, keep a list of URLs that keep dying, and get alerted when your success rate drops instead of finding out from an empty dashboard. A hosted service takes care of a good chunk of this, but how hard you hit someone’s site is still on you.<\/span><\/p>\n

When a hosted API or CLI wins<\/span><\/h2>\n

Doing it yourself makes sense when the target is small, stable and yours: an internal tool, a single vendor page, a site whose markup you can keep an eye on. You keep full control and pull in nothing extra, and that counts for something.<\/span><\/p>\n

Go with a hosted API or CLI when:<\/span><\/p>\n