Developer docs

The whole catalog, on your machine.

Subsets is a search engine and SQL layer over the world’s public statistical data — and it runs locally. One pip install clones cleaned, versioned tables straight from the public bucket and serves search and SQL on localhost. Free, open source, no account, no API key.

Overview

CLI

pip install subsetsio — clone one table, one source, a size bundle, or the whole catalog as valid Delta tables.

Local API

subsets serve puts search, metadata, and SQL on http://localhost:8080 — the endpoint you point your agents and notebooks at.

Open data

Data files come straight from data.subsets.io; every table is built by an open-source connector in a logged CI run.

Every dataset has a human-readable ID like ember-global-yearly. That ID is the table name in SQL, the argument to subsets sync, and the path segment in local API URLs. Datasets are built by open-source connectors in logged CI runs — browse them all on the connectors page.

Quickstart

From zero to a local, SQL-queryable catalog in about two minutes. No sign-up.

Bash

pip install subsetsio
subsets sync --bundle small
subsets serve

sync --bundle small clones a starter tier of the catalog (tens of MB); use medium or all for more, or name a source or dataset ID directly. subsets serve then exposes everything you synced at http://localhost:8080:

Bash

curl -s "http://localhost:8080/search?q=inflation"
curl -s -X POST "http://localhost:8080/query" \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT * FROM \"ember-global-yearly\" LIMIT 5"}'

Point an AI agent at those same endpoints and it can search the catalog, inspect schemas, and run SQL — all against data on your own disk.

CLI

The subsetsio package installs a subsets command: mirror one connector or the whole catalog as valid Delta tables, then query them with the local server — or directly with DuckDB, pandas, Spark, anything that reads Delta.

Bash

subsets list                   # per-source totals for the whole catalog
subsets list -v ember          # every table for one connector
subsets sync ember             # clone one connector's tables
subsets sync ember-global-yearly   # clone a single table
subsets sync --bundle small    # clone a size tier: small, medium, or all
subsets sync                   # clone everything
subsets sync --dry-run         # see what would change, download nothing
subsets serve                  # local search + SQL API on localhost:8080

Sync is incremental and idempotent: the client fetches the catalog manifest — every published table with its pinned version — in one request, compares each pin against the local copy, and only touches tables whose version moved. Files download directly from the public bucket; there are no credentials involved anywhere.

Flags

Flag	Meaning
--bundle {small,medium,all}	Bootstrap a cold clone from pre-packed size tiers: small = the long tail of small sources (tens of MB), medium = adds mid-size (~1 GB), all = the full catalog. Anything the bundles left stale is topped up per-table
--dir	Destination (default $SUBSETS_MIRROR_DIR or ./subsets-data)
-j, --jobs	Concurrent tables (default 8)
--max-mb	Skip tables larger than this many MB
--prune	Remove local tables no longer in the catalog selection
--dry-run	Plan only — show what would sync

Environment

Variable	Meaning
SUBSETS_MIRROR_DIR	Where synced tables live and what subsets serve serves (default ./subsets-data)

Local API

Run subsets serve and everything below is available at http://localhost:8080. It’s your machine — there is no authentication and no rate limit.

Endpoint	Returns
GET /search	Keyword search over your synced catalog
GET /datasets/:id	Metadata and schema for one dataset
POST /query	Run SQL against your synced tables
GET /catalog	Every synced table with its pinned version
GET /health	Server liveness — handy for agent tooling

Search

GET /search

Bash

curl -s "http://localhost:8080/search?q=carbon&limit=20"

Parameters: q (keywords), limit. A blank q returns recently updated datasets.

Dataset detail

GET /datasets/:id

Bash

curl -s "http://localhost:8080/datasets/ember-global-yearly"

One call gets everything an agent needs before querying: schema (column names and types), description, license, and size.

SQL

POST /query

Bash

curl -s -X POST "http://localhost:8080/query" \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT area, year, value FROM \"ember-global-yearly\" WHERE variable = '\''Renewables'\'' LIMIT 20"}'

Body: sql (DuckDB dialect — dataset IDs are the table names, double-quoted; joins across tables just work). Returns { columns, rows, row_count }. SQL errors come back as 400 with the engine’s message; an unsynced table is a 404 — subsets sync <id> fixes it.

Writing SQL

Queries run on DuckDB against your local Delta tables. Dataset IDs are the table names.

SQL

SELECT year, value
FROM "ember-global-yearly"
WHERE area = 'EU' AND variable = 'Renewables' AND unit = '%'
ORDER BY year DESC
LIMIT 10

Quote dataset IDs. They contain hyphens, which SQL otherwise parses as subtraction: FROM "ember-global-yearly".
Read-only. Results are returned directly as { columns, rows, row_count }; nothing is written.
Check the schema first. GET /datasets/:id returns column names and types — one lookup saves a failed query.
No server required. The synced tables are plain Delta directories — open them straight from DuckDB, pandas, or Spark if you’d rather skip HTTP entirely.

Data & licensing

Everything is fetched directly from data.subsets.io: the catalog manifest, dataset metadata, the search index, and the data files themselves. There is no API in front of it — the CLI and this website read the same public JSON.

The connectors and the subsetsio package are open source under the MIT license, and the service is free. Each dataset carries its upstream provider’s own license (CC-BY, public domain, provider terms, …) — it’s shown on every dataset page and in the metadata.

Agents: point them at the local endpoints. A Claude skill that teaches your agent to read from the local server is on the way. Something look wrong in the data? Every number has a paper trail — start from the connectors page or the connector source on GitHub.