my-library

OpenLibrary import

Turning a list of titles into three thousand catalogued books, and the matching problems that make it harder than it sounds.

The pipeline

title + author │ ▼ search.json ──────────────────────────────▶ best match? ──no──▶ _misses.json │ key, cover_i, isbn, publisher, (reviewed by hand) │ first_publish_year, pages, │ ratings_average, ratings_count ▼ /works/{OLID}.json ─────────────────────────▶ description │ ▼ library.NNN.json shards ──(review)──▶ import:library:commit ──▶ Postgres

One search.json call returns almost everything, because the fields parameter lets you ask for exactly the columns the schema has — including ratings_average and ratings_count. The one thing it does not carry is the description, which needs a second call to the work endpoint using the OLID the first call returned. That is why openlibraryId is stored: it is the stable key for going back for more.

Nothing writes to the database by accident

The import is deliberately two commands, with a file and a human in between:

npm run enrich:library          # OpenLibrary → JSON shards. Never touches Postgres.
#   …review scripts/enriched/…
npm run import:library:commit   # shards → Postgres, deduped by OpenLibrary id

Matching by title is unavoidably fuzzy, so the output is reviewed before it becomes data. Every wrong row caught in a JSON file is a row that never has to be found and corrected in the database later.

A one-hour job has to be resumable

~3,800 books at two calls each, politely spaced, is roughly an hour of wall time. A run that loses everything on an interruption is a run that never completes, so enrich-library.ts is built around that assumption:

Concern	How
Resume	A `_done.json` set keyed by normalised author+title. A re-run skips what is finished — including books added to the input since.
Bounded files	Output is sharded at 250 books per file (`library.000.json`…). One 3,800-entry JSON is unreviewable.
Checkpoints	State is flushed every 20 books and on SIGINT, so Ctrl+C is a safe way to stop.
Survivable errors	A per-book try/catch. One malformed record cannot end the run — it becomes a logged miss.
Transient failures	fetch retries on network errors, 429 and 5xx with a backoff. Over ~7,600 calls, blips are certain.

This was learned the hard way

The first full run died about 140 books in. OpenLibrary returned an empty string for a publisher, zod’s .min(1) rejected it, and the exception took down the whole process. The fix was two things: treat blank strings as null at the mapping boundary, and wrap each book so no single record can ever end a run again.

Matching is the actual problem

The source list came from folder names, which carry series prefixes and filename-safe punctuation. Those have to be cleaned before searching:

Throne of Glass Book 0.5 The Assassin's Blade   →  The Assassin's Blade
Crescent City 2_ House of Sky and Breath        →  House of Sky and Breath
DC Icons Catwoman_ Soulstealer                  →  Catwoman: Soulstealer

Then the search itself needs judgement, in this order:

Author-scoped search, exact title match. The best case.
Author-scoped, top result. Author plus title is already a narrow query, so the first hit is trustworthy.
Title-only, but only on an exact match. Without the author constraint the top result is frequently junk — a study guide, a box set, an unrelated book with a short shared title.

Two failures that shaped that order

Searching A Court of Mist and Fury returned a four-book omnibus, because OpenLibrary ranked the bundle first. Preferring an exact title match over the top result fixed it.

Then the title-only fallback matched Catwoman: Soulstealer to a “Study Guide Student Workbook” for it. Requiring an exact match in the fallback turned that into a clean miss — the right outcome, because a missing book is visible and a wrong one is not.

Results

	Books	Matched
Drive catalogue	3,785 unique	3,333 (88%)
Local ebook files	173	117, of which 51 trusted

3,323 of the enriched books went into Postgres (10 were duplicate OLIDs), plus 44 from the local files — the 3,367 that are there now.

Local files were a different problem

A second source was a folder of Kindle and PDF files. Their filenames carry a title and sometimes an ASIN, but no author — and without an author, title-only matching produces exactly the junk described above. Confidence is therefore recorded per book:

Tier	Count	Trust
asin	3	High — OpenLibrary's own id_amazon lookup.
title+author	27	High — author extracted from the file itself.
full-title	21	Good — exact match on the complete title.
reduced-title	69	Not trusted — matched only after stripping a subtitle or series.

Only the first three tiers were imported. The reduced-title tier contains real false positives — a Red Bourbon Mysteries thriller matched a children’s poetry collection sharing the title A Kick in the Head — and 69 quietly wrong books is worse than 69 missing ones.

Reading authors out of the files

The way to rescue those matches is to get the real author, which the files themselves contain. scripts/ebook-meta.ts parses it with no external tools:

Format	Where the author lives
`.azw3` `.azw` `.mobi`	Mobipocket EXTH header, record type 100.
`.pdf`	/Author in the Info dictionary, or dc:creator in an XMP packet.
`.kfx`	Amazon's proprietary container — not parsed. This is 135 of the 173 files.

It recovered an author from all 38 non-KFX files, which promoted 27 books to author-scoped matching and fixed several that were previously wrong — Atomic Habits to James Clear, Project Hail Mary to Andy Weir. The KFX majority stays filename-only, which is the honest ceiling without Amazon-specific tooling.

Getting the list out of Google Drive

The source catalogue was a Drive folder of ~2,100 author folders. Clicking through it in a browser was hopeless, so export-catsby-drive.gs is an Apps Script that walks the tree and writes a CSV. Apps Script kills a run at ~6 minutes, so it works in chunks:

scripts/export-catsby-drive.gs

var START_AUTHOR = 0;    // move this forward each run
var MAX_AUTHORS  = 150;
var TIME_BUDGET_MS = 5 * 60 * 1000;   // stop before the platform does

Authors are sorted by name before slicing, so chunk boundaries are stable between runs and nothing is skipped or repeated. When the budget runs out it stops and prints the index to resume from — the same resumability the enrichment needed, for the same reason.

The URLs this produces are the subject of Cover images.