Portfolio / Scraping & data pipelines
Digital Builder
A niche-research tool that fans one query out to four live public data sources — and refuses to display a number it didn't just fetch. A local rebuild of a product that gates the same research behind $44–$149 a month.
Built and run by Joel Wickremasinghe
running in production
21
bugs found and then independently confirmed by 25 adversarial review agents — none rejected
measured · review workflow artifact
Problem
What needed to exist
Choosing a niche is the decision that determines whether months of content work compound or evaporate — and the tools that inform it are subscription-gated and rate-limited. The commercial equivalent, digitalmaker.ai, caps its paid tier at 15 competitor scans a week and charges up to $149 a month for the privilege.
The deeper problem is what those tools display. Research products in this category habitually show confident, plausible-looking figures with no stated source, which is precisely how someone commits a year to a niche on the strength of a number nobody computed.
What we built
The system
A local Express server and a vanilla-JS front end — no cloud, no database, no build step, no account. One typed topic fans out in parallel to four live public sources: a 42-query Google Autocomplete sweep, Google Trends through its undocumented widget-token flow, twelve months of real Wikipedia readership, and YouTube view and channel data parsed out of the public page payload.
Those four feed three sub-scores — demand, keyword breadth, competition — that combine into a 0–100 opportunity score, which is then run through a published-benchmark ad-rate table by niche and audience geography. Four generators can turn the result into an ebook, planner, spreadsheet, or script, and the prompt builder targets ten AI models across text, image, and video.
The rule the whole thing is built around: no mock data anywhere. Every research figure is fetched at request time, and when a source fails the interface names the source that failed rather than quietly substituting a plausible value. Google Trends rate-limits this address often, so demand degrades to live Wikipedia readership — and the report says which one it used.
Architecture
Digital Builder — one query, four live sources
Stack
Built with
- Node.js 22.16
- Express 4.22.2 (only dependency)
- Vanilla JS · HTML · CSS
- Hand-rolled SVG charts
- Google Autocomplete
- Google Trends (undocumented endpoints)
- Wikipedia Pageviews API
- YouTube page parsing
- Anthropic Messages API (optional, user's key)
Outcome
Numbers, with provenance
21
bugs found and independently confirmed by 25 review agents (0 rejected) — 7 major, 14 minor, across 1.43M tokens and 187 tool calls
measured · review workflow artifact
4 / 4
live sources returning HTTP 200 when measured — autocomplete 2.67s, trends 3.32s, Wikipedia 1.69s, YouTube 1.69s
measured · timed HTTP calls, 2026-08-01
2.0–3.0s
end-to-end for a full four-source report across three test seeds (2,041 / 2,829 / 3,044 ms)
measured · timed /api/niche calls
392
unique keywords surfaced from 42 autocomplete probes for one seed — varies per run, because it is fetched live
measured · /api/keywords response
12 / 12
regression tests passing
measured · suite run 2026-08-01
9 / 9
text contrast ratios at or above 4.5:1 — WCAG AA across all three themes
measured · computed from live DOM
0
mock, cached, or seeded values in the research path — every figure is fetched at request time or its source is named as failed
architecture · no-mock-data rule, 11 API routes
1
npm dependency across 2,467 lines of application code — no framework, no bundler, no charting library
measured · npm ls --depth=0
there is no telemetry and no run log — the tool doesn't record what it produced, so there is nothing to report
Every figure above is read from the system of record it describes. Gaps are marked, not filled in.
Also in this system
Supporting detail
The bug that proves the rule
The niche classifier used substring matching, so "ai" matched inside "email" and "domain" — which classified personal finance as Technology & Software, applied the wrong ad-rate band, and produced wrong earnings figures. Found and fixed on 1 August 2026 with whole-word matching and a test. It's worth stating plainly: this is exactly the class of silent error the no-mock-data rule exists to catch.
Degradation is designed, not accidental
Google Trends rate-limits this address frequently. Rather than fail or fabricate, the app scores demand from live Wikipedia readership and labels the report with which source it used. The honest cost: the demand score isn't always computed the same way between two runs of the same query, and the interface says so.
Reddit was cut, not faked
A fifth demand signal was planned. Reddit returns HTTP 403 to this machine on every endpoint tried, so it was removed rather than filled with a placeholder. A missing source is a smaller lie than an invented one.
What isn't proven
The four AI generators are wired to the Claude API but have never completed a run — no key is configured on that machine, so only the failure paths (400 on malformed, 401 on invalid) are tested. The ad-rate estimates have never been checked against a real AdSense account and are labelled as benchmarks, not predictions. And the regression suite lives outside version control, so a fresh clone loses it.
Need data you can actually defend?
Four live sources, zero mock values, and an interface that names the source that failed instead of hiding it. We build collection pipelines that degrade honestly — because a confident wrong number costs more than a visible gap.