Method · 5 min read

The free certificate log everyone depends on is down more than it is up

crt.sh is the only free substring search over Certificate Transparency, it fails constantly, and most tools built on it report a clean result when it does. What we did instead.

Certificate Transparency is the best starting point for finding lookalike domains. Every publicly-trusted HTTPS certificate is logged, and a credential-harvest page needs HTTPS or the browser flags it — so the logs see the domain before anyone visits it.

The problem is getting at them. Searching the logs by *substring* — "show me every hostname containing `ledger`" — is what surfaces names nobody thought to generate, and exactly one free service does it: crt.sh.

It fails, constantly

Over a working week of testing, crt.sh returned HTTP 502 or timed out on the majority of queries. It runs on one heavily loaded Postgres instance serving the entire security industry for free, which is a remarkable public good and not a dependency you can build a paid product on.

The failure that matters is not the outage

It is what a tool does about it. If a scanner treats "the provider did not answer" the same as "the provider answered with nothing", every outage becomes a clean bill of health — delivered with the same confidence as a real one.

That is the worst possible failure for this category, because the customer acts on it. They read "no lookalike domains found" and stop looking.

What we did

Report the provider, not just the result. A scan returns `status: unavailable` with per-source detail, and the word "unavailable" appears in the customer-facing output. "Found nothing" and "could not look" are different answers and they are never merged.

Add a keyless second source. RapidDNS indexes on the hostname string rather than a registered parent, so a bare brand token still returns names across every TLD. With crt.sh failing every query, it returned 85 hostnames for one wallet brand, 19 of which scored as candidates.

Refuse to buy the obvious backstop. SSLMate's Cert Spotter is the usual recommendation, and its free allowance is a 30-day trial rather than a free tier. It is a fine paid backstop and a bad plan.

The general lesson

Every detection product is a pile of third-party data sources, and each one will fail. The engineering that matters is not the happy path — it is whether a dead source is visible in the output or silently becomes a reassuring answer.