Follow the right paths.
Planned URL scope, sitemap discovery, and duplicate controls would help keep a crawl focused on the questions you set out to answer.
INDEPENDENT SPIRIT. OPEN-SOURCE AMBITION.
Small crab.
Wide web.
Meet WebKrab. A web crawler being planned for people who want to explore the web with clarity, control, and care.
Open-source code development begins soon.
Built for a more considerate crawl.PLAN · LEARN · EXPLORE
01 / A CLEARER WAY TO CRAWL
Go deeper.
Leave a lighter footprint.
A crawler should do more than collect URLs. It should explain where it went, respect the sites it visits, and leave you with data you can actually use.
That is the direction for WebKrab: an upcoming open-source project with a focus on deliberate discovery, bounded requests, and readable results. The capabilities below are design goals, not available software.
Meet the projectKnow when to pause.
Robots rules, per-origin pacing, request budgets, and bounded retries belong at the center of the design.
Make every visit count.
The goal is inspectable results: response metadata, discovery context, and straightforward exports, with clear reasons for skipped URLs.
02 / OUR NORTH STAR
The web is shared.
Our approach should
reflect that.
Responsible crawling starts before the first request. Define the boundary, honor publisher signals, and stop when the evidence says stop.
Explore responsible crawling01
Respect the origin
Identify the crawler, follow robots rules, and limit the load.
02
Make scope explicit
Choose domains, paths, time, and data limits before crawling.
03
Keep the record honest
Distinguish fetched, skipped, failed, and not yet visited.
03 / ON THE HORIZON
A date to
look toward.
Code development will begin soon. We are aiming for an initial open-source release on 10 November 2026 at 00:00 UTC.
This is a tentative target, not a guaranteed release date. The repository, license choice, and first release scope will be announced when confirmed.
See the planned milestonesTENTATIVE LAUNCH TARGET
00:00 UTC · timing may change
The date above is a planning target. A release has not been confirmed.
04 / FIELD NOTES
Before the first request.
Practical thinking on what a capable, responsible crawler needs. These are the ideas shaping WebKrab.
Crawl ethics4 min read
Responsible crawling starts before the first request
A practical design note on robots.txt, failure states, crawler identity, and the decisions that should precede a crawl.
Field note · 01 October 2026Architecture4 min read
Designing a frontier that knows when to stop
Why URL parsing, explicit scope, cautious normalization, and trap detection belong together in a crawler frontier.
Field note · 01 October 2026HTTP field notes4 min read
Fetch less, learn more
Conditional requests, careful retries, and explicit budgets can make HTTP fetching both more useful and less wasteful.
Field note · 01 October 2026