🤖 About the SENTINEL URL Checker

Who we are

SENTINEL is an open-web intelligence platform operated by Oblivion Technologies LLC (Wyoming-registered, founded 2026). We maintain a curated directory of websites — an AI-enhanced revival of DMOZ and the Yahoo Directory — and we need to verify that every URL in our index is still alive and functioning.

This crawler is not indexing your content. It is not a search engine spider. It is a narrow URL-liveness checker that makes one HEAD request (falling back to GET if HEAD is rejected) to see if your page returns HTTP 200.

What you will see in your logs

User-Agent: SENTINEL-URLChecker/1.0 (+https://sentinel.oblivionzone.com/bot)
Method:     HEAD (fallback: GET)
Rate:       ≤1 request per second per host
Frequency:  Each URL is re-checked approximately once every 30 days
Source IP:  Resolvable via reverse DNS to our server

We follow up to 5 redirects, honor robots.txt, honor 429/Retry-After headers, and never scrape page content beyond the HTTP response code.

What we do with the result

• If your URL returns HTTP 2xx or 3xx → we keep it listed in our directory and our public sitemap.
• If your URL returns 4xx/5xx consistently for 3+ checks across 7+ days → we remove it from our public sitemap.
• If your URL returns 429 (Too Many Requests) → we back off with exponential delay.
• If your URL returns 403 only to bots → we flag it as "bot-blocked" but keep it listed (we assume it's alive for human visitors).
• If your domain is parked (for-sale landing page) → we remove it as dead.

How to opt out

We respect robots.txt. Add one of these to your robots.txt:

User-agent: SENTINEL-URLChecker
Disallow: /

Or block us entirely in your CDN / WAF by user-agent. We will see the 403 and mark the URL as "bot-blocked" (not "dead") — you will NOT be de-listed from our directory for opting out of the liveness check.

If you'd prefer your site be removed from our directory entirely, email [email protected] with the URL. We process removals within 72 hours and log them publicly for transparency.

Why we do this

The open web is full of dead links. DMOZ, Yahoo Directory, and every human-curated directory that came before us eventually collapsed under the weight of link rot. We are determined not to repeat that mistake.

Every URL in our public sitemap has been verified to return a valid HTTP response within the last 30 days. When you click a link on SENTINEL, you should land on a real page — not a 404, not a parked domain, not an expired cert.

That is the promise. The URL checker is how we keep it.

Contact

Questions, complaints, opt-outs, false-positive reports, or "please crawl my site I'm new":

Email: [email protected]
Operator: Oblivion Technologies LLC
Address: 30 N Gould St Ste R, Sheridan, WY 82801, USA
DMCA agent: US Copyright Office #DMCA-1070891