Crawler
About our crawler
Placeholder. This page will be completed before launch (milestone M7). Nothing here is final.
cute-dates runs a small crawler that reads public event calendars and venue pages every few days. It collects facts (title, date and time, place, price) so we can list them and link back to the original page. We write our own short summaries and don't republish source texts or images.
How it behaves
- It identifies itself with a user agent that starts with
cute-dates-botand includes a contact URL. - It respects robots.txt and machine-readable text-and-data-mining opt-outs.
- A robots.txt block for AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended …) stops it too, since it processes pages with a language model.
- It sends one request at a time per site, with a pause between requests.
What happens to a page
A language model reads what the crawler collects from a page to tag the listing (the kinds of date it suits) and to write our summary. The source's own description is kept privately, up to 1,500 characters, for that tagging, and is never published.
Opting out
To stop the crawler from visiting your site, add this to your robots.txt:
User-agent: cute-dates-bot
Disallow: /Opt-outs take effect within 48 hours.
You can also ask us to remove your site or a listing. A contact address will be added here before launch.