If you found this page in your server logs, this explains what the request
was and how to make it stop.
To stop it: email
crawler@propspine.co.uk and it will be stopped
within one working day, no explanation needed. You can also block the
user-agent below, or add a Disallow for it in your
robots.txt — both are honoured automatically and neither will be
worked around.
What it is
A crawler operated by Jack Ballinger as part of a UK residential
property research project. It reads publicly available property listing pages
and records structured summary data — price, status, date listed, bedroom
count, property type, location — so that changes over time can be measured.
The same operator also runs a smaller crawler, propspine/0.1,
which reads published open-data files and mapping services from local
authorities and the Department for Education. It behaves as described below,
and the same address stops it.
One request at a time, at most one per second per site, run outside peak hours.
robots.txt
Read at the start of every run and enforced in code, including wildcard
and Crawl-delay directives.
Backing off
A 429 or 503 is honoured, including
Retry-After. Repeated throttling or a 403 ends
the run rather than triggering a retry.
No evasion
No proxy rotation, no IP cycling, no user-agent spoofing, no CAPTCHA
circumvention. It requests as itself or it does not request.
Scope
Publicly accessible listing pages only. No logged-in areas, no paywalled
content. No names, phone numbers, email addresses or other contact
details of individuals are recorded; the marketing agency's branch
identifier is retained so that listings can be grouped, and nothing
further about the agency or its staff.
Images
Not stored. Where a photograph is processed, only a perceptual hash is
retained so that the same photograph can be recognised across listings.
No copy of the image is kept.
Republication
Nothing is republished verbatim. Listing text, descriptions and
photographs are not redistributed in any form. Only derived statistics
are produced.
Contact
Jack Ballinger — crawler@propspine.co.uk.
Questions, complaints and requests to stop are all welcome at the same
address, and a request to stop does not need to be justified.