About our crawler
Building Energy Group lists installers by state and county. To keep those lists current we read, once a week, the public lists that licensing boards and state energy programs publish. A program reads them for us: a crawler named BuildingEnergyGroupBot. This page says what it reads, when it comes and how it behaves, so you can recognise it in your logs and tell us to stop.
What it reads
These public sources, one state a day:
- Monday, California. CSLB: the Contractors State License Board's lists of licensed contractors, by classification.
- Wednesday, Virginia. Virginia DPOR: the Department of Professional and Occupational Regulation's lists of licensed contractors.
- Thursday, Maryland. Montgomery County: Montgomery County's list of master electricians, from its open data portal.
- Thursday, Maryland. MEA Solar Access Program: the Maryland Energy Administration's list of contractors in its Solar Access Program.
- Friday, Pennsylvania. PA Solar Center: the PA Solar Center's directory of qualified solar installers, one county page at a time.
- Saturday, New York. NYS Clean Heat: the NYS Clean Heat contractor search, a page of contractors at a time.
Each source is read once a week, on its day. We can start a read by hand on another day; it follows the same rules.
It reads public pages and files only. It does not read anything behind a captcha, a login, a payment page or a box you must tick to accept terms, and it does not work around any of them. It does not read a source whose terms forbid copying it. Where a state closes its list that way, the list reaches us only if a person downloads it by hand, and the crawler plays no part.
How it behaves
- It makes one request at a time, and waits at least 5 seconds between two requests to the same site. It waits 10 seconds for the sources it reads page by page (PA Solar Center and NYS Clean Heat). If your robots.txt asks for a longer wait with
Crawl-delay, it waits that long. - It reads your robots.txt first and obeys it. A site with no robots.txt file is read as allowing every public page. If the file cannot be read, the crawler leaves your site alone for that run.
- It asks before it downloads where a file says when it last changed. For Virginia DPOR it sends the
ETagorLast-Modifiedvalue from its last visit, so a file that has not changed costs one small request. It reads every other source in full, once a week. - It follows a redirect only to another page on the same site, and never to a different one.
- It sets its own limits: a fixed number of requests for each run, a largest size for each response, and a time after which it gives up on a request.
- It stops when asked. At the first 403, 429 or 503 answer it ends the run and leaves that source alone for 7 days. It does not retry.
Its name
Every request carries this user agent:
BuildingEnergyGroupBot/1.0 (+https://buildingenergygroup.co/about-our-crawler/)
To keep it off part of your site, add a group for BuildingEnergyGroupBot to your robots.txt. It matches its rules by that name.
What we do with what it reads
We keep a business’s name, licence number and class, city, county, phone number and website as the source gives them, and we print on each listing where it came from and the date we read it. Some registries list a licence under the holder’s own name (Montgomery County’s master electricians are people), and that name is the listing’s name. We publish no street address, no email address and no contact name.
Reaching us
To ask us to keep the crawler off a page or to pause a listing, there will be an address here once buildingenergygroup.co has an inbox. It has none yet.