AgingResearchNewsBot

AgingResearchNewsBot is the crawler of AgingResearch.News, which reports new research on human aging. It reads new papers, news and meeting pages so we can write short stories about them, show a paper’s own figure next to its story and link readers to the source.

• A new paper's article page, for its title and its preview image, and its Figure 1 or graphical abstract.
• A paper's open full text where the publisher or a repository offers it (JATS from bioRxiv, medRxiv and Europe PMC, an open-access article page).
• News articles' preview images, and the RSS feeds and APIs outlets publish for programs.
• The program and news pages of a scientific meeting while it is on, and a company's own pages about its work.

It never signs in, fills in forms, follows links behind a paywall or downloads more of a page than it needs.

• One request to a site at a time, at least a second apart, and slower where a site asks: 15 seconds apart at bioRxiv and medRxiv.
• A Crawl-delay in robots.txt is kept, up to a minute between requests.
• A 429 is waited out for as long as its Retry-After says. A refusal stops requests to that site for an hour, then for longer, up to two days.
• Across every site it reads, a few thousand requests a day; at any one site, rarely more than a few dozen.

It follows robots.txt as RFC 9309 sets out, under the product token AgingResearchNewsBot, and the rules for * when no group names it. It reads each site’s file at most once a day. Feeds and APIs a site publishes for programs are read as published.

Its User-Agent is:

Mozilla/5.0 (compatible; AgingResearchNewsBot/1.0; +https://agingresearch.news/bot; +https://agingresearch.news/contact)

Every request is signed with Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519): it carries Signature-Agent, Signature-Input and Signature headers, and the public key is published at https://agingresearch.news/.well-known/http-message-signatures-directory. A request that names AgingResearchNewsBot but is not signed with that key is not ours. Some pages only show in a browser; those are opened in Chrome on our server, and the page itself is asked for with the same signature. A few sites turn away any User-Agent that names a crawler; there it sends a browser’s that ends in AgingResearchNews/1.0, signed all the same.

Add a group for AgingResearchNewsBot to your robots.txt and it will be followed within a day. To ask us anything about the crawler, write from the contact page (choose Something else) and we will answer.

Paperfoot AI Pte. Ltd., Singapore
Write to us from the contact page.