KelpieDealsBot
KelpieDealsBot is the crawler behind Kelpie Deals. It checks public prices of audio, TV and home theatre, computers, PC parts and peripherals, phones and wearables, cameras, gaming, and home appliances at JB Hi-Fi, The Good Guys, Mwave, Addicted to Audio, Myer, Target, PLE Computers, Umart, Betta, Shopping Express, digiDIRECT and Camera House so we can show price history and genuine price drops. If you run one of these sites and have any concern, please email us and we'll respond quickly.
What it fetches
- Category listing pages for the product categories we track, to find products and their current prices.
- Product sitemaps (the ones listed in robots.txt), where a store's category pages can't be read, to find new products.
- Product pages for those products, to confirm price, stock and barcode.
- robots.txt, regularly, so it always follows the current rules.
It doesn't log in, add to cart, submit forms, run checkout, or download images or media.
How politely
- At most 20 requests per minute per store, with requests spaced out rather than in bursts. Current limits:JB Hi-Fi 15/min, The Good Guys 8/min, Mwave 8/min, Addicted to Audio 6/min, Officeworks 6/min, Myer 6/min, Target 4/min, PLE Computers 6/min, Umart 6/min, Betta 6/min, Shopping Express 6/min, digiDIRECT 6/min, Camera House 6/min, PC Case Gear 6/min.
- It honours robots.txt, including Disallow rules.
- It honours
Retry-Afterand backs off on HTTP 429 and 503 responses; repeated blocks pause it for that store until a person reviews it.
How it identifies itself
Every request carries a User-Agent starting with KelpieDealsBot and a link to this page. Requests are also signed withWeb Bot Auth (HTTP Message Signatures, RFC 9421), so sites can verify they really come from us. Our public keys are published at:
https://kelpiedeals.com.au/.well-known/http-message-signatures-directory
Contact
To ask us to slow down, stop, or anything else: hello@kelpiedeals.com.au.