Thingbot is the web crawler that powers Thingful, a search engine for the Internet of Things. It visits public IoT data networks, indexes the devices and sensors they expose, and makes them discoverable on the Thingful map with links back to the original source. It is not a data repository, but makes other data repositories discoverable.
Thingbot only indexes publicly accessible IoT networks — platforms that explicitly share their sensor data openly. It does not attempt to bypass password-protected resources, private APIs, or any data that is not already openly published by the network operator.
Thingbot runs roughly every one to two months. It is not a continuous crawler; between runs it is completely idle and makes no requests.
All requests made by Thingbot carry the following User-Agent header:
Thingbot respects robots.txt. To block it, add the following to your site's
robots.txt:
Thingbot pauses for several seconds between requests to any given host. For networks that publish explicit rate-limit guidelines, it follows those instead.
If you have questions or would like your network removed from the index, please email: info@thingful.net