Revival by Aspire Web

Aspire Web has revived this project as it aligns with our needs for getting information on the .au domain space.

There are some minor changes to this website (no download option for now) but also to the dataset as we've had to update and modify where the data is coming from.

The below "about us" section we've left intact for now for historical purposes and is not correct. Over time our goal is to integrate this into new domain tools we are building.

Purpose

This site is a project to make searching the .au ccTLD easier. In other words, it is a giant list of .au domains and subdomains.

I was not too happy with the fact that it is quite difficult to get your hands on the .au zonefile. This service certainly isn't the equivalant of having your hands on that zonefile, but it does have a few notable differences:

  • Lists subdomains where possible
  • Is historical - shows past entries and when they were first/last seen
  • Is free for all uses

The primary sources for this index are:

  • Certificate Transparency logs (hourly)
  • Common Crawl URL Index (monthly)
  • AusRegistry/auDA Drop List (multiple times an hour)
  • Censys.io Raw Data (occasional backfill)

Other sites certainly exist that to index this kind of data (like crt.sh, censys.io), but I have found them unusable when it comes to exploring the .au ccTLD. That's why this exists.

The full source code to every component, including this website, are available on Github:

github.com/ausdomainledger

Some example queries to give you inspiration

API

An API is available for any purposes. I will document it briefly here.

Query

The query endpoint is slightly complex due to the underlying schema. Refer to the source code of this front-end application for a reference implementation. 

GET https://api.ausdomainledger.net/api/v1/query

query
URL-encoded string to search for. Works like "SQL LIKE". You may use "%" as a wildcard character.
limit
How many records to return. The server may return less records than requested based on server policy.
from_time
This is the main pagination mechanism. Use the lowest `first_seen` you saw in a previous result set for the next records.
last_id
If you receive a full result set with fully identical `first_seen` timestamps, you can use `last_id` as a secondary pagination key. This is based on the `id` and `last` fields in a result.

Statistics

A statistics endpoint is available which shows how many domains and eTLD+1s are indexed.

GET https://api.ausdomainledger.net/api/v1/stats

Research

Australian cPanel Webhost Size Showdown - A comparison of cPanel Shared Hosting Providers in Australia at the end of 2017