Working on a Real Estate website (similar to http://www.trulia.com/
), I do not want people to be scraping the details of all the properties available on the site.
Therefore, I thought of having an additional "128-bit MD5 key" in the database and not use an incremental ID. (Ie. www.site.com/property?id=d41d8cd98f00b204e9800998ecf8427e
instead of www.site.com/property?id=50
). This way, it will be virtually impossible to look through all the permutations and capture the lot.
However, the problem I am currently facing is : what would prevent someone from getting the whole list of MD5 keys using the search results page ? (a robot can crawl through all the pages resulting of a wide search) All the IDs would then be visible in the search results page (on the link to the detailed page).
Does that constitute a risk ? How can this be avoided ?