Get internet domains/urls

I've tried to dig more into this and there was a kind of a smaller solution I figured out but after looking into it very close I see that it will be a problem in a long run so I'm trying to find the best way of getting let me say the list or all if possible internet web addresses, like if a new domain is registered I should also get it, my question is how can I find these, I know we may think crawling sites, but the reality now is that crawling sites will be something that wont be supported in a near future, that because I've also tried to check some sites that I wanted to crawl so that I can get more sites addresses to visit and crawl but people are now preventing crawling by hiding the addresses or links, like for instance a site that people post business ads for free, they hide the link that shows the full detailed ad details of the post by using some ad id which there script understands and dynamically loads the ad to the user, so crawling such sites require each custom site crawler after you have figured out how are they hiding this.

Bing, Google do find these information, how can I also find this information. I'm more interested in fining domains/sites urls.

Any one who can help on this?