The main purpose of my project is to check about 100,000 websites if they exist (not domain check only website check). Currently the way I am doing it is to run file_get_content for every url and add the number of characters returned to the database. So if the characters are over 0 then the website exists.
However this way takes way too long (more than 2 days) and the results are not very good (I have to run this 3-4 times to get better results since many website do not respond fast).
Do you have any ideas to improve this? For example I think that getting just the first 4-5 characters of the response could work as well. Or another thing could be to launch many instances of the script at the same time.
Thanks for all the help!