How do I create a script to crawl my site and pull data via .csv?

Hello Experts,

My company recently entered the business of redesigning client websites. During the discovery phase of these site redesigns we perform a site content audit. In the audit we gather - site structure information, page names, urls, downloadable documents etc. We hand click through every link and note all of these items in an Excel document. This may sound very primitive to you, which I am sure that it is, and I would like to figure out how to automate this process.

What can I do to move more quickly through these sites while maintaining documentation of what is being found within them?

I have never written a script before, and would not know where to start, but my Development team might. A tool may be more helpful to me at this time, but I am certainly not above asking my team for a bit of assistance with this.

Thank you!
Natalie KuhnAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

If you have never written a script this not one you want to learn on.  It is a complex bit of programming to do what you want.

You probably need to go outside for a solution. Perhaps one of these will be what you need:


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Natalie KuhnAuthor Commented:
Thanks COBOLdinosaur! Those solutions look quite promising.

Yes I was thinking it was not best to start with a script at my level. Would these solutions work for a Mac? I apologize for neglecting to mention that is what I am using.
You would have to look at the detailed requirements.  I believe one of those offers a custom solution, so they for sure would support mac.  I would think for the kind of coding to do what needs to be done it would not be real hardware specific anyway.  A crawler is basically dealing with code on a web page delivered by a server so the client OS should be transparent.

Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Natalie KuhnAuthor Commented:
Have you heard of WebLight?;rcol Would you recommend it?
I'm not familiar with the product, but I don't spend much time in Mac world.  It has a free 30 day trial, so it costs nothing but some of your time to check it out.  I don't see anything in the product description that causes concern.  Because I'm pc-centric rather being in the Mac zone I don't know that I am qualified to recommend Mac products, but from a general high level view i would not recommend against it.

Natalie KuhnAuthor Commented:
Sure, well you have been helpful COBOLdinosaur. Thank you!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.