Solved

Code to check text on a website

Posted on 2012-03-27
2
311 Views
Last Modified: 2012-03-28
I'm trying to write some code that will check regularly (say every few minutes) whether a particular user is online on a website. When that person comes online or goes offline I need it to send me an email. I could run it on a variety of platforms - ideally it would sit on a box that runs Debian Squeeze (6) but if OS X or Windows would be easier then that could work too.

What I'm looking for with this question is just some pointers on the best way to get started with this. I'm not expecting a turnkey solution and am more than prepared to do the learning and put the hard work in myself. Having said that I'm reasonably technical but my Linux skills are only very moderate - I've never written a script for example.

The basic structure is, I think, pretty clear. There needs to be code that checks the website for specific text that matches the name of the user that I'm interested in. When it finds or doesn't find the text it checks against the last known status and if that has changed it sends the email. Delay a while then repeat.

For the website checking I've been experimenting with:

      curl <url> | awk '/<matchtext>/ {<action>}'

That sort of works with some sites. But the specific problem is that the particular website I'm interested in is session based (not sure if I have the terminology right here) in that when I open the site in a browser it logs me in (as an anonymous/guest user - no username or password required) and the page just shows "please wait while we log you on" (or something like that) for 10-15 seconds before the page that I need to search on loads. How can I create that session using code?

As I said at the start I'm just looking for pointers on how to think about this. Really grateful for any help.

Thanks, Chris
0
Comment
Question by:chrwil
2 Comments
 
LVL 11

Accepted Solution

by:
un1x86 earned 500 total points
ID: 37775602
Hi Chris

Not sure if I understood you right. But you could use lynx. If you need to login you could record the steps with

lynx -cmd_log=filename http://theurl.com

and then run the script with

lynx -cmd_script=filename http://theurl.com

cmd_log will log all steps you do on lynx. With cmd_script it will run automated. You could do it as following

use cmd_log script
navigate to login
login
navigate to the webpage where you can see the logged in user
press "p"
enter path where to save page
exit

Then you run
lynx -cmd_script=filename http://theurl.com

this will save the page to the path you have given. You can proceed with that file by grepping its content.

Hope that helps. There is some good documentation out there using cmd_script.
http://blog.unixy.net/2009/06/script-to-automate-browsing-actions-using-lynx/

CURL by the way can do quite a lot with sessions. But you need to know how the login process works on the website. You can send POST variables using CURL and save the generated session.

Just google for CURL post session
0
 

Author Comment

by:chrwil
ID: 37780187
Thanks. I've been playing with Lynx since I saw your post. It may be the answer.
0

Featured Post

3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

930 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now