Solved

Code to check text on a website

Posted on 2012-03-27
2
303 Views
Last Modified: 2012-03-28
I'm trying to write some code that will check regularly (say every few minutes) whether a particular user is online on a website. When that person comes online or goes offline I need it to send me an email. I could run it on a variety of platforms - ideally it would sit on a box that runs Debian Squeeze (6) but if OS X or Windows would be easier then that could work too.

What I'm looking for with this question is just some pointers on the best way to get started with this. I'm not expecting a turnkey solution and am more than prepared to do the learning and put the hard work in myself. Having said that I'm reasonably technical but my Linux skills are only very moderate - I've never written a script for example.

The basic structure is, I think, pretty clear. There needs to be code that checks the website for specific text that matches the name of the user that I'm interested in. When it finds or doesn't find the text it checks against the last known status and if that has changed it sends the email. Delay a while then repeat.

For the website checking I've been experimenting with:

      curl <url> | awk '/<matchtext>/ {<action>}'

That sort of works with some sites. But the specific problem is that the particular website I'm interested in is session based (not sure if I have the terminology right here) in that when I open the site in a browser it logs me in (as an anonymous/guest user - no username or password required) and the page just shows "please wait while we log you on" (or something like that) for 10-15 seconds before the page that I need to search on loads. How can I create that session using code?

As I said at the start I'm just looking for pointers on how to think about this. Really grateful for any help.

Thanks, Chris
0
Comment
Question by:chrwil
2 Comments
 
LVL 11

Accepted Solution

by:
un1x86 earned 500 total points
ID: 37775602
Hi Chris

Not sure if I understood you right. But you could use lynx. If you need to login you could record the steps with

lynx -cmd_log=filename http://theurl.com

and then run the script with

lynx -cmd_script=filename http://theurl.com

cmd_log will log all steps you do on lynx. With cmd_script it will run automated. You could do it as following

use cmd_log script
navigate to login
login
navigate to the webpage where you can see the logged in user
press "p"
enter path where to save page
exit

Then you run
lynx -cmd_script=filename http://theurl.com

this will save the page to the path you have given. You can proceed with that file by grepping its content.

Hope that helps. There is some good documentation out there using cmd_script.
http://blog.unixy.net/2009/06/script-to-automate-browsing-actions-using-lynx/

CURL by the way can do quite a lot with sessions. But you need to know how the login process works on the website. You can send POST variables using CURL and save the generated session.

Just google for CURL post session
0
 

Author Comment

by:chrwil
ID: 37780187
Thanks. I've been playing with Lynx since I saw your post. It may be the answer.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

This is about my first experience with programming Arduino.
Whether you’re a college noob or a soon-to-be pro, these tips are sure to help you in your journey to becoming a programming ninja and stand out from the crowd.
The viewer will learn how to dynamically set the form action using jQuery.
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now