Solved

Code to check text on a website

Posted on 2012-03-27
2
347 Views
Last Modified: 2012-03-28
I'm trying to write some code that will check regularly (say every few minutes) whether a particular user is online on a website. When that person comes online or goes offline I need it to send me an email. I could run it on a variety of platforms - ideally it would sit on a box that runs Debian Squeeze (6) but if OS X or Windows would be easier then that could work too.

What I'm looking for with this question is just some pointers on the best way to get started with this. I'm not expecting a turnkey solution and am more than prepared to do the learning and put the hard work in myself. Having said that I'm reasonably technical but my Linux skills are only very moderate - I've never written a script for example.

The basic structure is, I think, pretty clear. There needs to be code that checks the website for specific text that matches the name of the user that I'm interested in. When it finds or doesn't find the text it checks against the last known status and if that has changed it sends the email. Delay a while then repeat.

For the website checking I've been experimenting with:

      curl <url> | awk '/<matchtext>/ {<action>}'

That sort of works with some sites. But the specific problem is that the particular website I'm interested in is session based (not sure if I have the terminology right here) in that when I open the site in a browser it logs me in (as an anonymous/guest user - no username or password required) and the page just shows "please wait while we log you on" (or something like that) for 10-15 seconds before the page that I need to search on loads. How can I create that session using code?

As I said at the start I'm just looking for pointers on how to think about this. Really grateful for any help.

Thanks, Chris
0
Comment
Question by:chrwil
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 11

Accepted Solution

by:
un1x86 earned 500 total points
ID: 37775602
Hi Chris

Not sure if I understood you right. But you could use lynx. If you need to login you could record the steps with

lynx -cmd_log=filename http://theurl.com

and then run the script with

lynx -cmd_script=filename http://theurl.com

cmd_log will log all steps you do on lynx. With cmd_script it will run automated. You could do it as following

use cmd_log script
navigate to login
login
navigate to the webpage where you can see the logged in user
press "p"
enter path where to save page
exit

Then you run
lynx -cmd_script=filename http://theurl.com

this will save the page to the path you have given. You can proceed with that file by grepping its content.

Hope that helps. There is some good documentation out there using cmd_script.
http://blog.unixy.net/2009/06/script-to-automate-browsing-actions-using-lynx/

CURL by the way can do quite a lot with sessions. But you need to know how the login process works on the website. You can send POST variables using CURL and save the generated session.

Just google for CURL post session
0
 

Author Comment

by:chrwil
ID: 37780187
Thanks. I've been playing with Lynx since I saw your post. It may be the answer.
0

Featured Post

Why Off-Site Backups Are The Only Way To Go

You are probably backing up your data—but how and where? Ransomware is on the rise and there are variants that specifically target backups. Read on to discover why off-site is the way to go.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This is about my first experience with programming Arduino.
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
Six Sigma Control Plans

729 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question