How do you detect in Visual Basic when new questions are posted on the Experts Exchange website, like EE quickpost does ??

hi
    u can get the source(HTML) of the webpage from VB.. u have to compare that with the previous version for the new questions table part... that will do...

regards
rajesh

Isnt there an easier way or another way of doing it ?

I increased points to 500, as I would like an example of how to do this in visual basic 6, in terms of checking for the question's status, ie open, closed, deleted or does not exist yet as well as doing the other stuff you explained IM, if that is ok:)

I'm on vacation so I can't give you any specific code until late next week or the week after that.  I'm guessing you can simply use a WebBrowser control and look for keywords in the response strings.

Stay tuned...

ok, no problem IM, Have fun on your vacation :)

I understand that you're after a VB solution -- but if you're not *too* fussed about the language, then I've just spent the past couple of hours creating this for you:



:======8<--------------[  EEPostScanner.java ]----------------
   
    /**
     *
     *  EE Spider.
     *  By R. Darkins -- InteractiveMind
     *  
     *  # Allows you to scan for the latest questions on the site.
     *  
     *  Time-taken to create: a few hours - lol.
     *  
     */
   
    import java.io.*;
    import java.net.*;
    import java.util.*;
   
    import java.awt.*;
    import java.awt.event.*;
    import javax.swing.*;
    import javax.swing.text.html.*;
    import javax.swing.text.*;
   
   
    public class EEPostScanner extends JFrame implements ActionListener
    {
       
        private int     iScrX           = (int ) Toolkit.getDefaultToolkit().getScreenSize().getWidth();
        private int     iScrY           = (int ) Toolkit.getDefaultToolkit().getScreenSize().getHeight();
       
        private int     iAppX           = 600;
        private int     iAppY           = 400;
       
        private JButton bGo             = new JButton("Go");
        private JButton bStop           = new JButton("Stop");
        private JButton bPause          = new JButton("Pause");
        private JButton bClear          = new JButton("Clear");
       
        private DefaultListModel model  = new DefaultListModel();
        private JList       list        = new JList(model);
        private JTextField  tThread     = new JTextField();
       
        private Thread      t;
        private Vector      v           = new Vector(10, 5);
       
        private boolean     paused      = false;
       
       
        public EEPostScanner ()
        {
            super( "EE QuickPost" );
            setBounds( (iScrX-iAppX)/2, (iScrY-iAppY)/2, iAppX, iAppY );
            setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
           
            JPanel pane = new JPanel();
            setContentPane(pane);
           
            GridBagLayout gridbag = new GridBagLayout();
            GridBagConstraints constraints = new GridBagConstraints();
           
            pane.setLayout(gridbag);
           
            JLabel label_start = new JLabel( "Starting Q: #" );
            buildConstraints( constraints, 0, 0, 1, 1, 40, 15 );
            constraints.fill = GridBagConstraints.NONE;
            constraints.anchor = GridBagConstraints.EAST;
            gridbag.setConstraints( label_start, constraints );
            pane.add( label_start );
           
            tThread.setForeground(new Color(60, 112, 182));         // RGB
            buildConstraints( constraints, 1, 0, 2, 1, 40, 15 );
            constraints.fill = GridBagConstraints.HORIZONTAL;
            gridbag.setConstraints( tThread, constraints );
            pane.add( tThread );
           
            buildConstraints( constraints, 3, 0, 1, 1, 20, 15 );
            constraints.fill = GridBagConstraints.NONE;
            constraints.anchor = GridBagConstraints.CENTER;
            gridbag.setConstraints( bGo, constraints );
            pane.add( bGo );
           
            JScrollPane scroller = new JScrollPane(list);
            buildConstraints( constraints, 0, 1, 4, 1, 100, 70 );
            constraints.fill = GridBagConstraints.BOTH;
            gridbag.setConstraints( scroller, constraints );
            pane.add( scroller );
           
            buildConstraints( constraints, 0, 2, 1, 1, 40, 15 );
            constraints.fill = GridBagConstraints.NONE;
            constraints.anchor = GridBagConstraints.CENTER;
            gridbag.setConstraints( bStop, constraints );
            pane.add( bStop );
           
            buildConstraints( constraints, 1, 2, 1, 1, 20, 15 );
            gridbag.setConstraints( bPause, constraints );
            pane.add( bPause );
           
            buildConstraints( constraints, 2, 2, 1, 1, 20, 15 );
            gridbag.setConstraints( bClear, constraints );
            pane.add( bClear );
           
            bGo.addActionListener(this);
            bStop.addActionListener(this);
            bPause.addActionListener(this);
            bClear.addActionListener(this);
           
            list.addMouseListener(new MouseAdapter() {
                public void mouseClicked(MouseEvent evt) {
                    JList list = (JList)evt.getSource();
                    if (evt.getClickCount() == 2) {          // Double-click
                        // Get item index
                        int index = list.locationToIndex(evt.getPoint());
                       
                        try
                        {
                            java.lang.Process proc = Runtime.getRuntime().exec( "rundll32 url.dll,FileProtocolHandler \"https://www.experts-exchange.com/Q_" + v.get(index) + ".html\"" );
                            proc.waitFor();
                        } catch (Exception e){}
                    }
                }
            });
           
           
            setVisible(true);
        }
       
        public void actionPerformed( ActionEvent evt )
        {
            Object source = evt.getSource();
           
            if ( source == bGo )
            {
                String sQ = tThread.getText();
                int iQ;
                try{
                    iQ = Integer.parseInt(sQ);
                    t = new Process( model, iQ, v );
                    t.start();
                }catch (Exception e) { e.printStackTrace(); }
            } else if ( source == bStop )
            {
                if ( t.isAlive() )
                {
                    t.stop();
                }
            } else if ( source == bPause )
            {
                if ( t.isAlive() )
                {
                    if ( paused )
                    {
                        t.resume();
                        paused =! paused;
                        bPause.setText("Pause");
                    }
                    else
                    {
                        t.suspend();
                        paused =! paused;
                        bPause.setText("Resume");
                    }
                }
            } else if ( source == bClear )
            {
                if ( v.size() > 0 )
                {
                    v.clear();
                    model.clear();
                }
            }
        }
       
        public static void main ( String [] args )
        {
            new EEPostScanner();
        }
       
       
        private void buildConstraints( GridBagConstraints gbc, int gx, int gy, int gw, int gh, int wx, int wy )
        {
            gbc.gridx = gx;
            gbc.gridy = gy;
            gbc.gridwidth = gw;
            gbc.gridheight = gh;
            gbc.weightx = wx;
            gbc.weighty = wy;
        }
       
       
    class Process extends Thread implements ActionListener
    {
       
        private DefaultListModel model;
        private int              startingThread;
        private Vector           v;
        private JButton          cont;
        private boolean          add = false;
       
        public Process( DefaultListModel model, int startingThread, Vector v )
        {
            this.model = model;
            this.startingThread = startingThread;
            this.v = v;
        }
       
        public void run ()
        {
            for ( int i = startingThread; ; i++ )
            {
                String url      = "https://www.experts-exchange.com/Q_" + i + ".html";
                String someHtml = getText( url );
               
                if ( someHtml != "deleted" )
                {
                    if ( someHtml != null )
                    {
                        String titleStartTag = "<title>";
                        String titleEndTag = "</title>";
                       
                        int start = someHtml.indexOf(titleStartTag);
                        int end = someHtml.indexOf(titleEndTag);
                       
                        if (start != -1 && end !=-1)
                        {
                            String titleText = someHtml.substring(start + titleStartTag.length(), end);
                           
                            model.add( model.getSize(), new String(titleText) );
                            v.add(v.size(), i);
                        }
                    } else
                    {
                        JWindow win = new JWindow();
                        win.setBounds( ((int ) Toolkit.getDefaultToolkit().getScreenSize().getWidth()-300)/2,
                                       ((int ) Toolkit.getDefaultToolkit().getScreenSize().getHeight()-200)/2,
                                       300,
                                       200 );
                        JPanel pane = new JPanel();
                        win.setContentPane(pane);
                        JLabel mess = new JLabel("Question: #" + i + " cannot be found.");
                        mess.setForeground(Color.WHITE);
                        pane.add(mess);
                        cont = new JButton("Continue");
                        pane.add(cont);
                        cont.addActionListener(this);
                        pane.setBackground(Color.RED);
                        win.pack();
                        win.setVisible(true);
                        try{Thread.sleep(5000);} catch (Exception e){}
                       
                        if ( add )
                        {
                            i++;
                            add = false;
                        }
                       
                        win.setVisible(false);
                       
                        i--;
                    }
                }
               
                try{ Thread.sleep(300); } catch (Exception e){}
               
            }
           
        }
       
        public void actionPerformed( ActionEvent evt )
        {
            if ( evt.getSource() == cont )
                add = true;
        }
       
        public String getText(String uriStr) {
           
            URLConnection conn = null;
            InputStreamReader in;
            BufferedReader data;
            String line;
            try
            {
                URL page = new URL(uriStr);
               
                conn = page.openConnection();
                conn.connect();
               
                in = new InputStreamReader(conn.getInputStream());
                data = new BufferedReader(in);
               
                while ( (line = data.readLine()) != null )
                {
                    if ( line.indexOf("<title>") > 0 )
                        return line;
                }
            } catch (Exception e)
            {
                String sRet = e.getMessage();
                if ( sRet.indexOf("code: 403") > 0 )
                    return "deleted";
            }
            return null;
        }
       
    }
   
    }
   
:======8<--------------[  EEPostScanner.java ]----------------


It's just over 300 lines.

But despite "doing it for you", I'll probably use it also... So, only accept my answer, if you really think that it's good, and it helps some what..

Best of luck :)
>> IM

*.. As you can see, it's in Java ..*

LOL...you have an Idle and an Interactive mind helping you out...

I would prefer a VB solution so that I can add other features to it, however what compiler can you reccomend that I use to get that code in java to work ?

Thanks for the help !

Just to let everyone know I am travelling back to england in about 10 hours time so I wont be able to reply for a few days ! I hope this isnt a problem !

lol -- I'll sign myself as "Inter" in this thread, from now on... because your "~IM" and my ">> IM" are very, very similar  lol.

That's no problem, Shane. Have a good flight (although, by the time that I've submitted this message, you're probably already here lol).

To compile that, and to run it (and also to develop your own Java programs one day), all you need is the JDK (Java Development Kit). Just go to this link, and you can download it: http://192.18.97.239/ECom/EComTicketServlet/BEGIN69AA94FED7F9E76CA98AEA990ADD4CFC/-2147483648/790021419/1/573074/ +
572810/790021419/2ts+/westCoastFSEND/j2sdk-1.4.2_07-oth-JPR/j2sdk-1.4.2_07-oth-JPR:14/j2sdk-1_4_2_07-windows-i586-p-iftw.exe
(It would seem obvious to me that you're on Windows.. but for anyone who want the JDK, that are *not* on Windows, then go here: https://jsecom16k.sun.com/ECom/EComActionServlet/LegalPage:~:com.sun.sunit.sdlc.content.LegalWebPageInfo; + jsessionid=69AA94FED7F9E76CA98AEA990ADD4CFC;jsessionid=69AA94FED7F9E76CA98AEA990ADD4CFC ).

If I remember correctly, on dial-up, it will take about an hour (not so long ago, I had dial-up.. it ran at 46kb/s  LOL -- but finally, an ISP started to offer 1mp/s in my area...... I doubt that many people appreciate BB as much as me - lol - after a few years on 46k!).

I'll just quickly re-cap what my Java program does/can do:

It has a textfield, that you type in a number (the Thread ID) to start at. For example: 21364302, then you hit the "Go" button - and it will list the Title and Topic Area, of every thread that carries on from that number, 1 by 1.. It skips all "Deleted" threads, by detecting a Server Error code of 403 (which indicates a deleted thread).. It will also detect once a number/thread ID has been reached, which does not yet exist... In this case, it does a few things:
   It displays a small window on the screen, which says "Question #(the Thread ID) cannot be found." It then has a button called "Continue", if you click that, it will check the next Thread ID....
   If you don't click the "Continue" button, it will keep looping, checking to see if this thread has been created yet.. once it has, it will again, be added to the List, in the application.
With this List, you can double click any of the Qs (in the List), it will then open that Q in a browser..

There's also a "Stop" button, so that you can stop it from scanning from Qs, and change the number to start scanning from... btw: You can get it to scan, several ranges at once (it's multi-threaded), so this speeds up the scanning process - however, in this case, the "Stop" button becomes useless... because my intentions were only for you to use 1 additional thread to scan, at a time.. but if you use several, the "Stop" button will only stop one of the threads, then be rendered useless.
There's also a "Pause" button, which as you can probably guess, pauses the Scanning thread, incase you need to evaluate each thread as it's been detected, but need to quickly leave the room - you can pause it.. the pause button will then become a "Resume" button - which I think is obvious what it does. :)
And finally, you have a "Clear" button - which clears all of the detected threads so far, from the List, and from memory.. so if there's *loads* appearing, and they're annoying you, just click on the "Clear" button, and what has appeared so far, will dissapear :)  (I recommend that you do this at least once an hour... because everytime a thread is found, it adds a number to memory, to be recalled later.. So, the "Clear" button also flushes the memory being used..

That's about it..

Oh, before compiling it, change this line:

    private Vector      v           = new Vector(10, 5);

(nearish the top of the code -- it's a class instance).. Change it to:

    private Vector      v           = new Vector(30, 15);

That should make it a little more efficient, and suitable for this type of application.


Good luck! :)
>> IM

Oops... I mean:

Good luck! :)
_Inter

hehe ^_^

Hi Gecko,

... hve a good flight!


Best regards,
Raisor

Gecko, why reinvent the wheel? Quickpost already does this...

Thanks Raisor, I want to re make it because I am always having problems with EE Quickpost saying I am not logged in which is un true because I have logged in to the website in an internet explorer window behind the application.

It says about clearing temp files, making sure that your cookies are enabled and I went through all of that to no avail. So I wanted to make it with a web control instead of what they did so that I could attempt to do it so that I would not have any errors with regards to logging in, I also wanted to make it so that it would give the user the option of either having the URL they can click on to reply to it or a submit box like they have provided which posts straight to the questoin which you have selected from the list of questions it found that were new( Had just been posted ).

BTW - I am back now and I will await any further answers / suggestions , if any and will go from there :) Thanks for everyones help btw :)

If I do eventually make it I will post it in the support Area and see what the admins think to it before I release it as I do not want to go against any EE rules or anything like that ! I will take a look at that URL after I post this :)

> I've edited InteractiveMind's post above to get rid of the horizonal scrolling
Cheers Netminder :)

If the use of this application lags the servers at all, then if anyone insists on using it, I recommend that you change this line:

     try{ Thread.sleep(300); } catch (Exception e){}

to this:

     try{ Thread.sleep(1000); } catch (Exception e){}

Or, if you're feeling really kind, then increase the "1000" to something even higher :)   (That is the delay in Milliseconds, between each connection attempt).

>> IM

I am going to wait until I get a visual basic reply to see if I can do it in visual basic because I have no clue how I will link 2 programs together, ie the one in java to detect the new questions and then the browser in visual basic 6.0.

Are there any suggestions at all with regards to this in a visual basic 6.0 sense ie a visual basic 6.0 reply with pointers to examples, source code or at least pointers in the correct direction with regards to what API's or even what controls I would need to use or anything that would be of use to me making this ?

I can't figure it out exactly, but when I get time to sit down and actually do some hands on planning , coding , etc. I will post a Q related to this :) thanks for the help everyone !