Section 1 — Introduction
Are you a frequent author here at Experts Exchange? Have you published a large number of Articles and/or Video Micro Tutorials?
If so, you may have the same wish that I do, namely, the ability to see how many Views, Endorsements, and Points each article/video has. (After initial publication of this article, there were requests for including the number of Comments, the Date Published, the Date Last Modified, and the URL. I added them, but left the title of this article the same, as changing it to include the new fields would result in a cumbersome title.) Manually visiting all articles and videos to record those figures would be time-consuming and error-prone for any author with more than a handful of publications — and certainly any author with dozens of them.
Experts Exchange sends an email on the first of every month with some of those figures, but the Views and Endorsements are only for the previous month (not total) and they are only for articles (not videos). Also, since the information resides in the body of an unstructured email, it is not easy to move it into something more manageable/useful, such as an Excel spreadsheet.
To address this issue, I decided to make it the project for my 100th EE publication. The idea is to develop an automated solution to visit every article/video and, for each one, place the Title, number of Views, number of Endorsements, number of Points, number of Comments, and Date Published in a structured file that can be easily loaded into Excel (or simply viewed as a text file).
I'm calling the program ArticlesVideosEE. I wrote it in AutoHotkey, my language of choice these days for Windows programs/scripts. If you are not familiar with AutoHotkey, my EE article will get you going on it:
AutoHotkey - Getting Started
AutoHotkey (and, therefore, ArticlesVideosEE) runs only on Windows. I tested the program on W7/64-bit and W10/64-bit, but it should run fine on XP, Vista, and W8/8.1, both 32-bit and 64-bit. If anyone finds a Windows platform on which it fails, please post a comment here.
Section 2 — Objective
The objective is to create a Results file that has tab-delimited text which loads nicely into Excel. The Results file has a header line with text that identifies each field (and an item with the date/time stamp of the run). The header looks like this:
Title of Publication (Run Date/Time: 2019-05-31_05.35.12) Views Endorsements Points Comments Date Published
After the header line, it has a line for each article/video with its Title, Views, Endorsements, Points, Comments, and Date Published. Here's an example for a video:
How to use the Windows Task Scheduler - An Introduction - Online Training Video 1,547 18 23,347 7 2015-12-21
ArticlesVideosEE prompts you for the output folder (where to store the Results file) via a Browse For Folder dialog:
The name of the Results file that it creates in that folder is:
The time stamp in that file name is when you ran the program.
ArticlesVideosEE also prompts you for the text file that contains the list of URLs:
Both prompts have error-checking and warn you of problems. For example:
Section 3 — How It Works
ArticlesVideosEE operates on a plain text file with the URL of each article and video on a separate line. I already had a list of all my articles and videos in a text file, as I suspect most EE authors do. If you do not, it's easy to create such a list, and you'll need to do it only once (with a one-line addition, of course, each time you publish a new article/video).
A line in the URL list with a semi-colon in the first column is a comment and is ignored. Also, blank lines are ignored. Here's an example of a small list:
; Publications list for ArticlesVideosEE demo ; Articles https://www.experts-exchange.com/articles/12380/Signature-Image-with-Transparent-Background.html https://www.experts-exchange.com/articles/18119/ https://www.experts-exchange.com/articles/18880/PaperPort-14-in-Windows-10-A-First-Look.html https://www.experts-exchange.com/articles/29533/ https://www.experts-exchange.com/articles/32441/How-to-fix-PaperPort-shows-the-splash-screen-then-immediately-closes-cannot-run-PaperPort.html https://www.experts-exchange.com/articles/33247/ ; Videos https://www.experts-exchange.com/videos/213/ https://www.experts-exchange.com/videos/1598/How-to-use-the-Windows-Task-Scheduler-An-Introduction.html https://www.experts-exchange.com/videos/1618/ https://www.experts-exchange.com/videos/2020/Xpdf-PDFtoPNG-Command-Line-Utility-to-Convert-a-Multi-page-PDF-File-into-Separate-PNG-Files.html https://www.experts-exchange.com/videos/6338/ https://www.experts-exchange.com/videos/9740/What-to-do-when-PaperPort-crashes-hangs-or-fails-to-start-delete-metadata-with-CheckPPFolders.html
Any valid URL works. As you can see above, I alternated short-form and long-form URLs for demonstration purposes.
ArticlesVideosEE reads the URL list and operates on each link, one at a time. It downloads the page into a variable for analysis. There is error-checking in the program at many places, such as this one, where it makes an entry in the Results file if the download fails, but it continues processing the list. Other errors are Fatal Errors, meaning the program terminates immediately, with a dialog box like this:
The program looks for the Title in the downloaded page. If it doesn't find one (should never happen), it puts an error message in the Results file. After finding the Title, it looks for Views, Endorsements, Points, Comments, and Date Published, setting the value for any of those that it doesn't find to 0 (or "No publish date" in that case). Here's an example line that it creates in the Results file for an article:
AutoHotkey - Getting Started 5,896 24 107,296 19 2015-04-29
Section 4 — Technical Details
ArticlesVideosEE finds the information in the downloaded page by searching for strings that identify the elements. For example, the article/video Title is between "<title>" and "</title>". Similarly, the number of Views is between "<span>" and " Views</span>". These identifying strings are in variables with Prefix and Suffix in the name of each (and note Plural and Singular versions for Views, Endorsements, and Comments). You'll need to update these variables in the program's source code (attached at the bottom of this article) if Experts Exchange modifies the underlying format of the article or video page. If you start getting incorrect results, that's the first issue to check, and note that the DebugRun=1 setting (discussed in Section 7) will help you to figure that out by not deleting the temporary file with the downloaded page. Also, I am running ArticlesVideosEE daily on my full list of publications (via the Windows Task Scheduler — see the 9-Jun-2019 enhancement in Section 7) and will post here as soon as I detect any change in the page layout that would cause the program to fail.
Section 5 — User Interface in System Tray
ArticlesVideosEE puts an icon in the system tray (notification area). The icon comes from shell32.dll (built into all versions of Windows) and is a piece of paper with a pencil on it (my attempt at signifying an article — feel free to change it). Hovering over the icon displays a tooltip showing the program name (ArticlesVideosEE) and Version (date/time stamp of source code file) on the first line and the URL currently being processed on the second line. For example:
A right-click on the system tray icon produces this context menu:
Clicking About shows this:
About is the default menu pick, so a double-click on the icon does that. Also, About acts as a pause for the program, as it does not continue running until you click the OK button.
Clicking Exit on the context menu immediately terminates the program, even if it has not yet processed the entire URL list, although it gives you a chance to continue running in case the exit was unintentional:
Note that the default button in that dialog is No so that an accidental Enter key does not exit. The program is paused during this dialog — it continues to run if you click No.
When it is done, ArticlesVideosEE displays a dialog box showing the number of publications processed in the Title bar and the location of the Results file in the body:
As you can see, that's from an actual run with my full list of publications — before this one. :) It's reasonably fast, with that run for the 99 publications taking 54 seconds (of course, that depends on Internet speed and computer performance).
Section 6 — Demonstration
Let's take a look at the program in action using the small list shown in Section 3. I published a five-minute EE video Micro Tutorial with a demonstration:
That's the first publication of my next 100. :)
Section 7 — Customization
You may want to customize ArticlesVideosEE to suit your needs/tastes. Inclusion of the full source code in this article allows you to do that. For example, I run a customized version with these changes:
• A hard-coded output folder, since I'm fine with the Results file always going into the same location — and it has the side benefit of avoiding the Browse For Folder prompt. Here's the line of code for that:
OutputFolder:="X:\Articles Videos Experts Exchange\" ; works with or without ending backslash
• A hard-coded file name with the URL list, since I'm fine with always using the same file — and, again, it has the benefit of avoiding the Browse For File prompt. Here's the line of code for that:
ArticlesVideosList:="X:\Articles Videos Experts Exchange\ArticlesVideosList.txt"
• It always opens the Results file in Excel before exiting. Here's the code snippet for that:
OpenResultsWith:="c:\Program Files (x86)\Microsoft Office\root\Office16\EXCEL.EXE" ; location of Excel executable - varies by system Run,"%OpenResultsWith%" "%ArticlesVideosResults%"
You may prefer to have it always open the Results file in Notepad. Here's the code snippet for that:
OpenResultsWith:="notepad.exe" Run,"%OpenResultsWith%" "%ArticlesVideosResults%"
Here's an example of what the Results file looks like (from the URL list shown in Section 3) after I opened it in Excel, sorted by Views descending, and created a TOTAL row at the bottom (and fancied it up a bit with AutoFit Column Width and bolding on the first and last rows):
Click the image above and expand the width of your browser for a better view of it.
Enhancement on 9-Jun-2019
I showed above the three customizations that I have in my version of the program. I decided to generalize that into an enhanced version of the program that supports those customizations without having to modify the source code. I did this by adding support for a configuration file that allows those parameters to be set.
The configuration file is called config.ini and it resides in the same folder as the program. It is a Windows standard format ".ini" file. If you're not familiar with that, the ".ini" format is a plain text file with one or more Sections. Each Section has a unique name and is enclosed in a set of square brackets. In this version of ArticlesVideosEE, there is only one Section — [Settings].
Each Section in a standard configuration file has lines with a "Key-Value pair" on it. The Key and its Value are separated from each other with an equal sign. A line with a semi-colon in the first column is a comment and is ignored.
Sidebar: For those readers interested, the AutoHotkey language has commands that directly handle a standard format ".ini" file: IniRead, IniWrite, IniDelete.
For example, here's what the configuration file looks like to implement what I had previously done by modifying the program's source code (with some commented-out lines for illustrative purposes):
[Settings] OutputFolder=X:\Articles Videos Experts Exchange\ ArticlesVideosList=X:\Articles Videos Experts Exchange\ArticlesVideosList.txt OpenResultsWith=c:\Program Files (x86)\Microsoft Office\root\Office16\EXCEL.EXE ;OpenResultsWith=c:\Windows\system32\notepad.exe ;OpenResultsWith=No ;OpenResultsWith=
The folder name in the Value does not require an ending backslash (it works with or without it). The Values in the Key-Value pairs do not require quotes around them, even if there are spaces in the Values (they work with or without the quotes). For example, both of these work:
OutputFolder="X:\Articles Videos Experts Exchange\"
OutputFolder=X:\Articles Videos Experts Exchange
If OutputFolder is not in the file (or is null, i.e., nothing after the equal sign), ArticlesVideosEE prompts for the folder with the Browse For Folder dialog. Likewise, if ArticlesVideosList is not in the file (or is null), it prompts for the file with the Browse For File dialog.
If OpenResultsWith is not in the file (or is null), ArticlesVideosEE displays the closing dialog box shown previously. If OpenResultsWith is set to No or None (case insensitive), the closing dialog box is not displayed (good for unattended operation — see below).
There is one other Key in the configuration file — DebugRun. If its Value is 0 (zero), it means that this is not a debugging run (the default), while a non-zero Value signifies that it is. For example:
In the future, there may be different debugging levels. Currently, the only non-zero Value is 1 (one), and its only purpose is not to delete the temporary file that contains the downloaded page (the ~TempPage.txt file). I used this new feature to help find the bug that was causing the previous version of the program not to detect the case of one Endorsement (because that word is singular, not plural).
By specifying all the variables in the configuration file, it is now possible to run ArticlesVideosEE periodically via the Windows Task Scheduler completely unattended, without any screen output or user intervention (except in the case of a Fatal Error). However, the tray icon (and tip) are still present in case you want to pause or exit the program before it finishes.
When creating the task in the Task Scheduler, simply set the Trigger to whatever frequency you want, such as Weekly, and set the Action to the location of the program file, such as:
X:\Articles Videos Experts Exchange\ArticlesVideosEE.ahk
If you're not familiar with the Task Scheduler, my EE article will get you started on it:
Since this new version of ArticlesVideosEE is a superset of the first one, I replaced the attached source code file with the new one (when the config.ini file is not present, the enhanced version performs the same as the previous one).
End 9-Jun-2019 Enhancement
Enhancement on 10-Jun-2019
Added a column in the Results file for the number of Comments and another column for the Date Published. Also, changed the Title column header to "Title of Publication" and moved the "Date/Time of this run" from the far right of the first line to a parenthetical inside the updated Title header (visually better). Updated the Excel screenshot above showing these changes. Also, updated the text in the article to include the new columns.
Once again, I replaced the attached source code file with the new one.
End 10-Jun-2019 Enhancement
Enhancement on 21-Jun-2019
Added a column in the Results file with the URL as it appears in the file with the list of articles and videos. Also, changed the name of the FullFileName variable to URL, which is a better name for it. Updated the Excel screenshot above showing these changes. Also, updated the text in the article to include the new column. As before, I replaced the attached source code file with the new one.
End 21-Jun-2019 Enhancement
Enhancement on 25-Jun-2019
Added a column in the Results file with the Date Last Modified. Updated the Excel screenshot above showing these changes. Also, updated the text in the article to include the new column. As before, I replaced the attached source code file with the new one.
End 25-Jun-2019 Enhancement
Section 8 — Command Line Interface (CLI)
[This is an update posted on 10-Jun-2019 at 6:30PM US Central Time. I replaced the attached source code file with a new one that contains this enhancement.]
ArticlesVideosEE provides a Command Line Interface (CLI). You may run it in a command (aka DOS) prompt, a batch file (.bat), the Task Scheduler, programs/scripts, etc. — anywhere that a command line call can be made.
ArticlesVideosEE looks for a parameter on the command line. If there isn't one, the program behaves exactly as discussed previously in this article. But if there is one, it is the full path to a file in the standard ArticlesVideosEE "config.ini" format (explained in Section 7). However, the file name can be whatever you want. Here are some examples (as in all command line calls, quotes are needed around file names with spaces):
c:\MyProgs\ArticlesVideosEE.ahk "x:\Articles Videos EE\Andrew Leniart.ini"
c:\MyProgs\ArticlesVideosEE.ahk "x:\Articles Videos EE\Tom Zucker-Scharff.ini"
ArticlesVideosEE includes the parameter's file name (without the path and extension) in the Results file name. For example:
Any number of ArticlesVideosEE instances may run simultaneously.
I developed this enhancement at the request of EE's Senior Editor so that Page Editors may easily obtain the statistics for the articles and videos of any number of authors on a periodic basis.
Section 9 — Conclusion
My hope is that the descriptive variable names along with the extensive comments in the program provide enough documentation for folks who want to modify it, but if you have any questions on that front, please post them here and I'll try to assist.
I've received feedback indicating that the video demo is extremely helpful and encouraging me to mention it again at the end of the article, so here goes:
How to download number of Views, Endorsements, Points for Experts Exchange Articles and Videos--Demo
The feedback went on to say that the video made it much easier to understand how to configure and use the program.
If you find this article to be helpful, please click the thumbs-up icon below. That way, I won't see zero Endorsements the next time I run the program. :)
Thanks very much! Regards, Joe