Link to home
Start Free TrialLog in
Avatar of NoxBestia

asked on

Perform the funcitons of a Perl script with a console application


I am working on a special data synchronization program that needs to be able to as an unattended console application, executed from the servers scheduler.  The program is actually a rewrite/modification of one I wrote previously, but I have to connect to a different system a much different way, and the only sample code that the company I need to synchronize with can give me is a Pearl script.

For what I am doing, I cannot use a script.  I definitely need to be able to have everything contained within a single console application, not having my console application pause its operation and run an independent script.  

For the part that replaces the script, I was planning on using the webclient class.  I do not want to use the webbrowser class as I have yet to be able to get it to work in an automated console application that runs without anyone logged into the server.  I cant seem to understand how to make my webclient do what the sample Perl script does.

I have attached the sample Perl script that the company gave me.  The URL referenced and the access code have been modified for security reasons, but otherwise this is the complete Perl script I am trying to replicate as a console application.

From what little I know about Perl and have been able to research, the script is acting as if it was a submitted multipart web form that is sending the access code (appkey => $appkey), a flag code (submit => 1), and then the local file that I want to upload (users_file => [ $filename ]).  My incomplete knowledge of Perl doesnt fully tell me what all is being sent in the users_file => [ $filename ] section.  From what I _think_ I understand the [ $filename ] part actually refers to multiple values as well as the data of the file itself.

Unfortunately the company that I am trying to synchronize data with is unable to really help with anything but giving me this Perl script.  

Any help would be GREATLY appreciated.  


use strict;
use HTTP::Request::Common;
use LWP::UserAgent;
use Getopt::Long qw/ :config auto_help /;
my $url_base = "";
my $appkey = "aLongAlphanumericSecurityCodeHere";
my $filename = "";
    'url=s'      => \$url_base,
    'appkey=s'   => \$appkey,
    'filename=s' => \$filename,
# checking argument variables
die "Error: No url to send request to.\n" unless ( $url_base );
die "Error: No appkey for authentication.\n" unless ( $appkey );
die "Error: No file specified to upload.\n" unless ( $filename );
die "Error: Specified file does not exist.\n" unless ( -e $filename );
# creating request to send
my $url = $url_base;
my $ua = LWP::UserAgent->new();
my $req = POST $url, Content_Type => 'form-data',
                     Content => [
                                 appkey => $appkey,
                                 submit => 1,
                                 users_file => [ $filename ]
# posting file to Dataync server
my $response = $ua->request($req);
# check for success of file upload
if ($response->is_success()) {
    print "OK: ", $response->content;
} else {
    print $response->as_string;

Open in new window

Avatar of Bob Learned
Bob Learned
Flag of United States of America image

So, you want to write a Console application that takes a file name as a command line argument, and upload it to a server?
Avatar of NoxBestia


Sort of.  

The console application doesnt actually take any arguments.  The file that will be uploaded will be dynamically generated each time the console application runs, based on the results of a query the application runs against out database.  The way that part of it works, the console application doesnt _have_ to save the data to a file and then upload it if there is a way to have it simply send the data directly to the web page that is getting it.

Since the web page that I am uploading this data to is not under my control and will not be altered for me in any way, I figured the easiest way for me to make it work would be to have my application create a file from that data and then upload said file to the web page.  If this can be done without that step, then I am fine with that.

If it helps, the file that is expected by the web page I am uploading to is a standard CSV format.
My first thought was a simple webclient.uploadfile but the web page that I have to post the file to also requires additional parameters, as defined in the Perl script by the variables appkey, submit, and users_file.  

I was told that the line "my $response = $ua->request($req);" is what actually sends the data to the target server, but the person tyring to explain the script doesn't actually know what all is sent by at that point and I do not know enough Perl to be able to figure it out.
WebClient.UploadFile is a simple wrapper for an HttpWebRequest.  If you need things above and beyond simple, then you would need to create an HttpWebRequest yourself.
I had originally though webclient.upload but my initial tests had failed.  However, I could very easily have been doing it wrong.

Will webclient.uploadfile allow me to add additional name/value pairs along with the file that is being uploaded?  If so, how is this accomplished?  Are they considered a header item, or should they simply be part of the URL as if I was using GET instead of POST?

Also, is there are better reference available for the webcleint class (and the .NET classes in general) than the default MSDN reference?  I find it to be a little less than clear in many areas and many of the books I have available to me are far from inclusive in what classes and methods they deal with.
Try reading about the System.Net.HttpWebRequest class:

HttpWebRequest/Response in a nutshell - Part 1
Learned One;
The tutorial on that page, as well as an additional tutorial I found on the same site was very helpful indeed.   Thank you so very much.
Based on that information, I have written code to attempt to perform the same function that the Perl script does; however, it does not quite work.  The target system denies me access when I run my visual basic code, but it works with the Perl script.  To rule out the most basic of possible errors, I did a copy/paste of the access key from the working Perl script into the visual basic program.
I created a small PHP test page to try to figure out any differences that I am getting between the two.  The primary difference I noted was a difference of 6 bytes in the size.  The problem is that I dont know what these six bytes are.
How would you suggest that I troubleshoot this?  I had thought that some sort of web page on my end that shows everything that both the script and the visual basic page try to upload, including the encoding that represent the file that I am attempting to upload; however, I cannot seem to find out how to do that.  My R&D server has IIS 6 on it as well as PHP now so anything that uses either of these would be great.
I have included the code I wrote, modified to remove any confidential information.  Do you see anything obviously missing or any obvious errors in my code?

Imports System
Imports System.Web
Imports System.Collections.Specialized
Imports System.Text
Imports System.Net
Imports System.IO
Module Module1
    Sub Main()
        Console.WriteLine("Starting progrem...")
        Dim cookies As New CookieContainer()
        Dim querystring As New NameValueCollection()
        Dim uploadfile As String = "v:\555-5555.csv"
        Dim url As String = ""
        Dim outdata As String
        Console.WriteLine("Debug: the url sent is: " & url)
        'querystring("appkey") = appkey
        'querystring("submit") = 1
        Console.WriteLine("executing function...")
        outdata = UploadFileEx(uploadfile, url, "users_file", "text/plain", querystring, cookies)
        Console.WriteLine("OKay, here it is: ")
        Console.WriteLine("Enter to continue...")
    End Sub
    Public Function UploadFileEx(ByVal uploadfilename As String, ByVal url As String, ByVal fileFormName As String, ByVal contenttype As String, ByVal querystring As System.Collections.Specialized.NameValueCollection, ByVal cookies As CookieContainer) As String
        Console.WriteLine("Inside function....")
        Dim appkey As String = "security code here"
        If (fileFormName Is Nothing) OrElse (fileFormName.Length = 0) Then
            fileFormName = "file"
        End If
        If (contenttype Is Nothing) OrElse (contenttype.Length = 0) Then
            contenttype = "application/octet-stream"
        End If
        Dim postdata As String
        postdata = "?appkey=" & appkey
        'If Not (querystring Is Nothing) Then
        '    For Each key As String In querystring.Keys
        '        postdata += key + "=" + querystring.Get(key) + "&"
        '    Next
        'End If
        Dim uri As Uri = New Uri(url + postdata)
        'Dim boundary As String = "----------" + DateTime.Now.Ticks.ToString("x")
        Dim boundary As String = "xYzZY"
        Dim webrequest As HttpWebRequest = CType(webrequest.Create(uri), HttpWebRequest)
        webrequest.CookieContainer = cookies
        webrequest.ContentType = "multipart/form-data; boundary=" + boundary
        webrequest.Method = "POST"
        webrequest.KeepAlive = False
        webrequest.Connection = "TE"
        webrequest.Expect = Nothing
        webrequest.UserAgent = "visual basic.NET by Johnny-dog"
        Dim sb As StringBuilder = New StringBuilder
        sb.Append("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")
        sb.Append("Content-Disposition: form-data; name=""submit""")
        sb.Append("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")
        sb.Append("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")
        sb.Append("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")
        sb.Append("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")
        sb.Append("Content-Disposition: form-data; name=""")
        sb.Append("""; filename=""")
        sb.Append("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")
        sb.Append("Content-Type: ")
        sb.Append("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")
        sb.Append("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")
        Dim postHeader As String = sb.ToString
        'Dim postHeaderBytes As Byte() = Encoding.GetEncoding("ISO-8859-1").GetBytes(postHeader)
        Dim postHeaderBytes As Byte() = Encoding.ASCII.GetBytes(postHeader)
        Dim boundaryBytes As Byte() = Encoding.ASCII.GetBytes("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "--" + boundary + "" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")
        Dim fileStream As FileStream = New FileStream(uploadfilename, FileMode.Open, FileAccess.Read)
        Dim length As Long = postHeaderBytes.Length + fileStream.Length + boundaryBytes.Length
        webrequest.ContentLength = length
        Dim requestStream As Stream = webrequest.GetRequestStream
        requestStream.Write(postHeaderBytes, 0, postHeaderBytes.Length)
        Dim sendBuffer(Math.Min(4096, fileStream.Length)) As Byte
        Dim bytesRead As Integer = 0
        'Dim r As New BinaryReader(fileStream)
            bytesRead = fileStream.Read(sendBuffer, 0, sendBuffer.Length)
            If bytesRead = 0 Then Exit Do
            requestStream.Write(sendBuffer, 0, bytesRead)
        requestStream.Write(boundaryBytes, 0, boundaryBytes.Length)
        Dim responce As WebResponse = webrequest.GetResponse
        Dim s As Stream = responce.GetResponseStream
        Dim sr As StreamReader = New StreamReader(s)
        Return sr.ReadToEnd
    End Function
End Module

Open in new window

Avatar of Bob Learned
Bob Learned
Flag of United States of America image

Link to home
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Learned One;

That is one incredible program!  And it showed me the errors of my ways.  

I have included the corrected line of code, which now adds the two extra dashes at the end of the multipart boundary, which the original code did not include.  The other differences in byte count were just in the formatting of the name and not relevant to whether the system would accept my upload or not.

So, with my problem solved and the system working, what should I accept as the answer, with so many of your replies necessary for me to work though the problem and get it working?

'Note the 2 extra dashes before the final crlf of this string!
Dim boundaryBytes As Byte() = Encoding.ASCII.GetBytes("" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "--" + boundary + "--" & Microsoft.VisualBasic.Chr(13) & "" & Microsoft.VisualBasic.Chr(10) & "")

Open in new window

Select the last comment, it was the most useful in this case.
In the end, I solved my problem myself, but I could not have done it without the guidence provided by TheLernedOne.  I am quite greatful for the help!!!