We help IT Professionals succeed at work.

Wget on a URL with the vertical bar symbol (|)

etech0
etech0 asked
on
I'm trying to run wget (in Windows XP) on a website with the vertical bar symbol | in its URL, and it won't go through.
How can I get this to work?

Thanks.
Comment
Watch Question

Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
You might be able to encode it as '%7C' and get it to go thru.

Author

Commented:
It looks like it got further, but I get an error - "cannot write to " and the URL.
Do you know how to fix this?
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
I don't even know what you're talking about.  "cannot write to " and the URL sounds like you're trying to write to the page on the server.  That's not likely to happen.  Sure it's not trying to write to the local hard drive?

Author

Commented:
Please see attached screenshot of error message. error
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
You can't use '|' or '/' in a Windows file name or path.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx

Author

Commented:
Where did I do that?
Thanks.
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
The normal download is to a directory and file that has the same name as the URL.  '-r' means download everything I guess.  Since you are downloading a page with the query string that has illegal characters for a filename, that is where the problem is coming from.

Author

Commented:
So how do I fix that?
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
I'm not sure that you can.  Here is the manual: http://www.gnu.org/software/wget/manual/wget.html#Download-Options   It says that there the query strings are going to be part of the file name and there's no option to not do that.

Author

Commented:
There's got to be a way. The only thing I can think of is to use the -nd switch, to specify no directory. The only reason I'm not using that is because the -o switch (to set a destination) was giving me trouble. Can you tell me how to use the -o switch when downloading more than just one page?
Thanks.
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
On the manual page, it says that the '-o' switch puts everything in ONE file which isn't what you want.  And no, there doesn't have to be a way with wget to do this.  There are other methods with different programming languages where you can choose what you download and what file name you store it under.  But wget isn't that flexible.

Author

Commented:
Not the -0 switch, the -o switch. It sets the destination directory.
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
I don't think so, this is what I see on http://www.gnu.org/software/wget/manual/wget.html :

‘-o logfile’
‘--output-file=logfile’
    Log all messages to logfile. The messages are normally reported to standard error.


‘-O file’
‘--output-document=file’
    The documents will not be written to the appropriate files, but all will be concatenated together and written to file.

Author

Commented:
What about the -p switch, would that do it?

Author

Commented:
Or maybe using > ?
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
All options but the '-O' use the complete URL as the file name and that's where your problem is.  The URL you are using has characters that are not legal in Windows file names.  '-r' doesn't do you any good because that for 'recursive' downloads which means downloading all the links in the original URL.

Author

Commented:
So do you have any possible solution for me? I want to download all the mp3 files from that webpage.
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
There are no direct links to MP3s on that page.  The links all go to a download page or a player, probably so you can't download them directly.

Author

Commented:
I see. Is there any way to get to the mp3s?
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
I can't read the Hebrew on the page but it looks like you can click on the download links.  You might have to be logged in or signed up to do that.  By the way, even when I put 'English' in the query string, the page did not show in English.

Author

Commented:
There is a button "English" that will switch over the site language.
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
Ok, if I go to http://www.kolhalashon.com/New/Shiurim.aspx?Lang=English&English=True there is a download button for each item.  But it is thru a download page and not a direct download link.  It did let me download an MP3.

Author

Commented:
Yes. Is there some sort of way to do this with a batch file to get each one?
Fixer of Problems
Most Valuable Expert 2014
Commented:
There might be.  Try using http://www.kolhalashon.com/New/Shiurim.aspx?Lang=English&English=True as your link instead of your original link and see what you get.

Are you aware that they have over 2000 MP3s averaging 10MB each?  That's at least 20GB of disk space and a lot of downloading time.

Before you do that, check out their MP3 Download Program in the upper right corner of the page and see if that is something that you want.

Author

Commented:
Thanks, will look at it.
Yes, I know. That's why I only wanted from that specific page - of a certain artist.
Thanks.

Author

Commented:
Looks like I'm stuck with doing this manually.
Thanks for trying, though!
Dave BaldwinFixer of Problems
Most Valuable Expert 2014

Commented:
You're welcome, sorry it didn't work better for you.