Solved

wget link autocompletion

Posted on 2010-11-19
19
898 Views
Last Modified: 2012-05-10
Hello,

is there a way that wget complete links (or give a few often alternatives) when I press tab? Similarly as
bash shell, but it should complete links.
Could you show me how to achieve this?

thanks
0
Comment
Question by:xRalf
  • 7
  • 4
  • 2
  • +3
19 Comments
 
LVL 10

Expert Comment

by:lucius_the
ID: 34174778
I'm afiraid not so. Bash and other shell interpreters can complete thecommand because they know what possible commands are there on a system.

wget expects a link as a parameter. It's not aware of it's location or it's contents until you completely tell him the URL.
0
 
LVL 11

Expert Comment

by:level9wizard
ID: 34174851
You would need to write your own bash extension that would dynamically and asynchronously fetch a sitemap with all the urls for that domain (i.e. http://example.com/sitemap.xml), parse it, and then load that into an array or existing class that supports auto-completion. It wouldn't be easy for me, maybe for others - Good luck!
0
 
LVL 6

Author Comment

by:xRalf
ID: 34175043
Thank you for answer,

maybe somebody has written the bash extension already but we don't know it's name or how to find it.
0
 
LVL 10

Expert Comment

by:lucius_the
ID: 34175160
I believe it's quite unlikely. Although you have similar things on wikipedia and goole, this is only because there is one backend that provides such data.

As wget can connect to ANY http or ftp address, so, expect some theoretical possibility, there is no practical way to achieve this. Another problem is you need the information back from the site instantly or near-instantly so... no way. Not yet at least.
0
 
LVL 6

Author Comment

by:xRalf
ID: 34222150
This is difficult question. I increased points and made the question easier as follows.

It's sufficient that the bash is able to autocomplete address that are already in firefox history.
0
 
LVL 11

Expert Comment

by:level9wizard
ID: 34283381
Just to confirm, you are wanting to input in a Linux command line after you type 'wget ' and then see URL's auto-complete (matching against FireFox's history) by starting to type and press tab?

If I understand correctly, this is likely going to require some c. You may want to consider re-posting this question in a c and linux zones; the ones you are using presently are not going to address that low level functionality.
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 34290863
The ncftp ftp client can do tab completion on remote file names. Also it bookmarks every site as you visit it for the first time, so you only have to input the address once ever. But it is an ftp client, not http (as wget is)
0
 
LVL 22

Expert Comment

by:NovaDenizen
ID: 34292773
In order for wget to do command line tab autocompletion it would have to connect to the server (assuming that the server is completely entered) and somehow via http get a list of all links in the appropriate directory.  Unfortunately, there is no standard way to do a directory listing via http.  In fact in some cases it is actually impossible to return any kind of meaningful directory listing because the local URLs might not have a manageable correspondence with files on the disk, and each of an infinite set of filenames could point to unique automatically generated content.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 6

Author Comment

by:xRalf
ID: 34293535
It's sufficient as I described in comment number 34222150
0
 
LVL 16

Accepted Solution

by:
The--Captain earned 500 total points
ID: 34296963
Why don't you just write a script that uses the URLs in your history file to create a local dummy directory structure that bash will parse (and autocomplete) just like any other part of the filesystem?  

Turn something like this:

http://www.wired.com/reviews/2010/11/review_bg_mobilephones/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29

into a local filesystem entry like this:

mkdir -p 'http://www.wired.com/reviews/2010/11/review_bg_mobilephones'

touch 'http://www.wired.com/reviews/2010/11/review_bg_mobilephones/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29'

then you can use wget to autocomplete by typing two characters (aside from TAB):

wget h[TAB]/[TAB]

(sorry, but I can't yet figure out how to get the second slash to autocomplete - you'll always have to type one of them if you use my idea).

Ugly, and by no means perfect, but it's a lot easier than coding ;-)  Also, do not forget the single quotes!

Cheers,
-Jon


0
 
LVL 16

Expert Comment

by:The--Captain
ID: 34297047
sed -e 's/^\(http.*\/\)\(.*\)/mkdir -p '\''\1'\''; touch '\''\1\2'\''/'

turns that URL above into the two commands above.

Cheers,
-Jon

P.S.  There are no double quotes in the above - those are a bunch of single quotes.
0
 
LVL 6

Author Comment

by:xRalf
ID: 34326946
Thank you The--Captain
for very original and great solution.
Could you help with two things yet?
I can't find the location of firefox history file.
Here is described http://www.linuxforums.org/forum/newbie/127046-storing-history-file-mozila-firefox-fedora-7-a.html that it should be in history.dat which is not on my system (Xubuntu)
Where should be created the dummy directory structure and files? In the root directory?
Is it possible somewhere it doesn't get in the way?
0
 
LVL 16

Expert Comment

by:The--Captain
ID: 34331520
"Where should be created the dummy directory structure and files? In the root directory?"

Just create a new directory somewhere and just put all the dummy filesystem stuff in there.

Run a locate or find for bookmarks.html to find the file.  If you need help parsing it, you might want to open another question specifically dedicated to that.

Cheers,
-Jon
0
 
LVL 6

Author Comment

by:xRalf
ID: 34337174
Thank you "locate" found it. I will try to parse and if needed I will open another question.

I've got an idea to download with curl the whole page and extract more links from that, so it will be able to do more completions, but I should probably restrict it for domain, because it could be infinite...
0
 
LVL 6

Author Comment

by:xRalf
ID: 34337318
But those are bookmarks.html. I have problem with finding history file.
I tried
locate history | grep irefox
I returned only formhistory.sqlite
0
 
LVL 16

Expert Comment

by:The--Captain
ID: 34341841
You're right - I forgot that you wanted history, not bookmarks.  I think the file you're looking for is called places.sqlite , but I don't know off the top of my head the layout - you'd probably have to search online (or ask a new question right here at EE) to find a way to use sqlite off the command line to dump the URLs.

Cheers,
-Jon
0
 
LVL 6

Author Comment

by:xRalf
ID: 34344614
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Suggested Solutions

It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Container Orchestration platforms empower organizations to scale their apps at an exceptional rate. This is the reason numerous innovation-driven companies are moving apps to an appropriated datacenter wide platform that empowers them to scale at a …
The viewer will learn additional member functions of the vector class. Specifically, the capacity and swap member functions will be introduced.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now