[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

curl download speed bytes megabytes

Posted on 2014-08-18
34
Medium Priority
?
1,772 Views
Last Modified: 2014-08-19
Using curl in a bash script to download a file how can I do the following;

-extract the average download speed in Bytes or Megabytes into two variables such as B or M. Example B 543 or M 23 ( $BYTES and $SPEED)
-extract the dns lookup time into a variable
-extract the 'time total' of the download into a variable

If I send the curl output to a 'report' file, then using
awk < report  'FNR == 3 {print $7, $9 }'
gives me the proper output.

However, when including it as part of a full command, nothing;
curl -o /dev/null -LO http://www.somesite..com/thebigfile | awk 'FNR == 3 {print $7, $9 }'
0
Comment
Question by:projects
  • 17
  • 17
34 Comments
 
LVL 23

Expert Comment

by:nemws1
ID: 40270430
You want to craft something with the '-w' command line option:

CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://www.somesite.com/thebigfile`
SPEED=`echo $CSTATS | awk '{print $1}'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`

Open in new window

0
 

Author Comment

by:projects
ID: 40270464
I must not be understanding something.

When using curl -h, I don't see any options which I keep reading about such as 'time_name' and 'time_total' for example.

I took your code and put it into a bash script then printed the following.

SPEED=`echo $CSTATS | awk '{print $1}'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`

echo "$SPEED, $DNSTIME, $TOTALTIME"

But the output is;

# ./go
2953493.000, 0.003, 3.550

However, I'm trying to output speed in KB or MB.
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40270494
Yup.  That's in Bytes/second, which you said was acceptable in your initial post.

To convert to KB and/or MB:
KBSPEED=`echo "4k $SPEED 1024.0 /p" | dc`
MBSPEED=`echo "4k $KBSPEED 1024.0 /p" | dc`

Open in new window

0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
LVL 23

Expert Comment

by:nemws1
ID: 40270508
And 'curl -h' doesn't tell you all the options to '-w' - it just tells you '-w' is there.  You have to read the manual page to see all the options.  Here's a link to the '--write-out' format spec:

http://curl.haxx.se/docs/manpage.html#-w
0
 

Author Comment

by:projects
ID: 40270521
Can you roll it all into one so I can see it completely. I like when folks post the full code, then I can mess with it and add to it, etc.

As to my original question what I am asking for is;

 -extract the average download speed in Bytes or Megabytes into two variables such as B or M. Example B 543 or M 23 ( $BYTES and $SPEED)

No one non technical would understand 2953493.000 as a result :)

What I need for speed output is more like mentioned above but I see I made an error asking for B.
That should be something more like;

978 KB/s or if the speed goes into Mb/s, 12Mb/s as output.
Kilo BYTES                                        Mega BITS
0
 

Author Comment

by:projects
ID: 40270533
I didn't know about the additional information on curl. I did search but since I didn't see the options when I typed -h for my own version, I assumed mine didn't have those options. Thanks for that heads up, very useful as I will learn more about curl now.
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40270536
Like this?
CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://www.somesite.com/thebigfile`
SPEED=`echo $CSTATS | awk '{print $1}'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
KBSPEED=`echo "4k $SPEED 1024.0 /p" | dc`
MBSPEED=`echo "4k $KBSPEED 1024.0 /p" | dc`
echo "Transfered $KBSPEED KB/sec in $TOTALTIME."

Open in new window


What is your earmark for switching from KB to MB?
0
 

Author Comment

by:projects
ID: 40270610
#!/bin/bash

CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://somesite.com/file`
SPEED=`echo $CSTATS | awk '{print $1}'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
KBSPEED=`echo "4k $SPEED 1024.0 /p" | dc`
MBSPEED=`echo "4k $KBSPEED 1024.0 /p" | dc`
echo "Transfered $KBSPEED KB/sec in $TOTALTIME."

# ./go
./go: line 7: dc: command not found
./go: line 8: dc: command not found
Transfered  KB/sec in 2.803.


Don't have dc on this device.

>What is your earmark for switching from KB to MB?

Figured 1000KBytes = 1MByte

Most things are tested in Mb/s these days so that's what I was thinking but if it goes below that, then it should ready in KBytes.
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40270625
How about 'bc' - do you have that?

run:
which bc

Open in new window

0
 

Author Comment

by:projects
ID: 40270697
Nope. This is an embedded router I'm working on so it has limited tools.
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40270764
It's probably running busybox then and not a full-fledged bash, in that case, you're limited to the math you can do.  Expecting anything complex on an embedded router is probably expecting too much.  At any rate, there are 2 solutions below, one using 'let', the other using 'expr'.  Neither does floating point math, so results are rounded to integers.

If 'let' doesn't work, comment out the two 'let' lines and uncomment the 2 'expr' lines and try again.

#!/bin/bash
CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://somesite.com/bigfile`
SPEED=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
let KBSPEED=$SPEED/1024
let MBSPEED=$SPEED/1048576
# If 'let' doesn't work, try expr:
#KBSPEED=`expr $SPEED / 1024`
#MBSPEED=`expr $SPEED / 1048576`
if [ $MBSPEED -gt 1 ] ; then
    echo "Transfered $MBSPEED MB/sec in $TOTALTIME seconds."
else
    echo "Transfered $KBSPEED KB/sec in $TOTALTIME seconds."
fi

Open in new window

0
 

Author Comment

by:projects
ID: 40270776
# ./test
Transfered 3 MB/sec in 3.150 seconds.

Seems to work!
Just have to add the dns timing back.
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40270794
Yup - just modify those last 2 echo commands and put it in wherever you want it. :)
0
 

Author Comment

by:projects
ID: 40270811
Oops, it's rounding out!

# ./test
Transfered 2 MB/sec in 3.409 seconds.
# ./test
Transfered 2 MB/sec in 4.024 seconds.
# ./test
Transfered 3 MB/sec in 3.024 seconds.
# ./test
Transfered 3 MB/sec in 2.663 seconds.


also...

LOL, I have no idea what I'm doing;

    echo "Transfered $MBSPEED MB/sec in $TOTALTIME seconds."
    else
        echo "Transfered $KBSPEED KB/sec in $TOTALTIME seconds."
        else
        echo "DNS Resolve Time was $DNSTIME."
       fi

# ./test
./test: line 15: syntax error near unexpected token `else'
./test: line 15: `        else'
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40270858
I mentioned that before - there isn't anything on your router that can do floating point math.  If you want bytes converted, it'll be rounded.  You could insist on reporting KB/sec, since that will at least give you more info than MB/sec.  Or, as I did below, I set the earmark for 10MB/sec.  9.9MB/sec and below will give results in KB instead (giving you more accurate info)

As for the DNS entry, just add it after the 'fi':

#!/bin/bash
CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://somesite.com/bigfile`
SPEED=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
let KBSPEED=$SPEED/1024
let MBSPEED=$SPEED/1048576
# If 'let' doesn't work, try expr:
#KBSPEED=`expr $SPEED / 1024`
#MBSPEED=`expr $SPEED / 1048576`
if [ $MBSPEED -gt 10 ] ; then
    echo "Transfered $MBSPEED MB/sec in $TOTALTIME seconds."
else
    echo "Transfered $KBSPEED KB/sec in $TOTALTIME seconds."
fi
echo "DNS Resolve Time was $DNSTIME seconds."

Open in new window

0
 

Author Comment

by:projects
ID: 40270874
You know, so long as the output is consistent, I can covert it later on, after it's stored in the database.
Maybe it's best to just leave it as Kb/s or KB/s, no sure which would be the most useful.
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40270929
I wouldn't convert it at all - store it as bytes.  It's easy to convert it later.
0
 

Author Comment

by:projects
ID: 40270934
Can you update your code to show bytes then and remove all of the things we no longer need.
I'd like to close this question and award you :).

Thanks.
0
 
LVL 23

Accepted Solution

by:
nemws1 earned 2000 total points
ID: 40270941
Sure.

#!/bin/bash
CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://somesite.com/bigfile`
SPEED=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
echo "Transfered $SPEED bytes/sec in $TOTALTIME seconds."
echo "DNS Resolve Time was $DNSTIME seconds."

Open in new window

0
 

Author Comment

by:projects
ID: 40270946
Done. Thanks very much for sticking to this and helping me out.
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40270948
Sure thing.  Glad we figured out something that works for you.
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40271011
Can you post the code of your function?  Maybe you're not passing in the URL correctly somehow or something is getting parsed oddly.  Probably want to make it a new question.
0
 

Author Comment

by:projects
ID: 40271030
I'm confused again.

When running your script, while it is taking longer than usual at 7 seconds, it is not failing.

However, when I put this code into my main script as a function, then it constantly fails, even when I add -m or --connect-timeout options, I get;

curl: (6) name lookup timed out

Any thoughts?
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40271032
Are you passing the URL to the function or is it hard-coded?
0
 

Author Comment

by:projects
ID: 40271033
function band_test()
{
        echo "Doing Bandwidth test"
        CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://siteurl/file`
        SPEED=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
        DNSTIME=`echo $CSTATS | awk '{print $2}'`
        TOTALTIME=`echo $CSTATS | awk '{print $3}'`

echo "$SPEED, $TOTALTIME, $DNSTIME"

        TESTTIME="$(date +"%F %T")"
        $CURL -F function=band_test -F test_time="$TESTTIME" -F speed=$SPEED -F dnstime=$DNSTIME -F totaltime=$TOTALTIME
}
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40271054
Dangit.  I don't see anything wrong in here. :(

I cut-paste what you pasted here into a new script (changed the URL) and called 'band_test' and it worked just fine. :(
0
 

Author Comment

by:projects
ID: 40271159
Must be something to do with the dns servers where I'm testing from.
I've got it in a loop, doing the test every few seconds.

Doing Bandwidth test
1266023, 8.282, 5.009
curl: (6) name lookup timed out
Doing Bandwidth test
1106805, 9.474, 5.009
curl: (6) name lookup timed out
Doing Bandwidth test
1292441, 8.113, 5.009
curl: (6) name lookup timed out

Notice how it's returning the results then THEN complaining about the DNS?
Also strange how it's saying that each DNS lookup is taking the exact same amount of time.
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40271175
The DNS lookup might be taking the same time every time.  Looks like you have a server with a bad DNS round-robin entry.  Thus it fails every other time.
0
 

Author Comment

by:projects
ID: 40271297
That's what it was alright. My fibre channel storage failed for some reason so lost all of the vms.

All good now, thanks again!
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40271302
awesome. :)
0
 

Author Comment

by:projects
ID: 40271326
If you don't mind... could I ask you one more tiny question...

Based on the output that we are getting, what do you think the proper field types might be in mysql?


speed       dns_time       total_time

I'll fully understand if you prefer I ask another question :)
0
 
LVL 23

Expert Comment

by:nemws1
ID: 40271333
I assume band_time is the time an which the stats were gathered. An UNSIGNED int would be able to store a value up to 4GB/sec (even storing it as a byte).  Should be good for a few years. :)

When you do your insert, specify the fields names and values and just don't provide band_time (and in my example, you don't specify 'id' either) - it (they) will get auto-filled in for you if you specify it like this.

CREATE TABLE bandwidth_monitor (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
    , speed INT UNSIGNED NOT NULL
    , dns_time FLOAT(10,4) NOT NULL
    , total_time FLOAT(10,4) NOT NULL
    , band_time TIMESTAMP
);

INSERT INTO bandwidth_monitor (speed, dns_time, total_time) VALUES (1292441, 8.113, 5.009);
0
 

Author Comment

by:projects
ID: 40271354
I'll start another question because this is going to lead to needing more examples :).

Thanks again.
0
 

Author Comment

by:projects
ID: 40271393
There you go, if you're still interested :)

http://www.experts-exchange.com/Programming/Languages/Scripting/PHP/Q_28501182.html

Open in new window

0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Background Still having to process all these year-end "csv" files received from all these sources (including Government entities), sometimes we have the need to examine the contents due to data error, etc... As a "Unix" shop, our only readily …
How to remove superseded packages in windows w60 or w61 installation media (.wim) or online system to prevent unnecessary space. w60 means Windows Vista or Windows Server 2008. w61 means Windows 7 or Windows Server 2008 R2. There are various …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…

826 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question