curl download speed bytes megabytes

Using curl in a bash script to download a file how can I do the following;

-extract the average download speed in Bytes or Megabytes into two variables such as B or M. Example B 543 or M 23 ( $BYTES and $SPEED)
-extract the dns lookup time into a variable
-extract the 'time total' of the download into a variable

If I send the curl output to a 'report' file, then using
awk < report  'FNR == 3 {print $7, $9 }'
gives me the proper output.

However, when including it as part of a full command, nothing;
curl -o /dev/null -LO http://www.somesite..com/thebigfile | awk 'FNR == 3 {print $7, $9 }'
projectsAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

nemws1Database AdministratorCommented:
You want to craft something with the '-w' command line option:

CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://www.somesite.com/thebigfile`
SPEED=`echo $CSTATS | awk '{print $1}'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`

Open in new window

0
projectsAuthor Commented:
I must not be understanding something.

When using curl -h, I don't see any options which I keep reading about such as 'time_name' and 'time_total' for example.

I took your code and put it into a bash script then printed the following.

SPEED=`echo $CSTATS | awk '{print $1}'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`

echo "$SPEED, $DNSTIME, $TOTALTIME"

But the output is;

# ./go
2953493.000, 0.003, 3.550

However, I'm trying to output speed in KB or MB.
0
nemws1Database AdministratorCommented:
Yup.  That's in Bytes/second, which you said was acceptable in your initial post.

To convert to KB and/or MB:
KBSPEED=`echo "4k $SPEED 1024.0 /p" | dc`
MBSPEED=`echo "4k $KBSPEED 1024.0 /p" | dc`

Open in new window

0
Cloud Class® Course: C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

nemws1Database AdministratorCommented:
And 'curl -h' doesn't tell you all the options to '-w' - it just tells you '-w' is there.  You have to read the manual page to see all the options.  Here's a link to the '--write-out' format spec:

http://curl.haxx.se/docs/manpage.html#-w
0
projectsAuthor Commented:
Can you roll it all into one so I can see it completely. I like when folks post the full code, then I can mess with it and add to it, etc.

As to my original question what I am asking for is;

 -extract the average download speed in Bytes or Megabytes into two variables such as B or M. Example B 543 or M 23 ( $BYTES and $SPEED)

No one non technical would understand 2953493.000 as a result :)

What I need for speed output is more like mentioned above but I see I made an error asking for B.
That should be something more like;

978 KB/s or if the speed goes into Mb/s, 12Mb/s as output.
Kilo BYTES                                        Mega BITS
0
projectsAuthor Commented:
I didn't know about the additional information on curl. I did search but since I didn't see the options when I typed -h for my own version, I assumed mine didn't have those options. Thanks for that heads up, very useful as I will learn more about curl now.
0
nemws1Database AdministratorCommented:
Like this?
CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://www.somesite.com/thebigfile`
SPEED=`echo $CSTATS | awk '{print $1}'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
KBSPEED=`echo "4k $SPEED 1024.0 /p" | dc`
MBSPEED=`echo "4k $KBSPEED 1024.0 /p" | dc`
echo "Transfered $KBSPEED KB/sec in $TOTALTIME."

Open in new window


What is your earmark for switching from KB to MB?
0
projectsAuthor Commented:
#!/bin/bash

CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://somesite.com/file`
SPEED=`echo $CSTATS | awk '{print $1}'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
KBSPEED=`echo "4k $SPEED 1024.0 /p" | dc`
MBSPEED=`echo "4k $KBSPEED 1024.0 /p" | dc`
echo "Transfered $KBSPEED KB/sec in $TOTALTIME."

# ./go
./go: line 7: dc: command not found
./go: line 8: dc: command not found
Transfered  KB/sec in 2.803.


Don't have dc on this device.

>What is your earmark for switching from KB to MB?

Figured 1000KBytes = 1MByte

Most things are tested in Mb/s these days so that's what I was thinking but if it goes below that, then it should ready in KBytes.
0
nemws1Database AdministratorCommented:
How about 'bc' - do you have that?

run:
which bc

Open in new window

0
projectsAuthor Commented:
Nope. This is an embedded router I'm working on so it has limited tools.
0
nemws1Database AdministratorCommented:
It's probably running busybox then and not a full-fledged bash, in that case, you're limited to the math you can do.  Expecting anything complex on an embedded router is probably expecting too much.  At any rate, there are 2 solutions below, one using 'let', the other using 'expr'.  Neither does floating point math, so results are rounded to integers.

If 'let' doesn't work, comment out the two 'let' lines and uncomment the 2 'expr' lines and try again.

#!/bin/bash
CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://somesite.com/bigfile`
SPEED=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
let KBSPEED=$SPEED/1024
let MBSPEED=$SPEED/1048576
# If 'let' doesn't work, try expr:
#KBSPEED=`expr $SPEED / 1024`
#MBSPEED=`expr $SPEED / 1048576`
if [ $MBSPEED -gt 1 ] ; then
    echo "Transfered $MBSPEED MB/sec in $TOTALTIME seconds."
else
    echo "Transfered $KBSPEED KB/sec in $TOTALTIME seconds."
fi

Open in new window

0
projectsAuthor Commented:
# ./test
Transfered 3 MB/sec in 3.150 seconds.

Seems to work!
Just have to add the dns timing back.
0
nemws1Database AdministratorCommented:
Yup - just modify those last 2 echo commands and put it in wherever you want it. :)
0
projectsAuthor Commented:
Oops, it's rounding out!

# ./test
Transfered 2 MB/sec in 3.409 seconds.
# ./test
Transfered 2 MB/sec in 4.024 seconds.
# ./test
Transfered 3 MB/sec in 3.024 seconds.
# ./test
Transfered 3 MB/sec in 2.663 seconds.


also...

LOL, I have no idea what I'm doing;

    echo "Transfered $MBSPEED MB/sec in $TOTALTIME seconds."
    else
        echo "Transfered $KBSPEED KB/sec in $TOTALTIME seconds."
        else
        echo "DNS Resolve Time was $DNSTIME."
       fi

# ./test
./test: line 15: syntax error near unexpected token `else'
./test: line 15: `        else'
0
nemws1Database AdministratorCommented:
I mentioned that before - there isn't anything on your router that can do floating point math.  If you want bytes converted, it'll be rounded.  You could insist on reporting KB/sec, since that will at least give you more info than MB/sec.  Or, as I did below, I set the earmark for 10MB/sec.  9.9MB/sec and below will give results in KB instead (giving you more accurate info)

As for the DNS entry, just add it after the 'fi':

#!/bin/bash
CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://somesite.com/bigfile`
SPEED=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
let KBSPEED=$SPEED/1024
let MBSPEED=$SPEED/1048576
# If 'let' doesn't work, try expr:
#KBSPEED=`expr $SPEED / 1024`
#MBSPEED=`expr $SPEED / 1048576`
if [ $MBSPEED -gt 10 ] ; then
    echo "Transfered $MBSPEED MB/sec in $TOTALTIME seconds."
else
    echo "Transfered $KBSPEED KB/sec in $TOTALTIME seconds."
fi
echo "DNS Resolve Time was $DNSTIME seconds."

Open in new window

0
projectsAuthor Commented:
You know, so long as the output is consistent, I can covert it later on, after it's stored in the database.
Maybe it's best to just leave it as Kb/s or KB/s, no sure which would be the most useful.
0
nemws1Database AdministratorCommented:
I wouldn't convert it at all - store it as bytes.  It's easy to convert it later.
0
projectsAuthor Commented:
Can you update your code to show bytes then and remove all of the things we no longer need.
I'd like to close this question and award you :).

Thanks.
0
nemws1Database AdministratorCommented:
Sure.

#!/bin/bash
CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://somesite.com/bigfile`
SPEED=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
DNSTIME=`echo $CSTATS | awk '{print $2}'`
TOTALTIME=`echo $CSTATS | awk '{print $3}'`
echo "Transfered $SPEED bytes/sec in $TOTALTIME seconds."
echo "DNS Resolve Time was $DNSTIME seconds."

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
projectsAuthor Commented:
Done. Thanks very much for sticking to this and helping me out.
0
nemws1Database AdministratorCommented:
Sure thing.  Glad we figured out something that works for you.
0
nemws1Database AdministratorCommented:
Can you post the code of your function?  Maybe you're not passing in the URL correctly somehow or something is getting parsed oddly.  Probably want to make it a new question.
0
projectsAuthor Commented:
I'm confused again.

When running your script, while it is taking longer than usual at 7 seconds, it is not failing.

However, when I put this code into my main script as a function, then it constantly fails, even when I add -m or --connect-timeout options, I get;

curl: (6) name lookup timed out

Any thoughts?
0
nemws1Database AdministratorCommented:
Are you passing the URL to the function or is it hard-coded?
0
projectsAuthor Commented:
function band_test()
{
        echo "Doing Bandwidth test"
        CSTATS=`curl -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s http://siteurl/file`
        SPEED=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
        DNSTIME=`echo $CSTATS | awk '{print $2}'`
        TOTALTIME=`echo $CSTATS | awk '{print $3}'`

echo "$SPEED, $TOTALTIME, $DNSTIME"

        TESTTIME="$(date +"%F %T")"
        $CURL -F function=band_test -F test_time="$TESTTIME" -F speed=$SPEED -F dnstime=$DNSTIME -F totaltime=$TOTALTIME
}
0
nemws1Database AdministratorCommented:
Dangit.  I don't see anything wrong in here. :(

I cut-paste what you pasted here into a new script (changed the URL) and called 'band_test' and it worked just fine. :(
0
projectsAuthor Commented:
Must be something to do with the dns servers where I'm testing from.
I've got it in a loop, doing the test every few seconds.

Doing Bandwidth test
1266023, 8.282, 5.009
curl: (6) name lookup timed out
Doing Bandwidth test
1106805, 9.474, 5.009
curl: (6) name lookup timed out
Doing Bandwidth test
1292441, 8.113, 5.009
curl: (6) name lookup timed out

Notice how it's returning the results then THEN complaining about the DNS?
Also strange how it's saying that each DNS lookup is taking the exact same amount of time.
0
nemws1Database AdministratorCommented:
The DNS lookup might be taking the same time every time.  Looks like you have a server with a bad DNS round-robin entry.  Thus it fails every other time.
0
projectsAuthor Commented:
That's what it was alright. My fibre channel storage failed for some reason so lost all of the vms.

All good now, thanks again!
0
nemws1Database AdministratorCommented:
awesome. :)
0
projectsAuthor Commented:
If you don't mind... could I ask you one more tiny question...

Based on the output that we are getting, what do you think the proper field types might be in mysql?


speed       dns_time       total_time

I'll fully understand if you prefer I ask another question :)
0
nemws1Database AdministratorCommented:
I assume band_time is the time an which the stats were gathered. An UNSIGNED int would be able to store a value up to 4GB/sec (even storing it as a byte).  Should be good for a few years. :)

When you do your insert, specify the fields names and values and just don't provide band_time (and in my example, you don't specify 'id' either) - it (they) will get auto-filled in for you if you specify it like this.

CREATE TABLE bandwidth_monitor (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
    , speed INT UNSIGNED NOT NULL
    , dns_time FLOAT(10,4) NOT NULL
    , total_time FLOAT(10,4) NOT NULL
    , band_time TIMESTAMP
);

INSERT INTO bandwidth_monitor (speed, dns_time, total_time) VALUES (1292441, 8.113, 5.009);
0
projectsAuthor Commented:
I'll start another question because this is going to lead to needing more examples :).

Thanks again.
0
projectsAuthor Commented:
There you go, if you're still interested :)

http://www.experts-exchange.com/Programming/Languages/Scripting/PHP/Q_28501182.html

Open in new window

0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.