Solved

Convert curl speed_dwonload to Mbps

Posted on 2016-09-22
29
66 Views
Last Modified: 2016-09-27
I found this in another question and tried it. However, I'm not sure how accurate it is and I also need a Mbps output.
Can someone edit this for me for the output I am needing.

#!/bin/bash

        BAND_TIME="$(date +"%F %T")"

        CSTATS=`curl -m 30 -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s https://example.com/50mbfile`
        BITSPS=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
        DNS_TIME=`echo $CSTATS | awk '{print $2}'`
        TOTAL_TIME=`echo $CSTATS | awk '{print $3}'`

        echo Time="$BAND_TIME"
        echo Bitsps="$BITSPS"
        echo Dns_time="$DNS_TIME"
        echo Total_time="$TOTAL_TIME"

Open in new window

0
Comment
Question by:Mark Lewis
  • 15
  • 14
29 Comments
 
LVL 76

Expert Comment

by:arnold
ID: 41813173
bits per second => bytes per second (devide number of bits by 8)
Bytes => kbytes (divide number of bytes by 1024)
Kbytes => Mbytes (divide number of kbytes by 1024)

To get mega bits per second, skip the conversion from bits to bytes.
0
 

Author Comment

by:Mark Lewis
ID: 41813178
Can you just adjust the code to make it do that. I'm not really sure how.
0
 
LVL 76

Expert Comment

by:arnold
ID: 41813186
divide bits
add
MBITSPS=${echo $BITSPS / 1024 / 1024 | bc )
Converts bits per sec to mega bits per second

and also add
echo "Mbpts = $MBITSPS"
0
 

Author Comment

by:Mark Lewis
ID: 41813200
Your code is broken above. Can you simply adjust the code I posted. I'm not 100% sure of the coding.

I've tried these;

       MBITSPS=$(echo $BITSPS / 1024 / 1024 | bc )
       MBITSPS=`echo $BITSPS / 1024 / 1024 | bc `

Neither works, leads to errors. Here is the last one based on the above.
0
 
LVL 76

Expert Comment

by:arnold
ID: 41813218
in the shell, run which bc
see if you have bc installed on your system. that might be the issue


try
MBITSPS=$(( $BITSPS /1024/1024 ))
0
 

Author Comment

by:Mark Lewis
ID: 41813250
Yes, bc is installed. Running your latest line;

        MBITSPS=$(( $BITSPS /1024/1024 ))

I get;

./band2: line 7: /1024/1024 : syntax error: operand expected (error token is "/1024/1024 ")
0
 
LVL 76

Expert Comment

by:arnold
ID: 41813259
Check what the value of $BITSPS
 

#!/bin/bash

        BAND_TIME="$(date +"%F %T")"

        CSTATS=`curl -m 30 -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s https://example.com/50mbfile`
        BITSPS=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
        DNS_TIME=`echo $CSTATS | awk '{print $2}'`
        TOTAL_TIME=`echo $CSTATS | awk '{print $3}'`

        echo Time="$BAND_TIME"
         MBITSPS=$(( $BITSPS / 1024 / 1024 ))
        echo Bitsps="$BITSPS"
        echo Mbitsps="$MBITSPS"
        echo Dns_time="$DNS_TIME"
        echo Total_time="$TOTAL_TIME"

Open in new window


Without knowing what the value in BITSPS if empty, you would get the above erro as the evaluation is on  /  1024 / 1024 .....
0
 

Author Comment

by:Mark Lewis
ID: 41813974
I'm missing something.

I added the code as you suggested and here is the result.

        echo Time="$BAND_TIME"
         MBITSPS=$(( $BITSPS / 1024 / 1024 ))
        echo Bitsps="$BITSPS"
        echo Mbitsps="$MBITSPS"
        echo Dns_time="$DNS_TIME"
        echo Total_time="$TOTAL_TIME"

# ./band
Time=2016-09-24 18:09:11
Bitsps=1002638
Mbitsps=0
Dns_time=0.008
Total_time=5.186

Not sure what I am missing.
0
 
LVL 76

Expert Comment

by:arnold
ID: 41813991
it is reporting that the 1002638 bits is 0 Mbits

It is a integer operation.
you could use
MBITSPS= $(( $BITSPS /1048576))  remainder $(( $BITSPS % 1048576 ))


The bc example posted earlier will provide a fractional/decimal response, but BITSPS must have a value...
So you need to test:
if ( /bin/test -z "$BITSPS") ; then echo "Error BITSPS has no VALUE, empty string"; exit fi
0
 

Author Comment

by:Mark Lewis
ID: 41814142
No idea how to make this work. I'm not a programmer.
0
 
LVL 76

Expert Comment

by:arnold
ID: 41814146
#!/bin/bash

        BAND_TIME="$(date +"%F %T")"

        CSTATS=`curl -m 30 -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s https://example.com/50mbfile`
        BITSPS=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
        DNS_TIME=`echo $CSTATS | awk '{print $2}'`
        TOTAL_TIME=`echo $CSTATS | awk '{print $3}'`

        echo Time="$BAND_TIME"
          if ( test ! -z "$BITSPS" ) ; then
         MBITSPS=$( echo "$BITSPS / 1024.00 / 1024.00" | /usr/bin/bc ) 
         
         else
                echo "BITSPS has no value"
         fi
        echo Bitsps="$BITSPS"
        echo Mbitsps="$MBITSPS"
        echo Dns_time="$DNS_TIME"
        echo Total_time="$TOTAL_TIME"

Open in new window


The issue the data is integer, Whole numbers ......
0
 

Author Comment

by:Mark Lewis
ID: 41814206
I'm not understanding why this is an issue if it is known? I updated using your code but still getting 0 for Mbps.

# ./band
Time=2016-09-25 01:41:43
Bitsps=1252
Mbitsps=0
Dns_time=0.385
Total_time=1.014
0
 
LVL 76

Expert Comment

by:arnold
ID: 41814215
whenn bitsps reaches 1048546 only then will MBITSPS will be 1.

In the option you have it only represent data as integer nor decimal/fractions.
0
 

Author Comment

by:Mark Lewis
ID: 41814230
Basically, I just took this code from another question and don't need specific limits.

The only thing I'd like to do is test to another server where I have put some 5,10,50,100MB sized files and would like to see the Mbps results of the test.

I changed the 50MB file to a 100MB one but the script doesn't seem to run since it ends at exactly the same time each time I run it no matter which file size I enter.

Can I not get both the bps rating and Mbps? I think you are saying if there is not enough of a transfer, it never reaches the Mbps but as explained, I've also changed file sizes which should have worked.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 76

Expert Comment

by:arnold
ID: 41814351
You selected to use bash as the shell script, there is/are more elaborate steps to get the fractional/decimal representations.

I added a repesentation for bits per second
kbitsps and mbitsps

#!/bin/bash

        BAND_TIME="$(date +"%F %T")"

        CSTATS=`curl -m 30 -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s https://example.com/50mbfile`
        BITSPS=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
        DNS_TIME=`echo $CSTATS | awk '{print $2}'`
        TOTAL_TIME=`echo $CSTATS | awk '{print $3}'`

        echo Time="$BAND_TIME"
          if ( test ! -z "$BITSPS" ) ; then
         MBITSPS=$( echo "$BITSPS / 1024 / 1024.00" | /usr/bin/bc )
         KBITSPS=$( echo "$BITSPS / 1024 " | /usr/bin/bc )

         else
                echo "BITSPS has no value"
         fi
        echo Bitsps="$BITSPS"
        echo Kbitsps="$KBITSPS"
        echo Mbitsps="$MBITSPS"
        echo Dns_time="$DNS_TIME"
        echo Total_time="$TOTAL_TIME"

Open in new window



Not sue what the bits output is, perform the calculation.  incease the timeout from 30 seconds to a minute and see if you ......
0
 

Author Comment

by:Mark Lewis
ID: 41815059
I've no idea then. I've run what you gave me the Mbps is always 0.
0
 
LVL 76

Expert Comment

by:arnold
ID: 41815064
Do the BITSPS you get is larger than 1048564 this will be the 1Mbit mark
Look at the interim Kbps which need to be 1024 to hit the 1Mbitsps

I do not understand what the issue is.
Your test does not t reach the needed treshhold to exceed the 1bps......

Are you open to using another/alternate scripting ?

For quick approximation, using the bitsps and use 1000 instead of 1024 thus moving the decimal place left by 6 to get to what it might be in Mbitsps.
0
 

Author Comment

by:Mark Lewis
ID: 41815077
I understand what you are saying but it does not make sense that the test never reaches 1Mbps.

Let me try to explain.

I am testing on a 50Mbps connection to a much faster network/server which doesn't have any limitations that could prevent the test from working.

When I set the file size to 1000mb file size, the script just dies, no output what so ever.
When I set the file size to 100mb, this is the output I get, as expected

Time=2016-09-25 21:09:33
Bitsps=2004284
Kbitsps=1957
Mbitsps=1
Dns_time=0.003
Total_time=30.000

BUT, it took 30 seconds for the test. Is it possible that curl is simply not able to do this kind of test well?
0
 
LVL 76

Accepted Solution

by:
arnold earned 500 total points
ID: 41815181
Try the following:

CSTATS=`curl -m 30 -w '%{speed_download}\t%{time_namelookup}\t%{time_total}\n' -o /dev/null -s https://example.com/50mbfile`
        BITSPS=`echo $CSTATS | awk '{print $1}' | sed 's/\..*//'`
        DNS_TIME=`echo $CSTATS | awk '{print $2}'`
        TOTAL_TIME=`echo $CSTATS | awk '{print $3}'`

        echo Time="$BAND_TIME"
          if ( test ! -z "$BITSPS" ) ; then
         MBITSPS=$( echo "$BITSPS" | awk ' { print $1 /1024/1024"Mbpts" } '  )
         KBITSPS=$( echo "$BITSPS" | awk ' { print $1/1024"Kbpts"  }  '  )

         else
                echo "BITSPS has no value"
         fi
        echo Bitsps="$BITSPS"
        echo Kbitsps="$KBITSPS"
        echo Mbitsps="$MBITSPS"
        echo Dns_time="$DNS_TIME"
        echo Total_time="$TOTAL_TIME"

Open in new window

0
 

Author Comment

by:Mark Lewis
ID: 41816585
Using a 50mb file.

# ./band2
Time=2016-09-26 18:03:48
Bitsps=1973957
Kbitsps=1927.69Kbpts
Mbitsps=1.88251Mbpts
Dns_time=0.004
Total_time=26.343

Not understanding why the test doesn't get very high since the local connection is 50Mbps to a much faster one. Using iperf or any other tool, I can easily get close to the 50Mbps+ when testing. However, I cannot always use iperf which is why I'm interested in this script working.
0
 

Author Comment

by:Mark Lewis
ID: 41816598
I notice your output says kbpts and Mbpts.
Are you specifying these as packets or bits?

To me, these results should look as follows;

bps=1973957
Kbps=1927.69Kbps
Mbps=1.88251Mbps
0
 
LVL 76

Expert Comment

by:arnold
ID: 41817007
Bits based on your BITSPS VARIABLE.
0
 

Author Comment

by:Mark Lewis
ID: 41818000
Oh yes, I see that now.

So, this script works but why do you think it can never ramp up to a real speed?
Basically, no matter what it does, so long as it's real and consistent that I could use as some sort of gauge, it might be useful.
0
 
LVL 76

Expert Comment

by:arnold
ID: 41818039
The issue was with trying to use bash arithmetic to perform the division which is what I referenced since bash arithmetic outputs integers only.
This is why adjusted to use awk to perform the arithmetic......

"real speed" is contingent on many things. Time of day, configuration of the server from whom the data is retrieved, prioritization of data types. Devices in the path, router, switches, etc.

Try using different/larger size files to make the connection establishing/terminating portion less significant.
Add the file size to your data and see whether the data size you are transferring makes a difference.

Glad I could help.
0
 

Author Comment

by:Mark Lewis
ID: 41818081
Of course I understand bandwidth and throughput, I was asking about that but was explaining that I am testing on an unused (lots of bandwidth not being used) 50Mbps connection to a very fast server.

I can test using iperf and other tools and can get close to my 50Mbps but never when using this script which only ever gives maybe a little over 1Mbps.
I also explained that I have used different file sizes and those make no sense either.

However, you have given me the solution to the original question so I'll award this and start a new question.
0
 

Author Comment

by:Mark Lewis
ID: 41818242
I just found something interesting. Using http rather than https gives a VERY different result.
Both tests using only a 10mb file

# ./band2 (https)
Time=2016-09-27 16:47:48
Bitsps=1380068
Kbitsps=1347.72Kbpts
Mbitsps=1.31614Mbpts
Dns_time=0.003
Total_time=7.598

# ./band3 (http)
Time=2016-09-27 16:47:59
Bitsps=6358913
Kbitsps=6209.88Kbpts
Mbitsps=6.06433Mbpts
Dns_time=0.005
Total_time=1.649

So, it is not so much a curl problem as a difference between http and https.
0
 
LVL 76

Expert Comment

by:arnold
ID: 41818422
https deals with encrypting data on the sending side while decrypting data on the other side. The encryption adds ovehead to the process/data.
The process of establishing a connection is the same, the https has additional steps dealing with the two sides exchanging credentials (certificates, public keys) using these each sides encrypts and send their data to the other.
The receiving side using their private key decrypts the information, and responds .....


This is also reflect in the time to complete.
0
 
LVL 76

Expert Comment

by:arnold
ID: 41818442
While you have a 50 MB feed, it does not mean that the systems for example.com are willing to dedicate their server and network resources to transmit the file to you at 50 MB no matter how wide their pipe is.
Think of it as the water department and water distribution. Lets say you hire a contractor to replace the 4 inch pipe that usually feeds your house and replacing it with a 12 inch pipe, the flow of water into your house will not increase three fold, the way water is pushed is through pressure, such that you changing the pipe going into your house will mean the water pressure will decline which will reduce the speed at which water flows.

With that said data throughput from servers of example.com relies on the slowest path along the way

example.com <=> their provider <=> their upstream providers <=> your providers upstream provider<=> your provider <=> your neighborhood <=> you
lattency, rerouting, transmission  of packets etc will impact the throughput.

All these tests are the performance of the transfer at the time. repeating the same process sequentially may offer different results. simultaneously retrieving the same file from two user shells may reflect no decline in performance where both are retrieved at the same time.
0
 

Author Comment

by:Mark Lewis
ID: 41818812
I'm aware of all the above and have no false hopes or misunderstandings.

In the end, I didn't notice that we were using https. Once I changed it, the results make sense. I was simply pointing that out.

All is fine now, the code works perfectly.

Thank you for that help.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Introduction This warning has to be one of the most commonly issued warnings in the history of PHP.  The article explains why this warning arises and what to do to mitigate the problem. How this Happens HTTP headers include many different kinds…
Utilizing an array to gracefully append to a list of EmailAddresses
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now