curl

Curl is a very powerful tool with lots of useful flags.. (it can be used to test various url and is sometimes more useful than using a browser)

Basic usage
Advanced usage
Troubleshooting a Specific Url from on/off the network (using firebug/strace)
troubleshooting slow sites 2
Setting up crons for troubleshooting problem sites
More reading

Basic usage

For more details on appache error codes see Error codes

    • curl http://www.bbc.co.uk # check you can connect and displays raw output
    • curl bbc.co.uk -v # verbose feedback
    • curl –o output.txt bbc.co.uk # Save output to a file or if you just want to see the output stats curl bbc.co.uk > /dev/null
    • curl –o /dev/null bbc.co.uk # To see just stats output to null
    • curl -vv https://signin.ebay.co.uk/ws/eBayISAPI.dll?SellItem 2>&1 | egrep ^\\* - To see https details (like clicking on cert in browser)
    • curl –I bbc.co.uk/index.php - read the http header only
    • curl -sv --header "host: bbc.co.uk" localhost -o /dev/null # s=silent mode,v=verbose,--header= Extra header to use when getting a web page
    • curl –w “%{http_code}\n” bbc.co.uk – includes error code, other parameters can be included to details time , download size…
    • curl -w '%{http_code} %{time_total}' http://bbc.co.uk/| tail -1 #check speed of a site
    • curl -L http://bbc.co.uk follows any redirects setup
    • curl --output /dev/null --write-out "resolve: %{time_namelookup}\nconnect: %{time_connect}\npretrans: %{time_pretransfer}\nstarttrans: %{time_starttransfer}\ntotal time: %{time_total}\n" --silent http://trusonic.com # good for testing the speed of a site
    • curl -k don't worry about bad ssl certs
    • curl -silent Don’t show progress meter or error messages.  Makes Curl mute
    • curl -I http://www.example.org 2>/dev/null | head -n 1 | cut -d$' ' -f2 will show you just the error code

Advanced usage

To get the amount of time between when a connection is established and when the data actually begins to be transferred, subtract the value of time_pretransfer from time_starttransfer. You can automate this by sending the output to bc with echo:
echo "`curl -s -o /dev/null -w '%{time_starttransfer}-%{time_pretransfer}' http://linux.com/`"|bc

A key metric is time to first byte (TTFB) i.e the time your browser sepnds waiting for the first byte (order dns lookup,initial connection,waiting,receive data,closing connection).
Scores
Under 100MS for static content – very good
200MS-500MS (dynamic content) –good
Else generally bad although obviously lots of variables (maybe slow disks/ram/http server)

To test:

curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://bbc.co.uk

To spoof another browser in curl run curl -A "UserAgentString" http://url.com i.e curl -A "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" http://www.apple.com

more details at
http://osxdaily.com/2011/07/16/change-user-agent-with-curl/

Troubleshooting a Specific Url from on/off the network (using firebug/strace)

worked example

For testing i have created https://linuxblog.info/slow.php which will take 5 seconds to load
A worked example of troubleshooting that site is https://linuxblog.info/worked-example-slow-speeds/

Some starter notes

note to get the full url use firebug (not firefox inspect element is a very similar program), open the url in questions, right click the relevant post line and select copy as curl
more info on this at http://www.softwareishard.com/blog/planet-mozilla/firebug-tip-resend-http-request/

you also need the extra flags with the strace to get the full output

you may need to add  -L so the curl command to follow any redirects, adding -o /dev/null -silent and time will make the output nicer so curl linuxblog.info becomes

time curl -L -o /dev/null -silent linuxblog.info

if you have lots of appache processes running echo "start `date`" ;time curl -L -o /dev/null -silent linuxblog.info; echo "end `date`"
will give you some dates to use in narrowing down the pid

working the problem

Example problem -you need to investigate a problem with a site (typically it being slow) and it not being clear why
(first steps check server performance, check DB, check for recent changes...)

Solution (a lot of these steps need to run at a similar time, so it may take a few goes to get the timing correct)

1) if you only have 1 Server hosting you Web/Database (this is a lot easier)

a) paste the url from firebug into a browser (note its easiest not to use the local server where the web/database are running otherwise you may need to change some of the curl parameters,if you want to go down this root see step 2)
b) make sure you have enough space before running ps auxw | grep httpd | awk '{print"-p " $2}' | xargs strace -ttT -s 200 > /tmp/l2 2>&1 unless you have another method of narrowing down the specific pid if so you can run strace -fp 5870 -ttT -s 200 > /tmp/l2 2>&1 )
c) cat /tmp/l2 | grep something_unique_from_url to find the pid that was hit
d) cat /tmp/l2 | grep 'pid' to find what happened to the process you ran, ideally you are looking for a gap in the logs which correlates to the timings a normal browser is taking i.e a page load time of 30 seconds would be presented in the logs by a 30 second gap (more details on strace )

2) if you have multiple web servers you will need to run the command either locally or with a specific hostname in the curl (depandant on your server/network setup) so it gets routed to a particular webserver and from there you can run your strace

a) as above but modify the curl to
curl 'http://webserver1/’-H '....... (i.e curl https://127.0.0.1/slow.php )

another example

time curl 'https://linuxblog.info/slow.php' -H 'Host: linuxblog.info' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: wp-settings-1=editor%3Dhtml%26ed_size%3D361%26wplink%3D1%26mfold%3Do%26libraryContent%3Dbrowse%26urlbutton%3Dpost; wp-settings-time-1=1463645731; wordfence_verifiedHuman=6cf4f1e06387ec412722104d4d9e87a9; wfvt_1893908242=577f75ba9f0c1' -H 'Connection: keep-alive'
real    0m5.200s
user    0m0.093s
sys     0m0.082s

changes to

time curl 'http://127.0.0.1/slow.php' -H 'Host: linuxblog.info' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate' -H 'Cookie: wp-settings-1=editor%3Dhtml%26ed_size%3D361%26wplink%3D1%26mfold%3Do%26libraryContent%3Dbrowse%26urlbutton%3Dpost; wp-settings-time-1=1463645731; wordfence_verifiedHuman=6cf4f1e06387ec412722104d4d9e87a9; wfvt_1893908242=577f75ba9f0c1' -H 'Connection: keep-alive'
real    0m5.035s
user    0m0.003s
sys     0m0.023s

and another example

curl 'https://abc/app/home' -H 'Host: abc' -H....

change to

curl -L 'http://mywebserver/app/home' -H 'Host:abc'

tip 1 if you are struggling to get it working on localhost try the original curl with a -IL parameter to check what the header should look like and then change the curl to  a hostname/ip that resolves to a local ip again with the -IL parameters.

tip 2 if you are struggling to find your curl in the appache strace try changing the parameters i.e Firefox/45.0 to Firefox/45.1

b) as above
c) as above
d) as above

what was the problem? (taken from real life examples)

1) in 1 case running as strace showed the delay was on file descriptor 8 i.e

[pid  7010] 07:44:05.138924 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED, NULL) = 19285 <0.000149>
[pid  7010] 07:44:05.139068 poll([{fd=8, events=POLLIN}], 1, 15000 <unfinished ...>
[pid  7010] 07:44:12.702324 <... poll resumed> ) = 1 ([{fd=8, revents=POLLIN}]) <7.563239>

running it all again and adding another shell running  ps auxw | grep httpd | awk '{print"-p " $2}' | xargs lsof >> /tmp/l7_1 2>&1 ; netstat -plant >> /tmp/l7_2 2>&1

showed it was the curl command that was bottleneck!

Running the curl command again with just the basic parameter i.e time curl -L linuxblog.info -o /dev/null showed it worked fine. So by process of elimination of various curl parameters the cookie parameter in the curl was in the end the problem. replacing the cookie parameter (-H 'Cookie:xxxx)  with literally anything i.e test123 ... and the site worked fine.

a similar approach as seen at http://endlessgeek.com/2014/03/strace-php-process-via-apache/

troubleshooting slow sites 2

method 1

screen 1

curl linuxblog.info/slow.php -o /dev/null 2> /dev/null
real 0m5.035s
user 0m0.010s
sys 0m0.017s

screen 2
HTTP_HOST=www.linuxblog.info REQUEST_URI=/ strace -tt -e trace=sendto,connect,open,write,read php /abc/abc/slow.php

method 2

screen 1

ps auxw | grep "/usr/sbin/httpd" | awk '{print"-p " $2}' | xargs strace -o outfile -f

screen 2

while true; do ps -ef | grep -v grep | grep php | awk '{print $2, $8}';done

screen 3 open problem url

then look at the terminal screen from Screen 2, and make a note of the process ID that was recorded WHEN you loaded the problem URL. In the directory where you ran the strace command from, you will find dozens or hundreds of strace log files named stracelog.[PID]. Just open the log file for the PID you logged in Screen 2.

Setting up crons for troubleshooting problem sites

A cron you can setup to help log in this case a slow loading site.

lf=/var/log/sitelatency
s1=linuxblog.info
s11=linuxblog
td=`date`

echo "date $td" >> $lf/sp_$s11

curl -o /dev/null -s -w %{time_total}\\n $s1 >> $lf/sp_$s11 2>&1

and then to grep for large occurrences try from /var/log/sitelatency
cat * | grep -B1 "^[1-3][0-9]"

More reading

Good examples and more general reading http://www.slashroot.in/curl-command-tutorial-linux-example-usage

To stress test a website see http://servermonitoringhq.com/blog/how_to_quickly_stress_test_a_web_server

Using curl for testing web applications
https://brainflush.wordpress.com/2008/03/18/using-curl-for-testing-web-applications/

example of troubleshooting a slow site http://endlessgeek.com/2014/03/strace-php-process-via-apache/