Commands To Help Me Understand CPU Usage in Linux

Not sure if this helps;

It appears there many tools available, some are paid for that have trial periods such as: DataDog
https://www.datadoghq.com/dg/monitor/linux-monitoring/?utm_source=Advertisement&utm_medium=GoogleAds&utm_campaign=GoogleAds-Linux&utm_content=Infra&utm_keyword=linux%20cpu%20usage&utm_matchtype=p&gclid=Cj0KCQjwwb3rBRDrARIsALR3XeayfVRDwdnYMVKy-BF-25mj45Xs3IUk_I3czOf4xGEueS0-k-A-erMaAmsaEALw_wcB

An interesting set of commands listed here:
https://www.cyberciti.biz/tips/how-do-i-find-out-linux-cpu-utilization.html

top

Question 0: I'm noticing that at certain times of the day, there is more CPU utilization that others.

Almost surely because of different traffic levels at different times of days.

Just look at your Apache + FPM PHP logs to determine the exact reason.

Question 1: What is the best way to determine those spikes?

Use SAR to record CPU usage or use top in batch mode to periodically record usage.

The other simple trick is to ignore CPU time (as this is fairly meaningless) + focus on log data. As logged requests increase, you know CPU is increasing.

Tip: CPU usage is fairly meaningless. I sometime see loads of 100s of load units on some high traffic sites.

What is meaningful is length of time required to process a request.

The only real site bottlenecks are either Disk I/O or Network I/O. CPU usage can be massively high + so long as requests complete near instantaneously, you'll be good.

Question 2: Is there a way to manipulate the "top" command so that I can see a measurable aggregate of processes running on the server? If so, can I adjust this for, say, 24 hours or 48 hours?

With top, no.

With SAR, yes.

Question 3: Could traffic or bots be causing a CPU spike?

This is the only likely reason.

Question 4: Is there an easy way to tell how many users are on a website at a given moment?

Yes.

Google Analytics is one easy way to see real time analytics.

Thanks everyone! I've installed sysstat on the server and have been playing around with it.

@DavidFavor -> How could I use the "sar" command to see historical data, say, from 8/27 to today? Is there a way see that?

Thanks!

sar is part of sysstat.  sysstat is a good toolkit for this.

Do you currently crunch, process your Apache logs,
From this you could get (extrapolate)  a fuller picture of what is going on on your system,
Based on your knowledge of what ...
You could look to correlate CPU usage and increase in traffic/requests.

Thanks David - Yes, I was able to create a CRON job based off instructions I found online.

The job I setup was for every 10 minutes for sa1:

*/10 * * * *

Select allOpen in new window

And at 7 minutes before the end of the day for sa2:

53 23 * * * 

Select allOpen in new window

Now that I have this data, what's a good sar command to be able to see a given day and time. Let's say I wanted to see yesterday (9/4/19), after I created this CRON job, which was at 5:00 PM Pacific... so, how could I see info for 9/4/19 between 19:00 and 21:00?

Thanks for your help.