nmon for Linux Screen Shots
On this webpage you can find
- Screen shots of using nmon for Linux online
- Sample graphs from the nmon Analyser after it generated them from nmon for Linux capture to a file
- Some other screen shots from a older nmon for Linux version
New features in the latest nmon for Linux 16b
- Here are the ten Utilisation categories of a model Linux version - user "U" to show them on screen
- The stats are also saved to the nmon files if you add the -U option
- Note nice is user time but at a lower priority (use the nice command).
- Note guest and guest_nice are for KVM host environments and show the guest OSes CPU use.
- Note steal is for a gust OS to show if the CPU is being used to run other Virtual machines.
- Note the irq and softirq are CPU time used for interrupts - I have seen some embedded CPU doing most of the CPU work at interrupt level.
- Small improvements in the Kernel view too
nmon for Linux Online screen shots
This is the start up screen when using nmon online
- There are plenty of hints on how to switch on/off (toggle) the various stats and text graphs
- The only one you have to remember if "q" to QUIT
This is the Help section
- If you hit "h" you get this Help section with loads of hints. Hit "h" again to remove the Help
- If you want more Help then stop nmon and run "nmon -h | less"
- Alternatively, read the same output with your brlower using the left-hand Menu "nmon -h" button or
Click: nmon -h
- Hit "r" for Resources to find out what nmon can discover about your processors and Linux OS version and other features
- This is a bit of a weak area in Linux and vendors do this differently or not at all.
- In this case we have Intel processor running Ubuntu 15.04 running on four 3.3 GHz
POWER LPAR details
- This is Linux on Power running in a Logical Partition (LPAR) for this platform only we have LPAR details (hit "p")
- We can see the LPAR is only consuming 1.667 Physical CPU cores out of a machine with 20 Cores with 1 cores and two virtual CPU cores in this LPAR
- If you are running on x86 this information is not available and "p" is ignored (actually, not compiled in to the nmon binary).
CPU Stats - Utilisation
- Hit "c" for the CPU stats.
- Here we see the four CPUs
- I ran a load generator (nstress) to get the graphs active
- "U" is user time, "s" is System time and "W" id waiting for Disk I/O time.
- Obviously blank is used for idle time.
- Note the ">" marks are the peak CPU usage - if you hit "0" (zero) these are reset to the far left.
CPU Long Term Graph
- Hit "l" (lowercase L) to see these. "l" for long term
- On peaky workloads like a web server you can drive yourself mad watching the above graph fluctuate up and down and guessing an average
- This graph lets you see the last 70 snapshots and moves to the right. Once full it will wrap around.
- Note the top left says 2 seconds
- Faster and Slower screen Updates - this works regardless of the stats you are watching:
- Hit "-" to halve the time nmon pauses between updates of the screen i.e. if it is set to 2 seconds now you get updates once a second
- Hit"+" to double the time nmon pauses between updates of the screen i.e. if it is set to 2 seconds now you get updates once in 4 seconds
- You can't go below 1 second - nmon would take too much CPU time for that to make sense
- Faster is good to watch for peaks
- Slower is good to even out the peaks to see a stable average
- Also note the "+" on the moving vertical bar - this shows the average number for the graphed data.
- Below I hit a "6" to get a 60% horizontal line drawn
CPU Wide or concise View
- Hit "C" (uppercase c) to see these.
- With lots of CPU cores and lots of CPU core threads (Power Simultaneous Multiple Threads or x86 Hyper-threads sometimes called logical CPUs) we have trouble showing them all on one screen.
- This Concise view shows smaller graphs across the screen (rather than down the screen) and up to 192 CPU core threads at a time.
CPU cores with variable MHz
- Hit "M" (uppercase m) to see these.
- Some machines like the POWER S812LC and S822LC plus new in 2015 Intel PC's (so I am told - if you have one please let me know) can run different CPU cores at different MHz frequencies. Normally, all the CPU cores run as one MHz frequencies (native whole machine or at a VM level) even if they can all rise and fall the MHz together.
- The stats are taken from /proc/cpuinfo and are point in time - i.e. not averages of time to there are either the lower CPU Idle frequency of the higher CPU busy and over clocking frequency and never in between.
- If you hit "M" three times you cycle through three displays of this feature and the 4th one toggles this off.
- 1st "M" = the CPU core threads MHz
- 2st "M" the CPU cores MHz
- 3st "M" the CPU cores MHz graphs
- Hit "m" for memory details (lowercase M)
- Note: this is a Power processor based system - there is no high memory as the Power processor has been 64 bit for a decade or more and these bizarre workaround to extend memory are not required.
- We can see 3102 MB of free RAM out of 4038 MB
- And the paging space has not been used, yet!
Virtual Memory Stats
- Hit "V" for memory details (uppercase v)
- Note: These are largely Linux Kernel details
- Hit "d" for disk graphs
- This is, of course, a very small machine with just the one disk.
- If we had lots more disk we may not be able to see them all.
- Try making the font smaller and/or the window larger to show more lines
- The graph shows "W" for Writing and "R" for Reading I/O
- Note: the disk like sda and the partition on the disks sda1 both report the same disk I/O = this is double counting the I/O - a kernel bug IMHO but we have a work around this using User Defined Disk Groups or now starting with the "-g auto" feature to automatically define the disks as disk groups (up to 64 disks).
- If you hit "." (dot) then only the busy disks are shown - this works on Top Processes too.
Disk Stats by more numbers
- Hit "D" for the Disks but this time the Numbers in more Detail.
- We can see the numbers of transfers and block sizes etc.
- Also we see the peak values - these can be reset to zero by hitting "0" (zero)
Disk Rain Diagram
- Hit "o" (lowercase Oh!)
- With just one disk this looks stupid (see the single "_" but imagine you had 640.
- The panel would have 10 lines and one character per disk
- As each disk gets busier more pixels are added to its character.
- This gives you a quick idea of how busy your disks are, as a whole.
- You also see unused disks or ranges of disks that are particularly busy.
- Hit "j" for (Journalled) File systems - OK "f" would make more sense but that was used for something else in the past!
- Here we see the root filesystem is 16% full
- Note the increasing trend to large numbers of fake file systems!!
- Hit "n"
- All pretty obvious (I hope)
Network Stats - without error numbers
- If you wait a few seconds and there are no errors the lower panel is switched off.
- It will reappear if network error happen.
- On some machines with a dozen networks this saves a lot of screen space
Network File System (NFS)
- Hit "N"
- OK this machine was not mounted or mounting NFS so the numbers are all zero.
- nmon 16a onwards just displays a warning the NFS is not running and covers NFS v2, v3 and v4
- NOTE: NEED TO GRAB AN UP TO DATE IMAGE OF NFS STATS
- Hit "k"
- Linux does not have so may of these but we have the important once like Run Queue, Forks, Load average are not that useful as nmon already collects the detailed CPU stats and we have Uptime here too.
Top Process Stats
- Hit "t" to see the processes
- Default mode is 3 and this arranges them in CPU use order.
- You can see the big hitters are processes called "yes" (I also use "ncpu" from the nstress tools in some screen shots) from the "%CPU Used"
- Note: 100% means using a whole single CPU core thread. If the process has more that 100% then it is obviously multi-threaded.
- "Res" columns are the Resident Set i.e. the parts of the program and its data that is currently in memory.
- "Res Set" is the total. "Res Text" is the program code. "Res Data" is the variable data. "Res Lib" is the libraries it is using but note these are probably shared between many running program.
Top Processes in mode 3 = User Commands
- (Hit "t") then hit "u" for User command rather than process names - this usually shows yu the command line options used
Top Processes in mode 1 = more details
- Have the top Processes on screen (Hit "t") then hit "1" for mode 1
- There are different stats shown including status, nice and priority
Top Processes in mode 4 = size order
- Have the top Processes on screen (Hit "t") then hit "4" for mode 4
- Here the processes are in memory size order
- You can see the largest process in 10432 MB and called "ssh" - Normally "Java" is the biggest!
Top Processes in mode 5 = I/O order
- Have the top Processes on screen (Hit "t") then hit "5" for mode 5
- Here the processes are in I/O in KB per second order
- Note: You must be logged in as root for this to work due to odd /proc filesystem read/write permissions = Kernel bug IMHO
Graphical CPU Accelerator stats
- Use "a" to see these
- Of course, you need to have a NVIDIA GPU adapter or two installed, a special build nmon version and the nvidia ML library available
Examples of nmonchart graphs
- For the complete set of nmonchart Graphs take the left-hand menu link or Click: nmonchart
Sample Graphs from the nmon Analyser
CPU with Disk I/O to show the relationship between them
Individual CPU use on this 6 CPU Logical partition
CPU compared with Shared CPU Entitlement (Power Systems only)
Disk read and Write with IO/second
Hot Disks in order with Average, Weighted Average (average when in use) and Peak values
Network Read (top) and Write (bottom) - Iceberg view
Top Processes - in this case DB2
Process switches - kernel switching to a new process
Pivot table covering Top Processes over time
Older nmon version Screen Shots