From nmon for Linux

Site: NmonFAQ

nmon for Linux and AIX Frequently Asked Questions (FAQ)


This is a work in progress and may never be finished - Last update Jan 2017.

This website is about nmon for Linux but many users also use nmon for AIX so both are covered in this FAQ.


Frequently Asked Question may be Answered by a Quick Video


Prefer to watch a YouTube Video from the nmon designer / developer?

This is Nigel Griffiths' YouTube Channel - for lots of videos on nmon, POWER Chips, Power Systems servers, Performance, AIX, PowerVM, PowerVC, PowerSC, Linux on Power.

nmon on AIX on Power

nmon for Linux on Power, x86/AMD64, mainframe, ARM


Frequently Asked Question


Colour key:

Summary of the questions

  1. Which nmon version am I running?
  2. Which nmon for my version of AIX or Linux?
  3. nmon crash shortly after starting a data capture please send me the next version?
  4. Significant nmon dates?
  5. All I get is "nmon not found"?
  6. What is the most reported errors for nmon?
  7. Can I decide the filename it saves data too?
  8. What is the default output filename?
  9. I want nmon output piped into a further command, how?
  10. Why do you support all these old unsupported AIX versions?
  11. What if I want support?
  12. Why don't you add a Java front end to nmon and get graphical output?
  13. The command line options don't seem to work right for file capture?
  14. What is paging to a filesystem?
  15. Where can I get nmon and further information?
  16. TOP process stats get switched on when I request Asynchronous I/O stats?
  17. nmon2rrd fails, please fix it?
  18. What are NANQ and INF?
  19. nmon reports more than 100% for a process - clearly it is wrong?
  20. On AIX the disk adapters are wrong?
  21. On AIX the adapter busy goes over 100%. That is impossible surely?
  22. What about nmon for HP/UX, Solaris on Sparc or x86 or Linux on Itanium?
  23. What about nmon for Windows?
  24. Seeing double the number of CPUs?
  25. Hello, I am new to UNIX and want to tune AIX, what do you recommend?
  26. CPU wait is too high, how can I reduce it?
  27. On AIX, free memory is near zero, how do I free more memory?
  28. How can I set numperm better?
  29. What format is the nmon output file?
  30. I have collected once a second for 8 hours but I can't get the Analyser to work?
  31. nmon does not work on my Linux machine!!
  32. When do we get nmon 10 for Linux?
  33. The boxes and lines in nmon do not work right online with: DTterm, xterm, rvxt, putty, VNC, (whatever you have)?
  34. I have 2400 disk (small SAN LUNs) and nmon is slow to collect the stats from so many, can you help?
  35. What is CharIO (a column of the TOP processes stats)?
  36. On Linux the disk stats are all doubled?
  37. On AIX the disk seem to be mostly on the first adapter?
  38. On nmon for Linux the CPU Wait for IO number is zero or odd?
  39. On nmon for Linux the paging details are missing and the PAGE lines for the capture to file are missing.
  40. I want to collect data every second and then see weekly and monthly reports. How?
  41. How do I work out the Physical CPU use on Linux on POWER for shared processor LPARs?
  42. Automatic starting with certain statistics for nmon online mode?
  43. Sort order problems with massive nmon output files?
  44. Does nmon capture point in time stats or averages?
  45. When will nmon collect data from lots of machines or LPARs?
  46. When will nmon collect data like "topas -C"?
  47. nmon will no stay running - What should I check?
  48. Why isn't nmon for Linux on the Distro media or online repository or it is there but out of date?
  49. Do you have nmon presentations I could use for training others?
  50. nmon Analyser: What is Wavg?
  51. LPAR Tab/Statistics missing with Dedicated CPU mode?
  52. Adding External Data Collectors to nmon files so it graphs your extra data ?
  53. Sharing nmon files - Are they a security risk?
  54. The Disk stats are far too high or 100%, nmon is broken?
  55. What files does nmon for Linux use to get its data?
  56. Can you add the monitoring tape drive on AIX?
  57. How to user an External Data Collector with nmon?
  58. How to RDBMS Oracle Transaction Counters External Data Collectors Example?
  59. How to use the AIX Workload Manager stats?
  60. How to use change the Top Processes Minimum CPU Threshold?
  61. How to start nmon file collection with cron?
  62. Can I reset the peak counters for disks, network, AIO (AIX only) and CPU graphs online?
  63. Is sharing nmon data capture file a possible security risk?
  64. How to determine optimal memory size for a VM from nmon data?
  65. Please explain the TOP Process Memory stats?
  66. What are User Defined Disk Groups for?
  67. Using User Defined Disk Groups with nmon for AIX?
  68. Using User Defined Disk Groups with nmon for Linux?
  69. How do I get more disk stats because I can never get enough of these?
  70. How to limit top processes to certain commands?
  71. How does nmon for AIX extract its data?
  72. How can I see 100's of disks on-screen?
  73. On-screen displaying only busy Top Processes and Hot disks?
  74. Do not use kill -9 on nmon as kill -USR2 will end it cleanly!

- To do:

  1. nmon for Linux 16 - major user interface upgrade with pictures:nmon for Linux v16 - New Stats On screen & Face lift
  2. Got a suggestion ?

- Historic questions


Question 1: Which nmon version am I running?

Question 2: Which nmon for my version of AIX or Linux?

Question 3: nmon crashes shortly after starting a data capture, please fix this send me the next version?

Question 4: Significant nmon dates?

Question 5: All I get is "nmon not found"?

Question 6: What is the most reported errors for nmon?

  1. nmon crashes as it starts in collecting to a file mode.
    • See question 2.
  2. nmon Analyser does not work
    • Quite often the nmon output file is empty or only has the config info due to not waiting long enough - if you request data every 5 minutes then wait 16 minutes (three snapshots of performance data) before you try analysing the file!
    • Incomplete last line. if nmon is still running and outputting data and you can grab the file it is possible to have an incomplete last line of the file - you could edit with vi to remove the last set of output - see the lines starting ZZZ.
  3. Can we have a new feature XYZ?
    • But it turns out XYZ is already implemented (and has been for a few years)
    • So read the nmon -h output and you might find it
    • See below for External Data Providers (question 57) and User Defined Disk Groups (questions 66 to 68)
  4. I have a problem with the nmon options
    • Turns out the user can't read the nmon -h output which states: The -f or -F MUST be the first option on the line
  5. How do I interpret nmon output?
    • First do your home work by learn UNIX and Linux performance statistics: read the command manuals, take a course or spend 5 years in a benchmark centre.
    • Sorry but I can't write nmon and teach the world the basics on UNIX/Linux performance tuning.
  6. The AIX and Linux memory stats are different or missing?
    • The answer is "Yes you are correct". Some of the basic memory stats map OK between AIX and Linux for example: memory total size and memory free but the bulk are very different.
    • Also note that early Linux on Intel/AMD had to cope with small memory size with high and low memory areas due to 16 bit then 32 bit hardware. This has died out now with the move to 64 bit memory addressing.
    • Linux and AIX are very different in the memory area and it not me forgetting to implement some of the stats.
    • For example, the AIX NEWMEM starts are not available under Linux and never will be.
  7. What is causing AIX to run at 99% memory used?
    • This is perfectly normal and show AIX is making use of memory to optimise performance. This is normal and it is a "good thing".

Question 7: Can I decide the filename nmon saves data too?

Question 8: What is the default output filename?

Question 9: I want nmon output piped into a further command, how?

    mkfifo /tmp/xyz
    nmon -F /tmp/xyz s 5 c 300
    your-command </tmp/xyz

Question 10: Why do you support all these old unsupported AIX versions?

Question 11: What if I want support for nmon?

If it is something fairly simple you could ask a question on the IBM Performance Tools Forum (if you have or get a IBM DeveloperWorks user account = not hard): IBM Performance Tools Forum -

  1. How to report an nmon problem well?

Then you get your question answered sooner.

Question 12: Why don't you add a Java front end to nmon and get graphical output?

Question 13: The command line options don't seem to work right for file capture?

Question 14: What is paging to a filesystem (rather than to paging space)?

Question 15: Where can I get nmon and further information?

Question 16: TOP process stats get switched on when I request AIX Asynchronous I/O stats?

Question 17: nmon2rrd fails, please fix it?

Question 18: What are NANQ and INF?

Question 19: nmon reports more than 100% for a process - clearly it is wrong?

Question 20: On AIX the disk adapters are wrong?

Question 21: On AIX the adapter busy goes over 100%. That is impossible surely?

Question 22: What about nmon for HP/UX, Solaris on Sparc or x86 or Linux on Itanium?

Question 23: What about nmon for Windows?

Question 24: Seeing double the number of CPUs on my POWER server?

Question 25: Hello, I am new to UNIX and want to tune AIX, what do you recommend?

Question 26: CPU wait is too high, how can I reduce it?

Question 27: On AIX, free memory is near zero, how do I free more memory?

Question 28: How can I set numperm better?

Question 29: What format is the nmon output file?

Question 30: I have collected once a second for 8 hours but I can't get the Analyser to work?

Question 31: nmon does not work on my Linux machine!

Question 32: When do we get nmon for AIX version X for Linux?

Question 33: The boxes and lines in nmon do not work right online with: DTterm, xterm, rvxt, putty, VNC, (whatever you have)?

Question 34: I have 2400 disks (or 2400 small SAN LUNs) and nmon is slow to collect the stats from so many, can you help?

Question 35: What is CharIO (a column of the TOP processes stats)?

Question 36: On Linux the disk stats are all doubled?

Question 37: On AIX the disk seem to be mostly on the first adapter?

Question 38: On nmon for Linux the CPU Wait for IO number is zero or odd?

Question 39: nmon for Linux has paging details missing and the PAGE lines for the capture to file are missing.

Question 40: I want to collect data every second and then see weekly and monthly reports. How?

Question 41: How do I work out the Physical CPU use on Linux on POWER for shared processor LPARs?

Linux

Question 42: Automatic starting with certain statistics for nmon online mode?

Question 43: Sort order problems with massive nmon output files?

Question 44: Does nmon capture point in time stats or averages?

Question 45: When will nmon collect data from lots of machines or LPARs?

Question 46: When will nmon collect data like the AIX "topas -C"?

Question 47: nmon will not stay running - What should I check?

First the regular house keeping:

Still got a problem? Get some help

Question 48: Why isn't nmon for Linux on the Distro media or online repository or it is there but out of date?

Question 49: Do you have nmon presentations I could use for training others?

Roughly 45 minutes plus either of the two popular graphing tools: nmonchart (browser graphs) or nmon analyser (Excel) - which both work doe Linux and AIX files.

Question 50: nmon Analyser: What is Wavg?

Question 51: LPAR Tab/Statistics missing with Dedicated CPU mode?

Question 52: Adding External Data Collectors to nmon files so it graphs your extra data?

Question 53: Sharing nmon files - Are they a security risk?

Question 54: The Disk stats are far too high or 100%, nmon is broken?

Question 55: What files does nmon for Linux use to get its data?

  1. Performance stats
    • /proc/cpuinfo
    • /proc/stat
    • /proc/version
    • /proc/meminfo
    • /proc/uptime
    • /proc/loadavg
    • /proc/net/rpc/nfs
    • /proc/net/rpc/nfsd
    • /proc/vmstat
    • /proc/ppc64/lparcfg - POWER systems only
    • /proc/net/rpc/nfs
    • /proc/net/rpc/nfsd
    • /proc/diskstats
    • /proc/partitions
    • /proc/net/dev
  2. Process stats where PID is replaced with the Process ID number in turn
    • /proc/PID/stat
    • /proc/PID/statm
    • /proc/PID/io
  3. Configuration data - includes the above in full text and then these too
    • /proc/device-tree/host-model
    • /proc/device-tree/host-serial
    • /proc/device-tree/ibm,partition-name
    • /proc/diskinfo
    • /proc/sysinfo
    • /proc/modules
  4. Some extra data it extracted using classic UNIX system calls like those to detail the file systems and mount points

Question 56: Can you add the monitoring tape drive on AIX?

Question 57: How to use External Data Collectors with nmon?

Any other nmon user wants to be able to track the username of processes that are using a lot of CPU time. This is the approach recommended

Question 58: How to RDBMS Oracle Transaction Counters External Data Collectors Example?

Question 59: How to use the AIX Workload Manager Statistics?

Question 60: How to use change the Top Processes Minimum CPU Threshold?

Question 61: How to start nmon file collection with cron?

Question 62: Can I reset the peak counters for disks, network, AIO (AIX only) and CPU graphs online?

Question 63: How do I use User Defined Disk Groups to monitor large numbers of disks in ESS disk ranks?

Question 63: Is sharing nmon data capture file a possible security risk?

Question 64: How to determine optimal memory size for a VM from nmon data?

Question 65: Please explain the TOP Process Memory stats?

Before answering I am going to assume you are aware there is no single number the tells you everything about the memory of a processes. This is because of many complications like programs share program code memory (one read-only copy for all processes running the same program) and partially share data (on a fork() the memory is shared with a Copy-On-Write flag to make different copies only if a page is written too) and then some of the program can be paged to/from disk or paged from file systems and some not exist in memory unless its updated (static data in the program file).

TOP process stats (switched on with - tot -T) have a header line describing the columns like this for Linux

and like this for AIX

Size, ResSet, ResText, ResData are the Memory stats

If you want one number for the memory size of a process then use (ResText + ResData) but note some of that memory is shared between processes.

nmonchart in it's TOP Process bubble chart reports the maximum value found in all the memory sizes reported for a particular process i.e ResText + ResData.

Question 66: What are User Defined Disk Groups for?

Here are a few good use cases for this nmon feature that is covered in more details in the following three questions:

  1. Servers with 100's or 1000's of disks are very difficult to monitor on screen.
    • Unless you have a screen that can display 100's of lines!
    • You have reduce the disks on-screen graphs to just tiny font size and use a modern HD screen but there are limits.
  2. Servers with 100's or 1000's of disks are very difficult to graph later.
    • One extreme case with 4000 disks produced a black oblong because there was so many lines.
    • They complained that they could not see the details so the disks were unmanageable. They are correct - the problem was their default LUN size on the Fibre-Channel disks was ridiculously small but this is not nmon's fault. It was set-in-stone, out of date systems management practices.
  3. With many disks with the same data it is useful to group the disks together and then see the total I/O to that group of disks.
    • For example: the disks that make up a RDBMS data, RDBMS index and RDBMS logs - each should have different I/O characteristics in RW ratio, and block sizes.
    • For example: the disks used for backup, batch processing or background tasks like data arriving to be loaded in to a database - will be busy at different times.
  4. On AIX hdiskN and on Linux sdX are not helpful names while monitoring - changing the name to something meaningful aids comprehension
    • For example: rootvg, paging, webpages or rdbms_log immediately lets you know the data on the disk(s).

This feature is covered in the nmon -h output as follows

Question 67: Using User Defined Disk Groups with nmon for AIX?

Then when you start nmon add at the end:

Question 68: Using User Defined Disk Groups with nmon for Linux?

Question 69: How do I get more disk stats because I can never get enough of these?

nmon for Linux

I hope the names are clear enough for you to understand the meaning.

nmon for AIX

Question 70: How to limit top processes to certain commands?

Question 71: How does nmon for AIX extract its data?

Question 72: How can I see 100's of disks on-screen?

Question 73: On-screen displaying only busy Top Processes and Hot disks?

Question 74: Do not use kill -9 on nmon as kill -USR2 will end it cleanly!


      - - - F r e q u e n t l y  -  A s k e d  -  Q u e s t i o n s  -  E n d  - - -

Retrieved from http://nmon.sourceforge.net/pmwiki.php?n=Site.NmonFAQ
Page last modified on January 03, 2017, at 04:11 PM