This post was originally a quick email to a client and their other partners detailing what the output from the "top" tool means, which is a utility which shows a list of running processes on a UNIX system along with statistics on their memory and CPU consumption.
Here's an example of what top displays:
I'll dismantle the lines one by one, from the 'top'. I'm sure a lot of this is already known by you all or be obvious, but it may be worth exploring it all just in case, as there's a lot of confusion about some of the data here.
Line 1: time, uptime, logged in users, load averages
- Current local time
- Time system the system booted
- Logged in users: Number of interactive sessions (console, ssh, etc)
- Load averages: There are three load average numbers, and these are the average "load" of the system over 1 minute, 5 minutes, and 15 minutes. Load is calculated by adding 1 for any process or thread that is one of:
- Runnable if CPU resources available (R-state)
- Waiting on IO (D-state)
In the above image, the instantaneous load is exactly 1 (there is one process, top itself, running), but this number is not useful to know. This is why the numbers are averaged over time.
Line 2: Task statistics
- Total number of tasks (or processes).
- Running tasks (or more accurately, runnable; as you can only run as many tasks simultaneously as you have hardware threads).
- Sleeping tasks which are not currently requesting any CPU or IO resources at all (perhaps waiting on a socket, for example.)
- Stopped tasks, which have been sent SIGSLEEP to temporary remove them from the scheduler's task list.
- Zombie tasks. These have died (naturally or otherwise), but their parent process is yet to call wait() on them to collect their exit status.
Line 3: CPU usage distribution
Total userland usage, kernel/system usage, "nice" process usage, idle time, IO wait, hardware interrupt time, software interrupt time, stolen time.
The important thing to remember is that this is all "wallclock time", time as a clock on the wall might measure it, not CPU time, and they don't always add up. They are all expressed as percentages of available CPU time used since top last refreshed.
- Total userland usage ('us'): The amount of available CPU time used by userland processes.
- Kernel/system usage ('sy'): The amount of available CPU time used by Linux itself.
- "Nice" process usage: The amount of available CPU time used by low-priority processes (manipulated by the 'nice' command).
- Idle time ('id'): The amount of available CPU time spent with nothing to do.
- IO wait time ('wa'): The amount of available CPU time spent waiting for IO to complete.
- Hardware interrupt time ('hi'): The amount of available CPU time used servicing hardware interrupts.
- Software interrupt time ('si'): The amount of available CPU time used servicing software interrupts. (System calls)
- Stolen time ('st'): Estimate of time stolen by the Hypervisor's overhead when running the system in a VM. This should always be zero on bare-metal systems, and not all hypervisors report it to their guests, meaning Linux cannot know the available CPU time is varying.
Line 4: Hardware memory summary
Total hardware memory, total used by processes and kernel, unused memory, kernel buffers.
- Total hardware memory is what is wired to the CPU and not reserved for other parts of the system. In this case, 4GB minus the memory used by PCI and the GPU.
- Entirely inused memory is free.
- Total used by processes is what is consumed in userland, ie applications.
- Buffers and cache are essentially memory used by Linux itself for managing IO and housekeeping. This is physical memory that has been used to cache the contents of block devices (eMMC, SSD, USB sticks, etc). It is not "used" in a traditional sense; the instant a userland process requires more RAM and there is no unused RAM, this RAM is instantly raided to satisfy the request.
Line 5: Swap memory summary
Total swap, free swap, used swap, total available memory. Swap memory is block-device (such as a hard drive, or a compressed RAM disc) memory used to page out data from real memory that Linux believes not to be in use, freeing real memory for use by applications and Linux for caching.
Now comes a table of processes and statistics. The columns are as follows:
- PID: Process identifier. Typically a 16 bit number, 0 is forbidden, 1 is 'init': It is responsible for booting the system, starting and monitoring services, and is the parent of last resort that adopts processes whose parent has died.
- USER: The username or number the process is running as.
- PR: The process's priority. This normally runs from -20 to 20, where the lower the number, the higher the priority. There is also an 'RT' real-time priority which trumps all. It is not "real time", in the embedded sense.
- NI: Niceness. This is an offset to apply to the normal priority. It is normally zero.
- VIRT: Virtual memory size. This is the size of the process's address space. This doesn't mean it is all memory used, however: some may be maps of files on disc, or may be yet unused and not had real RAM allocated to back it.
- RES: Resident size. This is how much RAM the process actually has allocated to it. This may be backed by physical RAM or by swap.
- SHR: Sharable memory. This is the amount of memory being used that could possibly be or might even be shared with other processes. This includes shared memory used for IPC, as well as memory maps of files on disc (such as the executable itself, shared libraries, etc.)
- S: State. S = sleeping, R = runnable, D = blocked waiting for IO, Z = waiting for parent process to collect exit status.
- CPU: Percentage of CPU used by this process since the last refresh. Note that 100% is 100% of one thread. In a quad-core system, this could reach the high 300s.
- MEM: Percentage of available RAM (combined physical and swap) that this process is using, as a factor of its RES, described above.
- TIME: CPU seconds used by this process since it started. What this means is that if it were to use a whole CPU thread for a second (or 50% of a thread for two seconds), 1 will be added.
- COMMAND: Title of process.
Note: This document is licensed under CC-BY-SA and was originally created by Codethink Ltd.
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: exploring a bug in stack unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Full archive