Performance Co-Pilot on Rocky and Alma

Performance Co-Pilot is already on your Rocky or Alma system

Performance Co-Pilot (PCP) sits in the AppStream repository of Rocky Linux and AlmaLinux, so a full metrics collector with on-disk history is one dnf command and two systemctl commands away. There is no vendor agent to add and no port to open toward a central server. This guide was written against Rocky Linux 9 and AlmaLinux 9 with the AppStream build of PCP 6.3, checked in August 2026, and every command below is one you run yourself on your own server.

The reason so few people use it is the naming. The commands are short abbreviations that hide what they do. Learn what the four pieces are called and the rest of the stack becomes discoverable from the command line.

Rocky and Alma ship the same PCP packages at the same versions, because both rebuild the same source. If you are still choosing between them, that is covered in the comparison of Rocky Linux and AlmaLinux. The same stack is packaged as pcp on Debian and Ubuntu with identical command names, so only the install line changes: see the dnf and apt command equivalents for that translation.

How Performance Co-Pilot is put together

PCP is four moving parts, and each one does a single job.

• pmcd is the performance metrics collector daemon. It listens on TCP port 44321 and answers one question: what is the value of this metric right now. It keeps no history.
• A PMDA (performance metrics domain agent) is a plugin that feeds pmcd. Each agent owns one domain of metrics. The linux agent covers the kernel, so CPU, memory, disk and network numbers all come from it. Other agents cover the systemd journal, NGINX, PostgreSQL, SMART data from drives, or any exporter that speaks OpenMetrics. The agents pmcd loads are listed in /etc/pcp/pmcd/pmcd.conf, and each agent's install script edits that file for you.
• pmlogger samples metrics on a fixed interval and writes them into an archive on disk. This is what turns a live reading into history you can replay later.
• pmie is the performance metrics inference engine. It evaluates expressions against live metrics and runs an action when one of them becomes true.

Metric names are dotted paths in one namespace, such as kernel.all.load or disk.dev.read. A metric can have instances: disk.dev.read carries one value per block device, and filesys.full one value per mounted filesystem. That idea comes back later, because both the reporting tools and the pmie rules work per instance.

Because pmcd stores nothing, a query always tells you about now. Anything you want to look at tomorrow has to be inside a pmlogger archive today. That split explains most of the configuration in this guide.

Install and start Performance Co-Pilot

sudo dnf install -y pcp pcp-system-tools
sudo systemctl enable --now pmcd pmlogger
systemctl is-active pmcd pmlogger

is-active should print active twice. Installing the pcp package does not start anything, so skipping the enable --now line leaves you with tools that cannot reach a collector. Now confirm the namespace answers:

pminfo | wc -l
pcp

pminfo with no arguments prints every metric name pmcd knows, which is a few thousand lines on a stock install. pcp prints a one screen summary of the host. An error that mentions Cannot connect to PMCD means pmcd is not answering, so read journalctl -u pmcd -n 30 and /var/log/pcp/pmcd/pmcd.log.

The second package, pcp-system-tools, brings the reporting commands that read those same metrics: pmrep, pcp-atop, pcp-iostat and pcp-dstat.

There is also pcp-zeroconf. Its own description calls it configuration tweaks to increase metrics gathering frequency, plus extended pmlogger configurations and automated pmie alerting for the local host. Its post-install script enables and restarts pmcd, pmlogger and pmie for you. Note what it costs: the packaged configuration samples every 10 seconds where pmlogger's own default is 60 seconds, so archives grow several times faster. On a small plan, read the retention section before you install it.

Which metrics does my VPS actually expose?

pminfo is the discovery tool. Give it a subtree and it lists the leaves. Give it a flag and it tells you what a metric means or what it currently reads.

pminfo disk
pminfo -t kernel.all.cpu
pminfo -dfmtT disk.all.read_bytes
pminfo -f kernel.all.load
pminfo -f kernel.all.cpu.steal

-t prints the one line help for each metric, -T the long help, -d the descriptor, -m the internal metric id, and -f fetches current values for every instance. The descriptor is the part people skip and then misread their own graphs. It gives the data type, the units, and the semantics: whether the value is an instant reading, a discrete count, or a counter that only ever grows.

That last case matters. disk.all.read is a counter of operations since boot, so a single fetch tells you nothing about the last minute. Two fetches and the time between them do. The reporting tools do that subtraction for you, which is why pmrep shows rates by default and needs -r before it will show you the raw counter.

kernel.all.cpu.steal is the metric to know on shared hardware. It counts time your virtual CPU was ready to run while the host gave the physical core to somebody else, so a rising steal figure is not your workload's fault. The mechanism, and what you can do about it, is in CPU steal time and noisy neighbours.

Run pminfo kernel.all.pressure as well. If it lists metrics, your kernel publishes pressure stall information, and kernel.all.pressure.io.full.avg tells you how long every task on the box was blocked waiting for storage. If the name is unknown, your kernel is not exposing /proc/pressure, so use the disk metrics instead.

Watch metrics live with pmrep

pmrep -t 2sec -p kernel.all.load mem.util.available disk.all.read disk.all.write
pmrep -p -b MB :vmstat
pmrep -t 5sec -s 12 -p -o csv kernel.all.cpu.steal filesys.full

-t sets the sample interval, -p adds timestamps, -s stops after that many samples, -b scales byte units, and -o csv writes comma separated output you can keep. A name that starts with a colon, like :vmstat, is a saved metric set from pmrep's own configuration, so you get a familiar column layout without typing metric names.

The same command reads other sources. -h followed by an address fetches from pmcd on another host, and -a followed by an archive path reads recorded history instead of the live daemon. One tool and one syntax for both. Once you are past a couple of servers, that -h form is the cheap way to look at all of them from one place, and the wider approach is in managing multiple Linux servers.

Retention: pmlogger will fill a small disk

Archives live in /var/log/pcp/pmlogger/<hostname>. Each set has a basename of YYYYMMDD.HH.MM with an .index file, a .meta file, and one or more numbered data volumes.

ls -l /var/log/pcp/pmlogger/$(hostname)
du -sh /var/log/pcp/pmlogger/$(hostname)
systemctl list-timers 'pm*'

Two systemd timers do the housekeeping. pmlogger_check.timer restarts any logger that died. pmlogger_daily.timer fires at 00:10 with Persistent=true, so it catches up after a reboot, and it merges the day's archives into one, compresses them, and discards anything past the retention limit. That limit defaults to 14 days.

Measure before you tune. Leave it running for a day, then run du -sh again. Archive size depends on your sample interval and on how many metrics are logged, so a figure published for someone else's server is not yours.

Set a time limit and a size limit in /etc/pcp/pmlogger/control.d/local. Leave the existing control line as it is and add the two assignments above it, because the file is read from the top and an assignment only applies to control lines below it.

$version=1.1
$ PCP_CULLAFTER=3
$ PCP_SPACELIMIT=200M

PCP_CULLAFTER=3 discards archives older than three days, the same thing as passing -k 3 to pmlogger_daily. PCP_SPACELIMIT=200M is the hard stop: when the daily run finds the host's archive directory above that size, it removes archives oldest first until the directory fits, and it never removes today's. Both limits are applied when pmlogger_daily runs, which means a bad day can overshoot until 00:10. Apply the change now rather than waiting for the timer:

sudo systemctl start pmlogger_daily
du -sh /var/log/pcp/pmlogger/$(hostname)

The other lever is logging less. The primary logger's metric list is /var/lib/pcp/config/pmlogger/config.default, because pmlogger resolves a non-absolute -c name under /var/lib/pcp/config/pmlogger. Edit it by logging group with pmlogconf instead of by hand:

sudo pmlogconf -r /var/lib/pcp/config/pmlogger/config.default
sudo systemctl restart pmlogger

It walks the groups and asks about each one, so decline the groups you will never read. Per-process metrics from the proc agent are the expensive group, because they record a set of values for every process on the box.

Replay an incident from an archive

ls /var/log/pcp/pmlogger/$(hostname)
pmlogdump -l /var/log/pcp/pmlogger/$(hostname)/20260816
pmrep -a /var/log/pcp/pmlogger/$(hostname)/20260816 -S '@08:00' -T '@09:00' -t 5min -p :vmstat

Pass the basename with no suffix, so 20260816, not 20260816.0. pmlogdump -l prints the archive label, which tells you the host it came from and the time range it holds. If that command is missing, older PCP releases call it pmdumplog. Then -S and -T cut a window out of the archive and -t sets how often you want a row inside that window.

This is the whole reason to keep archives. An alert tells you something broke at 08:40. The archive tells you that steal time climbed first and disk activity followed, which is a different conclusion from your own service leaking memory. You cannot collect that after the fact.

Does the SMART PMDA work on a VPS virtual disk?

SMART (self-monitoring, analysis and reporting technology) counters live on the physical drive. On a VPS you are a tenant, and your /dev/vda is usually a virtual block device. Whether anything real is visible behind it is the host's decision, not yours. So check what your own server reports before you install an agent to read it.

lsblk -d -o NAME,SIZE,ROTA,TRAN,MODEL
sudo dnf install -y smartmontools
sudo smartctl --scan
sudo smartctl -i /dev/vda
sudo smartctl -H /dev/vda

Read what comes back instead of assuming. If -i prints a device model and serial number, and -H prints a health assessment, the underlying device is exposed and the PCP agent will have something to read. If it reports that the device type cannot be detected, or that SMART support is unavailable, then the host is not passing the device through, and no PCP configuration changes that. Use the device name that --scan reports if your disk is not /dev/vda.

When SMART data is there, add the agent:

sudo dnf install -y pcp-pmda-smart
cd /var/lib/pcp/pmdas/smart
sudo ./Install
pminfo smart
pminfo -f smart.health
pminfo -f smart.info.device_model
pminfo -f smart.attributes.reallocated_sector_count.raw

The ./Install script registers the agent with pmcd and adds the smart subtree to the namespace, so the metrics are queryable straight away. The agent reads its data by running smartctl, and its own errors go to /var/log/pcp/pmcd/smart.log rather than to your terminal. On an NVMe device the useful names sit under smart.nvme_attributes, including percentage_used and media_and_data_integrity_errors. A metric that exists with nothing behind it makes pminfo -f answer No value(s) available!, which means the agent ran and the device gave it nothing.

If your host hides the device, watch behaviour instead of firmware counters: disk.dev.avactive for how busy the virtual disk is, disk.all.read_bytes and disk.all.write_bytes for throughput, and filesys.full for the failure that actually takes servers down. What a tenant can and cannot learn about storage is covered in disk health monitoring on a VPS.

One pmie rule that pages someone

A rule file is a set of expressions with an action attached. Write this to /var/lib/pcp/config/pmie/vps.rules:

// any mounted filesystem over 90 percent full
some_inst (
    filesys.full > 90
) -> syslog "pcp: filesystem %i is over 90 percent full"
   & shell 60 min "/usr/local/bin/pcp-alert";

some_inst makes the expression true when at least one instance satisfies it, and %i in the message expands to the instances that made it true. The 60 min on the shell action is a suppression time: after the action runs, pmie will not run it again for an hour, which is what stops a full disk from paging you at every evaluation. Keep %i out of the shell command, because pmie expands these selectors by repeating the whole argument once per matching instance, and a repeated command line is not what you want. The syslog string may also start with -p and -t, which pmie extracts and passes on the way logger does.

Check the syntax, then watch the rule evaluate in the foreground:

sudo -u pcp pmie -C -c /var/lib/pcp/config/pmie/vps.rules
sudo -u pcp pmie -v -t 10sec -c /var/lib/pcp/config/pmie/vps.rules

-C parses the file, reports any error, and exits without evaluating. -v prints the value of each expression at every sample, so you can see the rule reading real numbers. Lower the threshold to 1 for a moment and it will fire, which is the only way to prove the action works. Press Ctrl+C to stop.

The action script can do whatever you want. This one posts to a push server:

sudo tee /usr/local/bin/pcp-alert >/dev/null <<'EOF'
#!/bin/bash
curl -fsS -H 'Title: PCP alert' \
  -d 'filesystem over 90 percent full, see journalctl -t pmie' \
  https://ntfy.example.com/my-vps-alerts
EOF
sudo chmod 755 /usr/local/bin/pcp-alert

Point it at your own endpoint. Running that receiver yourself is covered in a self-hosted ntfy push server. One detail decides whether any of this works: as a daemon, pmie runs under the unprivileged pcp account, so the shell action runs as pcp. The script has to be executable by that user, any token file it reads has to be readable by that user, and a command that needs root fails. The reason is written to pmie's log file, so that is where to look when the alert never arrives.

Make the rule permanent as its own pmie instance:

sudo install -d -o pcp -g pcp /var/log/pcp/pmie/$(hostname)
sudo tee /etc/pcp/pmie/control.d/vps >/dev/null <<'EOF'
$version=1.1
LOCALHOSTNAME   n   n   PCP_LOG_DIR/pmie/LOCALHOSTNAME/vps.log   -c /var/lib/pcp/config/pmie/vps.rules -t 2min
EOF
sudo chmod 600 /etc/pcp/pmie/control.d/vps
sudo systemctl start pmie_check
sudo tail /var/log/pcp/pmie/$(hostname)/vps.log

The five fields are the host to monitor, whether this is the primary instance, whether it needs pmsocks, the log file, and the arguments handed to pmie. LOCALHOSTNAME is a reserved word that becomes the local pmcd in the first field and your hostname inside the path. The primary field is n on purpose, because pmie.service starts only the primary instance. Any extra instance is started by pmie_check, whose timer runs at 28 and 58 minutes past every hour, so starting it by hand only skips the wait. The control file must be readable by root alone, which is what the chmod 600 is for.

If you would rather not write expressions, pmieconf generates them. sudo pmieconf rules lists the packaged rule groups and shows which are enabled, and the primary pmie reads the file pmieconf maintains at /var/lib/pcp/config/pmie/config.default. That is the machinery pcp-zeroconf switches on. Hand-written rules are for the conditions only you know about, such as the one directory that fills because of one application.

Where Performance Co-Pilot stops

The base install has no web dashboard and no view across several hosts. pmproxy adds a REST API over the same metrics, which is what the Grafana data source for PCP reads, and that is the route to charts. Alert routing with on-call schedules is outside pmie's job as well: pmie runs an action, and the action is your script.

PCP answers what happened on this one box, at high resolution, and it is already installed. A central monitoring server is a different job, with per-service checks, a user interface, and alerting across hosts. a Zabbix monitoring server fills that role, and Uptime Kuma answers the simpler question of whether a service still responds. Nothing conflicts: pmcd is a local daemon on port 44321, and another agent on the same server neither knows nor cares that it is there.

The useful first step is to do nothing for a week. Leave pmcd and pmlogger running, cap the archive directory, and the next time the box slows down at 03:00 you will have numbers from 03:00 instead of a guess.

FAQ

Do I need to install anything to use Performance Co-Pilot on Rocky Linux or AlmaLinux?

The packages are in the AppStream repository, so sudo dnf install -y pcp pcp-system-tools is the whole download step and nothing comes from outside the distribution. Installing pcp does not start it, so run sudo systemctl enable --now pmcd pmlogger next, then pminfo | wc -l to prove the collector answers. The pcp-zeroconf package does the enabling for you and raises the sample rate to 10 seconds, which is worth knowing before you add it to a server with a small disk.

Where does pmlogger store its archives, and how long does it keep them?

In /var/log/pcp/pmlogger/<hostname>, with archive sets named for their date and time. pmlogger_daily.timer runs at 00:10, merges the day, compresses it, and discards archives older than 14 days by default. To keep less, add $ PCP_CULLAFTER=3 to /etc/pcp/pmlogger/control.d/local above the control line, and add $ PCP_SPACELIMIT=200M as a size cap: the daily run then deletes oldest first until the directory fits, and it never deletes today's archive. Check where you stand with du -sh /var/log/pcp/pmlogger/$(hostname).

Why does the SMART PMDA report no values on my VPS?

Because SMART counters come from the physical drive, and your disk is probably a virtual block device. Run sudo smartctl -i /dev/vda before blaming PCP: if smartctl cannot detect the device type, or reports no SMART support, the host is not exposing the underlying hardware and the agent has nothing to read. In that state the metrics still exist in the namespace and pminfo -f smart.health answers No value(s) available!. Errors from the agent itself land in /var/log/pcp/pmcd/smart.log. Watch disk.dev.avactive and filesys.full instead, because those describe the disk you actually have.

Why does my pmie rule never fire?

Work through it in order. pmie -C -c <file> proves the file parses. sudo -u pcp pmie -v -t 10sec -c <file> prints the expression's value at every sample, so you can see whether the condition is ever true, and dropping the threshold to 1 forces it. A suppression time such as 60 min on the action allows one firing per hour and no more. If the rule works in the foreground but not as a service, remember that pmie.service starts only the primary instance while extra control files are started by pmie_check, and that the daemon runs as the unprivileged pcp user, so a shell action needing root fails. The reason appears in the log file named in the control line.

Can I run Performance Co-Pilot alongside Zabbix or Prometheus?

Yes. pmcd is a local daemon on TCP 44321 and it does not touch what another agent collects, so a Zabbix agent or a node exporter keeps working unchanged. PCP can also consume from the other direction: the OpenMetrics agent pulls an existing exporter's numbers into the PCP namespace, and pmproxy publishes PCP metrics over a REST API for Grafana. A common split is PCP for high resolution local history, and the central server for alerting across hosts.