Skip to content

Prometheus Metrics

Sylve exposes local jail, VM, container, and ZFS measurements at GET /metrics on its management listener. Prometheus or VictoriaMetrics collects and stores these measurements; Grafana queries that store through its Prometheus data source.

To require authentication, generate a dedicated scrape token:

Terminal window
openssl rand -hex 32

Add a metrics section to your existing config.json, replacing the placeholder with the generated token, then restart Sylve:

"metrics": {
"enabled": true,
"token": "<a-random-scrape-token>"
}

The endpoint defaults to disabled and returns HTTP 404. Set token to an empty string (or omit it) to allow unauthenticated scrapes:

"metrics": {
"enabled": true,
"token": ""
}

When a nonempty token is configured, requests must send Authorization: Bearer <token>. Missing or incorrect credentials receive HTTP 401. Tokens must contain no whitespace. The scrape token is separate from login sessions and cluster credentials and only authorizes metrics access.

Use the node’s HTTPS management address when scraping. Token changes take effect after restarting Sylve.

For token-authenticated scrapes, save the token in a file readable by the scraper, then add this job to its Prometheus-compatible scrape configuration:

scrape_configs:
- job_name: sylve
scrape_interval: 15s
scheme: https
metrics_path: /metrics
authorization:
type: Bearer
credentials_file: /config/sylve-metrics.token
# For a private CA, configure tls_config.ca_file with the trusted CA PEM.
static_configs:
- targets: ['sylve.example.com:8181']

If Sylve’s metrics token is empty, omit the authorization block. Scrape each Sylve node directly. Each endpoint reports its local node, and the scraper adds job and instance labels to distinguish nodes.

For the Daemonless VictoriaMetrics catalog app, place the scrape configuration at /config/prometheus.yml inside the container. Configure Grafana’s Prometheus data source with the VictoriaMetrics address, for example http://victoriametrics.example.com:8428. With Prometheus, use its server address instead.

Sylve provides a ready-made Grafana dashboard for Grafana 10 or later. In Grafana, open Dashboards, choose New and then Import, and upload sylve-node.json. After importing, pick your Prometheus or VictoriaMetrics data source from the dashboard’s Data source selector; the Node and Pool selectors narrow the view to specific nodes and pools.

Section Contents
Overview Collector health, running and total jails, VMs and containers, unhealthy pools, ARC hit ratio and pending image updates
ZFS pools Pool state table, capacity, IOPS, throughput and latency
Jails, Virtual machines, Containers CPU and memory of each running guest
ARC Size against target and maximum, and hit ratio
Guest inventory Tables of every jail, VM and container with state, CPU, memory and limits
Datasets Largest datasets, quota usage and a table of every filesystem and zvol

Guest inventory and Datasets are collapsed by default, so the dashboard stays fast on nodes with many guests or datasets; Grafana only queries them once expanded. Pool, ARC and dataset legends include the scrape instance label, because several nodes commonly share pool names such as zroot.

Scrapes read thread-safe in-memory snapshots. Collection happens in the background, so a scrape performs no host commands or database writes.

Collector Source and cadence
jails Existing jail monitoring, every 5 seconds
vms Existing VM monitoring, roughly every 16 seconds
containers RACCT usage of each container’s jail, every 15 seconds while metrics are enabled
zfs_pools Managed-pool inventory and properties, every 10 seconds
zfs_datasets Recursive filesystem and zvol listings on managed pools, every 30 to 40 seconds while metrics are enabled; a failed listing is retried on the next 10-second pass
zfs_arc Host-wide ARC sysctls, every 10 seconds

sylve_collector_success{collector="..."} is 1 when the latest collection succeeded and is fresh. It is 0 before monitoring starts, after a collection failure, when the snapshot expires, or while the corresponding service is disabled. sylve_collector_last_success_timestamp_seconds gives the Unix timestamp of the last fully successful collection, or 0 before the first success. For vms, a failure means the database or libvirt connection was unavailable; a single VM that could not be measured is reported by sylve_vm_collection_success instead.

Measurements are omitted once they are more than 60 seconds old. When a collection fails outright, the previous measurements remain available with a collection-success value of 0 until they reach that age, so a transient error does not leave gaps. A partial failure, such as one pool’s dataset listing failing, exposes the successfully collected results alongside a collection-success value of 0. Deleted resources disappear after the next successful collection. Collection health is independent of HTTP scrape success: up can be 1 while a Sylve collector reports 0.

Pool I/O has its own freshness check: samples expire after 30 seconds based on the original zpool iostat sample time. sylve_zfs_pool_io_collection_success reports whether a recent sample is available. Unavailable I/O and optional ZFS properties are omitted rather than reported as zero.

Jail measurements use the ctid label. Names and hostnames appear in the information metric.

Metric Meaning
sylve_jail_info CTID, name and hostname
sylve_jail_running Running = 1, stopped = 0
sylve_jail_cpu_usage_ratio CPU utilization normalized to allowed cores, capped at 1
sylve_jail_cpu_cores Core count used for normalization
sylve_jail_memory_rss_bytes Summed process RSS, including descendant jails; shared pages can be counted repeatedly
sylve_jail_resource_limits_enabled Whether resource limits are configured
sylve_jail_memory_limit_bytes Configured limit; omitted when unlimited

Stopped jails remain in the inventory with zero CPU and RSS measurements.

VM measurements use the rid label. Names appear in the information metric.

Metric Meaning
sylve_vm_info RID and name
sylve_vm_running, sylve_vm_state Running flag and one-hot libvirt state
sylve_vm_cpu_seconds_total Cumulative CPU seconds; resets on guest restart
sylve_vm_memory_rss_bytes Host bhyve process RSS, rather than guest application memory usage
sylve_vm_memory_configured_bytes, sylve_vm_vcpus Configured memory and vCPU count
sylve_vm_collection_success Whether the latest measurements for this VM were collected successfully

Stopped VMs remain in the inventory. CPU and RSS measurements are omitted while stopped or unavailable. VM CPU time is a counter; use rate() to calculate utilization.

Container measurements use the ocid label. Names, images, pools and stacks appear in the information metric. Usage comes from RACCT, so sylve_collector_success{collector="containers"} reports 0 on hosts where RACCT is disabled.

Metric Meaning
sylve_container_info OCID, name, image, pool and stack
sylve_container_running Running = 1, stopped = 0
sylve_container_cpu_seconds_total Cumulative CPU time of the container’s jail in whole seconds; can reset when the container restarts
sylve_container_cpu_cores Configured core limit, or host logical cores when unlimited
sylve_container_memory_rss_bytes Resident memory of the container’s jail
sylve_container_memory_limit_bytes Configured limit; omitted when unlimited
sylve_container_image_update_available Whether the registry has a newer digest for the container’s image tag

Stopped containers remain in the inventory. CPU and memory measurements are omitted while stopped. Because RACCT reports CPU time in whole seconds, use rate() over a few minutes for a smooth utilization graph.

Pool measurements use the pool label and cover imported pools selected for management in Sylve.

Metric Meaning
sylve_zfs_pool_info Pool name and GUID
sylve_zfs_pool_health One-hot health with a state label; the current state is 1 and other states are 0
sylve_zfs_pool_size_bytes, sylve_zfs_pool_allocated_bytes, sylve_zfs_pool_free_bytes Total, allocated and free pool space
sylve_zfs_pool_fragmentation_ratio Fragmentation from 0 to 1
sylve_zfs_pool_dedup_ratio Deduplication ratio
sylve_zfs_pool_io_collection_success Whether a recent pool I/O sample is available
sylve_zfs_pool_read_operations_per_second, sylve_zfs_pool_write_operations_per_second Logical read and write IOPS
sylve_zfs_pool_read_bytes_per_second, sylve_zfs_pool_write_bytes_per_second Logical read and write throughput
sylve_zfs_pool_read_latency_seconds, sylve_zfs_pool_write_latency_seconds Average total wait time, including queueing and device service, where supported

Health states are ONLINE, DEGRADED, FAULTED, OFFLINE, REMOVED, UNAVAIL, CORRUPT_DATA, and UNKNOWN. I/O values describe the latest 10-second zpool iostat interval. These are gauges containing rates, so use them directly rather than applying rate().

Dataset measurements use pool and dataset labels. They cover filesystems and zvols on managed pools, including pool-root filesystems. Container image layer datasets under <pool>/sylve/containers/images are excluded; the usage of the images dataset itself includes them.

Metric Meaning
sylve_zfs_dataset_info Pool, dataset path, GUID and type (filesystem or volume)
sylve_zfs_dataset_used_bytes Space consumed by a dataset and its descendants
sylve_zfs_dataset_available_bytes Space available to the dataset
sylve_zfs_dataset_referenced_bytes Data accessible by the dataset
sylve_zfs_dataset_logical_used_bytes Logical space consumed, ignoring compression and copies
sylve_zfs_dataset_quota_bytes Filesystem quota; omitted when unlimited or unavailable
sylve_zfs_dataset_volume_size_bytes Logical zvol size
sylve_zfs_dataset_compression_ratio Compression ratio

Individual snapshots are excluded. Dataset usage can include descendants and snapshots, so summing parent and child dataset usage double-counts space. Use pool capacity metrics for total pool utilization.

ARC measurements are host-wide and include activity from every pool on the node.

Metric Meaning
sylve_zfs_arc_size_bytes Current ARC size
sylve_zfs_arc_target_size_bytes Adaptive target size
sylve_zfs_arc_min_size_bytes, sylve_zfs_arc_max_size_bytes Minimum and maximum target sizes
sylve_zfs_arc_hits_total, sylve_zfs_arc_misses_total Cache-hit and cache-miss counters since ZFS initialization

VM CPU utilization as a percentage of configured vCPUs:

100 * rate(sylve_vm_cpu_seconds_total[5m]) / sylve_vm_vcpus

Container CPU utilization as a percentage of available cores:

100 * rate(sylve_container_cpu_seconds_total[5m]) / sylve_container_cpu_cores

Pool capacity used, as a percentage:

100 * sylve_zfs_pool_allocated_bytes / sylve_zfs_pool_size_bytes

ARC cache-hit ratio:

rate(sylve_zfs_arc_hits_total[5m])
/
(rate(sylve_zfs_arc_hits_total[5m]) + rate(sylve_zfs_arc_misses_total[5m]))

Collectors requiring attention:

sylve_collector_success == 0