How to monitor iocaine with Prometheus and Grafana?

Last updated: 2025-09-07

iocaine can be configured to expose Prometheus-compatible metrics, on an endpoint separate from the garbage generator. When enabled, two groups of metrics are exposed: iocaine’s own metrics, and process metrics such as the process start time, CPU seconds used, resident memory used, and so on. Where relevant, the iocaine version that introduced the metric will be highlighted.

Metrics are not enabled by default, one has to manually configure iocaine to make them available:

[metrics]
enable = true

By default, metrics are lost when iocaine shuts down, but - since iocaine 2.4.0 - if the [metrics].persist-path option is set, then iocaine will save its metrics on shutdown, and will load compatible ones when starting back up.

iocaine metrics

There are three iocaine-specific metrics exposed:

iocaine_requests_total

The number of requests served by iocaine.

iocaine_garbage_served

The amount of garbage generated, in bytes. This does not include HTTP headers, and is calculated before any compression by the fronting reverse proxy.

Note: Prior to iocaine 1.1.0, this metric was always unlabeled.

iocaine_maze_depth

The depth of the explored maze, in number of path elements. This is the deepest a visitor has gone in the maze.

Introduced in: iocaine 1.1.0

iocaine_challenges

The number of challenges served by iocaine. To serve a challenge, a custom request handler is required.

Introduced in: iocaine 2.2.0

iocaine_request_handler_hits{id="..."}

The number of times a particular “id” in the request handler was hit. This metric is fully controlled by the request handler, if any, and as such, the id label’s meaning and purpose is also controlled by the request handler.

Unlike most other metrics, id is the only label attached to this one.

Introduced in: iocaine 2.3.0

iocaine_version{version="2.0.0-snapshot"}

The version of the iocaine instance, provided within the version label. The value is always one, and no other labels are provided.

Introduced in: iocaine 2.0.0

All of these metrics (except iocaine_version and iocaine_request_handler_hits) are augmented with the same set of - optional - labels, and are emitted without labels by default:

# HELP iocaine_garbage_served Total amount of garbage served (in bytes)
# TYPE iocaine_garbage_served counter
iocaine_garbage_served 56924

# HELP iocaine_requests_total Total number of requests served
# TYPE iocaine_requests_total counter
iocaine_requests_total 69

# HELP iocaine_maze_depth Maximum explored depth of the maze (in path parts)
# TYPE iocaine_maze_depth counter
iocaine_maze_depth 10

# HELP iocaine_challenges Number of challenges presented
# TYPE iocaine_challenges counter
iocaine_challenges 0

# HELP iocaine_request_handler_hits Total number of times the request handler was invoked
# TYPE iocaine_request_handler_hits counter
iocaine_request_handler_hits{id="some-random-id"} 1

# HELP iocaine_version Version of the running iocaine (in the 'version' label)
# TYPE iocaine_version gauge
iocaine_version{version="2.0.0"} 1

Labels

Important
Labels are only applied to the iocaine metrics. The process metrics are always unlabeled.

There are four labels that can be enabled via [metrics].labels: host, user_agent, user_agent_group, and verdict. These can be used to group the iocaine metrics by host, user agent, user agent group, verdict, or any combination of those. As the user agent is entirely externally controlled, it is recommended to keep that label turned off, because it makes it very easy to fake a million different user agents, which will make metrics very expensive to keep track of, to ingest, and to work with.

When using the user_agent_group label, it is strongly recommended to have a catch-all rule as the last one, to limit how many groups there may be.

Lets see a few examples!

Host labels

[metrics]
enable = true
labels = [ "Host" ]

This will expose the following metrics on http://127.0.0.1:42042/metrics:

# HELP iocaine_garbage_served Total amount of garbage served (in bytes)
# TYPE iocaine_garbage_served counter
iocaine_garbage_served{host="example.com"} 1508
iocaine_garbage_served{host="another.example.com"} 4074

# HELP iocaine_requests_total Total number of requests served
# TYPE iocaine_requests_total counter
iocaine_requests_total{host="example.com"} 1
iocaine_requests_total{host="another.example.com"} 2

# HELP iocaine_maze_depth Maximum explored depth of the maze (in path parts)
# TYPE iocaine_maze_depth counter
iocaine_maze_depth{host="example.com"} 10
iocaine_maze_depth{host="another.example.com"} 1

User agent labels

Important
It is usually not a good idea to use the UserAgent label, because that can lead to metrics balooning out of control. Only use that label if you keep a close eye on Prometheus.

[metrics]
enable = true
labels = [ "UserAgent" ]

This will expose the following metrics on http://127.0.0.1:42042/metrics:

# HELP iocaine_garbage_served Total amount of garbage served (in bytes)
# TYPE iocaine_garbage_served counter
iocaine_garbage_served{user_agent="curl/8.11"} 7916
iocaine_garbage_served{user_agent="bad bot/666"} 262144

# HELP iocaine_requests_total Total number of requests served
# TYPE iocaine_requests_total counter
iocaine_requests_total{user_agent="curl/8.11"} 6
iocaine_requests_total{user_agent="bad bot/666"} 202

# HELP iocaine_maze_depth Maximum explored depth of the maze (in path parts)
# TYPE iocaine_maze_depth counter
iocaine_maze_depth{user_agent="curl/8.11"} 1
iocaine_maze_depth{user_agent="bad bot/666"} 71

User agent grouping

The downside of the user_agent label is that it is unbounded, and the user agent strings tend to be long. To make it easier to group by agents, another label is available: user_agent_group. This label is applied by matching a list of regexps against the user agent, and if any matches, the corresponding group will be set for the value of the user_agent_group label. This is best demonstrated by an example:

[metrics]
enable = true
labels = [ "UserAgent", "UserAgentGroup" ]

[[metrics.agent_group]]
agent = "(?i:bad bot)"
group = "BadBot"

[[metrics.agent_group]]
agent = "."
group = "Other"

Do note that the UserAgentGroup label can be used without UserAgent! They’re used here together for demonstration purposes. With the above setup, we’ll get metrics like the following exposed on http://127.0.0.1:42042/metrics:

# HELP iocaine_garbage_served Total amount of garbage served (in bytes)
# TYPE iocaine_garbage_served counter
iocaine_garbage_served{user_agent="curl/8.11", user_agent_group="Other"} 7916
iocaine_garbage_served{user_agent="bad bot/666", user_agent_group="BadBot"} 262144
iocaine_garbage_served{user_agent="bad bot/666.1", user_agent_group="BadBot"} 1279

# HELP iocaine_requests_total Total number of requests served
# TYPE iocaine_requests_total counter
iocaine_requests_total{user_agent="curl/8.11", user_agent_group="Other"} 6
iocaine_requests_total{user_agent="bad bot/666", user_agent_group="BadBot"} 202
iocaine_requests_total{user_agent="bad bot/666.1", user_agent_group="BadBot"} 1

# HELP iocaine_maze_depth Maximum explored depth of the maze (in path parts)
# TYPE iocaine_maze_depth counter
iocaine_maze_depth{user_agent="curl/8.11", user_agent_group="Other"} 1
iocaine_maze_depth{user_agent="bad bot/666", user_agent_group="BadBot"} 71
iocaine_maze_depth{user_agent="bad bot/666.1", user_agent_group="BadBot"} 18

Request handler verdict

Important
The verdict label is new in iocaine 2.2.0, and is mostly useful when using the customisable request handler. If not using it, the verdict will always be “accept::garbage”.

The iocaine 2.2.0 release introduced a scriptable request handler, which lets the custom script decide what to do with an incoming request: accept it and serve it garbage, accept it, and serve it a challenge, or reject to handle it. The verdict label contains the outcome of this decision.

[metrics]
enable = true
labels = [ "Verdict" ]

This will expose the following metrics on http://127.0.0.1:42042/metrics:

# HELP iocaine_garbage_served Total amount of garbage served (in bytes)
# TYPE iocaine_garbage_served counter
iocaine_garbage_served{verdict="accept::garbage"} 56924

# HELP iocaine_requests_total Total number of requests served
# TYPE iocaine_requests_total counter
iocaine_requests_total{verdict="accept::garbage"} 69
iocaine_requests_total{verdict="accept::challenge"} 1
iocaine_requests_total{verdict="reject::not_for_us"} 42

# HELP iocaine_maze_depth Maximum explored depth of the maze (in path parts)
# TYPE iocaine_maze_depth counter
iocaine_maze_depth{verdict="accept::garbage"} 10

# HELP iocaine_challenges Number of challenges presented
# TYPE iocaine_challenges counter
iocaine_challenges{verdict="accept::challenge"} 1

Process metrics

Important
Process-specific metrics are new to iocaine 1.1.0, except for process_start_time_seconds.

When metrics are enabled, the following process metrics are exposed:

process_start_time_seconds: Start time of the process since unix epoch in seconds.

Note: Prior to iocaine 1.1.0, this metric always had a service label with iocaine as its value. That is no longer the case, it is unlabeled in more recent versions.
process_cpu_seconds_total: Total user and system CPU time spent in seconds.

Introduced in: iocaine 1.1.0
process_resident_memory_bytes: Resident memory size in bytes.

Introduced in: iocaine 1.1.0
process_virtual_memory_bytes: Virtual memory size in bytes.

Introduced in: iocaine 1.1.0
process_threads: Number of OS threads in the process.

Introduced in: iocaine 1.1.0
process_open_fds: Number of open file descriptors.

Introduced in: iocaine 1.1.0
process_max_fds: Maximum umber of open file descriptors.

Introduced in: iocaine 1.1.0

Prometheus configuration

The iocaine metrics are exposed on an unauthenticated address, and Prometheus can be told to scrape it with a configuration akin to the following example:

scrape_configs:
- job_name: iocaine-job
  static_configs:
  - targets: ['localhost:42042']

Dashboard

A Grafana dashboard is available to get you started with monitoring iocaine with Grafana.

Dashboard

It assumes a configuration where the following labels are enabled:

[metrics]
enable = true
labels = [ "Host", "UserAgentGroup", "Verdict" ]

[[metrics.agent_group]]
agent = "(?i:ClaudeBot)"
group = "ClaudeBot"

[[metrics.agent_group]]
agent = "(?i:AmazonBot)"
group = "Amazon"

[[metrics.agent_group]]
agent = "(?i:GPTBot|ChatGPT)"
group = "GPTBot"

[[metrics.agent_group]]
agent = "."
group = "Other"