Prometheus commands

PromQL expressions, promtool subcommands, server flags and HTTP API calls for Prometheus 3.15, checked against the project's documentation and the built-in help of the 3.15.0 release binaries. PromQL rows are literal examples from the docs; capitalised words such as SERVER or RULE-FILES stand for your own values. Apply rate(http_requests_total[5m]) to counters before summing, and note that reloading over HTTP needs --web.enable-lifecycle. Experimental functions and storage tuning flags are left out.

149 commands in 14 sections. Checked row by row against Prometheus 3.15 command-line reference: prometheus (docs/command-line/prometheus.md, tag v3.15.0) on . Edited by Jay.

Selecting series 11

CommandWhat it does
upTarget statusHolds 1 for each target whose last scrape succeeded and 0 for one that failed.
http_requests_totalSelect by metric nameReturns the newest sample of every series with that name, which may expand to thousands of series.
http_requests_total{job="prometheus",group="canary"}Filter by exact label valuesMatchers go in braces separated by commas; here both job and group must match.
http_requests_total{environment=~"staging|testing|development",method!="GET"}Regex and negative matchersThe =~ matcher takes a regular expression, always fully anchored, while != excludes one exact value.
http_requests_total{job=~".*server"}Jobs ending in a suffixSince regex matches are anchored at both ends, the leading .* is what allows any prefix.
http_requests_total{status!~"4.."}Exclude 4xx status codesThe !~ matcher keeps only series whose status label does not fit the pattern.
http_requests_total{environment=""}Series without a labelAn empty-value matcher also returns series that carry no environment label at all.
http_requests_total{replica!="rep-a",replica=~"rep.*"}Two matchers on one labelEvery matcher for the same label has to pass, so only the rep-b series is returned.
{__name__=~"job:.*"}Match metric names by regexUses the internal __name__ label, here to pick every metric whose name starts with job:.
{job=~".+"}Selector without a metric nameA nameless selector needs one matcher that rejects empty values; {job=~".*"} on its own is illegal.
http_requests_total{job="prometheus"}[5m]Range vector selectorA duration in brackets returns all samples from that window; the result shows in the table view but cannot be graphed.

Offsets, @ and subqueries 9

offset and @ must follow the selector itself: sum(http_requests_total{method="GET"} offset 5m) works, the same modifier after sum(...) does not.

CommandWhat it does
http_requests_total offset 5mShift back in timeThe offset has to follow the selector directly: sum(http_requests_total{method="GET"} offset 5m) is valid.
rate(http_requests_total[5m] offset 1w)Rate one week agoOffset also works on the range selector inside a function, useful for week-over-week comparisons.
rate(http_requests_total[5m] offset -1w)Negative offsetCompares forward in time, so the query may read samples newer than its own evaluation point.
http_requests_total @ 1609746000Evaluate at a fixed timeThe value after @ is a Unix timestamp; this one is 2021-01-04 07:40 UTC.
http_requests_total @ start()Pin to the query startIn a range query start() stays fixed for every step; in an instant query it is the evaluation time.
rate(http_requests_total[5m] @ end())Rate at the range endend() resolves to the end of a range query, so every step of the graph shows one fixed value.
rate(http_requests_total[5m])[30m:1m]Rate as a subqueryEvaluates the inner rate once a minute across the past 30 minutes and yields a range vector.
max_over_time(deriv(rate(distance_covered_total[5s])[30s:5s])[10m:])Nested subqueryAn empty resolution, as in [10m:], falls back to the global evaluation interval.
rate(http_requests_total[5m * 2])Arithmetic in a rangeRanges and offsets accept duration expressions; 5m * 2 gives a ten-minute window.

Rates and changes 9

Take rate or irate first and aggregate afterwards; summing raw counters before the rate hides counter resets when a target restarts.

CommandWhat it does
rate(http_requests_total[5m])Request ratePer-second average over five minutes with counter resets handled; take the rate before any sum.
irate(http_requests_total{job="api-server"}[5m])Instant rateBuilt from the last two samples only, so it suits fast-moving graphs but not alerts.
increase(http_requests_total{job="api-server"}[5m])Increase over a windowRequests added in five minutes; extrapolation can give fractions, and recording rules should use rate instead.
delta(cpu_temp_celsius{host="zeus"}[2h])Gauge change over a windowChange in a gauge across two hours, extrapolated to the window edges; meant for gauges only.
idelta(v range-vector)Difference of last two samplesUnlike delta, it ignores everything in the range except the newest pair of samples.
deriv(v range-vector)Per-second slope of a gaugeFits a simple linear regression, so the range must hold two or more float samples.
predict_linear(v range-vector, t scalar)Forecast a gaugeProjects each series t seconds ahead from a linear fit over the range; gauges only.
changes(v range-vector)Count value changesWorks per series across the range; a switch between float and histogram samples also counts.
resets(v range-vector)Count counter resetsAny drop between two consecutive float samples counts as one, so use it on counters only.

Aggregating over time 9

CommandWhat it does
avg_over_time(range-vector)Average over timeEvery sample in the window carries equal weight, even when the samples are unevenly spaced.
max_over_time(range-vector)Peak within the rangeLooks at float samples only; histogram samples in the window are left out.
min_over_time(range-vector)Lowest value in the range
sum_over_time(range-vector)Sum within the rangeAdds up each series across its window; a range mixing floats and histograms is dropped from the output.
count_over_time(range-vector)Samples in the rangeCounts float and histogram samples alike, separately for each series.
quantile_over_time(scalar, range-vector)Quantile within the rangeThe scalar is a quantile between 0 and 1, so 0.95 asks for the 95th percentile of the window.
last_over_time(range-vector)Newest sample in the range
first_over_time(range-vector)Oldest sample in the rangeUnlike m offset 1m, it takes the first sample inside the window rather than one before it.
present_over_time(range-vector)Series seen in the rangeReturns 1 for every series that has any sample in the window, whatever its value.

Aggregation operators 10

CommandWhat it does
sum by (job) ( rate(http_requests_total[5m]) )Sum by groupAdds the per-instance rates together and keeps only the job label in the output.
sum without (instance) (memory_consumption_bytes)Sum without labelRemoves only the listed label and keeps the rest; here it equals the by (application, group) form.
sum by (application, group) (memory_consumption_bytes)Sum keeping listed labelsLabels missing from the by list are dropped even when their values are the same everywhere.
sum(memory_consumption_bytes)Total across all seriesWith no by or without clause, every label is aggregated away into a single value.
avg by (job, instance, mode) (rate(node_cpu_seconds_total[5m]))Average CPU rate per modeKeeps job, instance and mode, averaging away every other label; the guide stores it as a recording rule.
topk(5, memory_consumption_bytes)Top K seriesOriginal labels are kept, and an instant query returns the series sorted from highest down.
topk(3, sum by (app, proc) (rate(instance_cpu_time_ns[5m])))Top three CPU usersThe inner sum groups by app and proc first, so topk chooses among those pairs.
count by (app) (instance_cpu_time_ns)Count by labelWith one series per running instance, this gives the number of instances for each app.
count(prometheus_target_interval_length_seconds)Count returned seriesAnswers how many series the selector matched instead of listing their values.
count_values("version", build_version)Count identical valuesEmits one series per distinct value, labelled version, holding how many binaries report that build.

Binary operators and matching 9

CommandWhat it does
(instance_memory_limit_bytes - instance_memory_usage_bytes) / 1024 / 1024Free memory in MiBSeries with identical label sets on the two sides are paired before the subtraction.
up == 0Targets that are downComparisons filter by default, so only failed targets remain; the alerting guide waits 5m before InstanceDown fires.
job:request_latency_seconds:mean5m{job="myjob"} > 0.5Threshold comparisonKeeps only series above 0.5; the HighRequestLatency example also sets for: 10m.
boolReturn 0 or 1 insteadWritten after a comparison operator, it turns the filter into a 0 or 1 result for each match.
andSet intersectionKeeps left-hand elements that have an exactly matching label set on the right.
orSet unionAdds right-hand elements whose label sets are missing on the left side.
unlessSet complementKeeps left-hand elements for which the right side has no exactly matching label set.
method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5mMatch while ignoring a labelGives the share of 500 errors per method; without ignoring(code) the label sets would never line up.
method_code:http_errors:rate5m / ignoring(code) group_left method:http_requests:rate5mMany-to-one matchinggroup_left lets several left-hand series share one right-hand series, so the code label survives.

Histograms 6

CommandWhat it does
histogram_quantile(0.9, rate(http_request_duration_seconds_bucket[10m]))90th percentile, classicReads the _bucket series and their le label; the rate window sets the period measured.
histogram_quantile(0.9, rate(http_request_duration_seconds[10m]))90th percentile, nativeNative histograms need no _bucket suffix because each sample holds a whole histogram.
histogram_quantile(0.9, sum by (job, le) (rate(http_request_duration_seconds_bucket[10m])))Percentile per job, classicAggregate with sum inside the call and keep le in the by clause, since the function needs it.
histogram_count(rate(http_request_duration_seconds[10m]))Requests per secondCounts observations in native histogram samples; plain float samples are left out of the result.
histogram_avg(rate(http_request_duration_seconds[5m]))Average request durationSame result as histogram_sum divided by histogram_count of that native histogram rate.
histogram_fraction(0, 0.2, rate(http_request_duration_seconds[1h]))Share under 200 msBounds that miss bucket boundaries make the answer an estimate, especially with classic histograms.

Labels, absence and time 10

CommandWhat it does
absent(nonexistent{job="myjob"})Absence checkReturns 1 labelled job="myjob" when nothing matches, the usual way to alert on a vanished metric.
absent_over_time(nonexistent{job="myjob"}[1h])Absent for a whole windowReturns 1 only if the selector found no sample at any point in the last hour.
label_replace(up{job="api-server",service="a:c"}, "foo", "$1", "service", "(.*):.*")Copy a label via regexWrites capture group $1 of the service value into a new foo label, here a; other labels stay.
label_join(up{job="api-server",src1="a",src2="b",src3="c"}, "foo", ",", "src1", "src2", "src3")Join labels into oneCombines src1, src2 and src3 with a comma separator, giving foo="a,b,c".
time()Evaluation timeSeconds since the Unix epoch at the moment the expression is evaluated, not the wall clock.
timestamp(v instant-vector)Sample timestampsGives each sample its own time in epoch seconds, for floats and histograms alike.
vector(s scalar)Scalar to vectorWraps a plain number in a one-element vector that has no labels.
scalar(v instant-vector)Vector to scalarAny input other than a single float element gives NaN instead of a number.
clamp(v instant-vector, min scalar, max scalar)Limit values to a rangeRaises values below min and lowers values above max; histogram samples are skipped silently.
round(v instant-vector, to_nearest=1 scalar)Round valuesTies round up, and the optional to_nearest sets the multiple to round to, which may be a fraction.

promtool checks and tests 12

CommandWhat it does
promtool check config CONFIG-FILESCheck configAlso validates files the configuration refers to, such as rule files, unless --syntax-only is set.
promtool check config --syntax-only CONFIG-FILESCheck config syntax onlySkips validating the files and content that the configuration refers to.
promtool check config --lint=none CONFIG-FILESCheck config without lintingBy default the check also lints for duplicate rules; all and too-long-scrape-interval are the other choices.
promtool check rules RULE-FILESCheck rulesWith no file named it reads rules from standard input; duplicate rules are linted unless --lint=none.
promtool check web-config WEB-CONFIG-FILESCheck web config filesValidates the files passed to --web.config.file, which set up TLS or authentication.
promtool check service-discovery CONFIG-FILE JOBTest service discovery for a jobReports what discovery returns for one job, relabeling included, and waits up to 30s by default.
promtool check healthy --url=http://localhost:9090Check server healthThe URL shown is the default, so the flag only matters for a server elsewhere.
promtool check ready --url=http://localhost:9090Check server readinessAsks whether the server is ready to answer queries; --url points it at another address.
cat metrics.prom | promtool check metricsLint exposition textReads metrics from standard input and checks them for consistency and correctness.
curl -s http://localhost:9090/metrics | promtool check metrics --extendedLint with cardinality reportWith --extended the output adds cardinality figures for the metrics read.
promtool test rules TEST-RULE-FILEUnit-test rulesAdd --debug for debugging output, or the experimental --diff for a coloured comparison of results.
promtool test rules --run=RUN TEST-RULE-FILERun selected test groupsLimits the run to test groups with matching names; the flag can be repeated for more patterns.

promtool queries and debugging 11

The promql subcommands are experimental and only run when the global --experimental flag is given.

CommandWhat it does
promtool query instant SERVER EXPRInstant querySERVER is the address of a running Prometheus, and EXPR is the PromQL expression to evaluate.
promtool query instant --time=TIME SERVER EXPRInstant query at a set timeTIME is an RFC3339 date or a Unix timestamp.
promtool query range --start=START --end=END --step=STEP SERVER EXPRRange queryStart and end take RFC3339 or Unix timestamps; the step is a duration such as 15s.
promtool query series --match=MATCH SERVERFind seriesLists series matching the selector; repeat --match to combine several selectors.
promtool query labels SERVER NAMEList label valuesNAME is the label whose values are returned; --match and a time range narrow the result.
promtool query analyze --server=SERVER --type=TYPE --match=MATCHAnalyse metric usageHistogram is the supported type, and one hour is analysed unless --duration says otherwise.
promtool debug all SERVERFetch all debug data
promtool debug pprof SERVERFetch profiling data
promtool push metrics REMOTE-WRITE-URL METRIC-FILESPush metrics for testingMeant for testing only; files default to standard input and samples get job=promtool unless --label says otherwise.
promtool promql format QUERYPretty-print PromQLThe promql commands require the global --experimental flag, written before the command name.
promtool --help-longHelp for every commandPrints the usage line and flags of all subcommands at once.

promtool TSDB tools 6

CommandWhat it does
promtool tsdb listList TSDB blocksReads the database in data/ unless a database path is given.
promtool tsdb list -rList blocks, human-readablePrints the values in human-readable form rather than as raw numbers.
promtool tsdb analyzeAnalyse churn and cardinalityLooks at the last block unless a block ID is given, and shows 20 items per list by default.
promtool tsdb dumpDump series and samplesUses the prom output format by default; --format=seriesjson is the other choice.
promtool tsdb create-blocks-from openmetrics INPUT OUTPUTBackfill from OpenMetricsINPUT is an OpenMetrics text file; OUTPUT is the directory for the new blocks, data/ by default.
promtool tsdb create-blocks-from rules --start=START RULE-FILESBackfill recording rulesQueries the server at --url to compute past values; without --end it backfills up to three hours ago.

Server flags 23

The 3.15 reference marks both retention flags as deprecated: set storage.tsdb.retention.time and storage.tsdb.retention.size in the configuration file instead.

CommandWhat it does
-hShow help--help-long and --help-man print the longer variants of the flag list.
--versionPrint the version
--config.fileConfiguration file pathWithout it the server looks for prometheus.yml, the default value.
--config.auto-reloadReload config on changeChecks the file every 30s by default, adjustable with --config.auto-reload-interval.
--web.listen-addressListen addressDefaults to 0.0.0.0:9090 for the UI, API and telemetry, and may be given more than once.
--web.external-urlPublic URL behind a proxyUsed to build links back to the server; a path in it prefixes every HTTP endpoint.
--web.enable-lifecycleAllow reload and quit over HTTPOff by default; it switches on POST /-/reload and POST /-/quit.
--web.enable-admin-apiEnable admin endpointsOff by default; the TSDB snapshot, delete-series and clean-tombstones calls need it.
--web.enable-remote-write-receiverAccept remote writeOff by default; turns on the endpoint that takes remote write requests from other senders.
--web.enable-otlp-receiverAccept OTLP metricsOff by default; opens the endpoint that takes OTLP write requests.
--web.config.fileTLS and auth configPoints to a separate file that can switch on TLS or authentication; marked experimental.
--storage.tsdb.pathData directoryDefaults to data/ and applies in server mode; agent mode uses --storage.agent.path instead.
--storage.tsdb.retention.timeRetention period (deprecated)Marked deprecated: set storage.tsdb.retention.time in the config file; with no limit set, 15d applies.
--storage.tsdb.retention.sizeSize limit (deprecated)Needs a unit such as 512MB counted in powers of 2; the config file field replaces it.
--agentRun in agent modeThe --storage.agent.* flags only take effect in this mode.
--storage.agent.pathAgent data directoryDefaults to data-agent/ and is read only in agent mode.
--enable-featureEnable feature flagsTakes a comma-separated list, such as promql-experimental-functions for the experimental PromQL functions.
--query.lookback-deltaStaleness lookbackHow far back a selector looks for the latest sample, 5m by default; one query can override it.
--query.timeoutQuery time limitQueries still running after this, 2m by default, are aborted.
--query.max-concurrencyConcurrent query limitUp to 20 queries run at the same time by default.
--query.max-samplesSamples per query limitA query loading more than 50,000,000 samples into memory fails, which also caps what it can return.
--log.levelLog levelOne of debug, info, warn or error; deprecated in favour of runtime.log_level in the config file.
--log.formatLog formatEither logfmt, the default, or json.

Running and reloading the server 7

CommandWhat it does
./prometheus --config.file=prometheus.ymlStart the serverRun from the binary directory; data lands in ./data and the status page is at localhost:9090.
GET /-/healthyHealth checkAlways answers 200; a HEAD request works too.
GET /-/readyReadiness checkReturns 200 once the server can answer queries; HEAD is accepted as well.
POST /-/reloadReload configurationRe-reads configuration and rule files; needs --web.enable-lifecycle, and PUT works too.
POST /-/quitGraceful shutdownDisabled unless --web.enable-lifecycle is set; a SIGTERM signal has the same effect.
kill -s SIGHUP PIDReload via signalReloads without restarting the process; replace PID with the server process ID.
kill -s SIGTERM PIDStop via signalSIGINT works as well, as does the interrupt character, Ctrl + C by default, at the terminal.

HTTP API 17

CommandWhat it does
curl 'http://localhost:9090/api/v1/query?query=up&time=2015-07-01T20:10:51.781Z'Instant query over HTTPEvaluates up at one moment; without the time parameter the current server time is used.
curl 'http://localhost:9090/api/v1/query_range?query=up&start=2015-07-01T20:10:30.781Z&end=2015-07-01T20:11:00.781Z&step=15s'Range query over HTTPEvaluates up across 30 seconds at 15-second steps and returns a matrix.
curl 'http://localhost:9090/api/v1/format_query?query=foo/bar'Pretty-print an expressionReturns the expression in formatted form, with any comments removed.
curl -g 'http://localhost:9090/api/v1/series?' --data-urlencode 'match[]=up' --data-urlencode 'match[]=process_start_time_seconds{job="prometheus"}'Find series by selectorEach match[] adds a selector, and series matching any of them come back with their labels.
curl 'localhost:9090/api/v1/labels'List label namesStart and end are approximate, so names from series without samples in the window may appear.
curl http://localhost:9090/api/v1/label/http_status_code/valuesList values of a labelOptional match[] selectors limit which series contribute values.
curl http://localhost:9090/api/v1/targetsList scrape targetsShows active and dropped targets, with labels both before and after relabeling.
curl 'http://localhost:9090/api/v1/targets?state=active'Only active targetsThe state parameter also accepts dropped or any; other values are ignored.
curl http://localhost:9090/api/v1/rulesList loaded rulesAdd type=alert or type=record to get only alerting or only recording rules.
curl http://localhost:9090/api/v1/alertsList active alerts
curl http://localhost:9090/api/v1/status/configShow loaded configurationReturns the YAML currently in use, without the comments of the original file.
curl http://localhost:9090/api/v1/status/flagsShow flag valuesLists the command-line flags the server was started with, every value as a string.
curl http://localhost:9090/api/v1/status/buildinfoShow build informationIncludes version, revision, branch and build date; the exact fields may change between versions.
curl http://localhost:9090/api/v1/status/tsdbTSDB cardinality statisticsHead block counts plus top series per metric name and label pair, 10 items each unless limit is set.
curl -XPOST http://localhost:9090/api/v1/admin/tsdb/snapshotSnapshot the TSDBNeeds --web.enable-admin-api; the snapshot lands in snapshots/ under the data directory.
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=up&match[]=process_start_time_seconds{job="prometheus"}'Delete matching seriesWithout start and end all data of the matched series is marked deleted; compaction frees the disk space later.
curl -XPOST http://localhost:9090/api/v1/admin/tsdb/clean_tombstonesFree deleted dataRemoves deleted samples from disk, typically after a delete_series call.

Notes

Rates and increases

Counters only go up, so their raw value says little; rate(http_requests_total[5m]) gives the per-second average over a five-minute window and adjusts for counter resets, which makes it the standard choice for graphs and alerts. increase(http_requests_total{job="api-server"}[5m]) reports the total added in the window and reads better in an alert message, but recording rules should stick to rate. irate(http_requests_total{job="api-server"}[5m]) looks only at the last two samples and suits volatile graphs. Gauges can go down, so use delta(cpu_temp_celsius{host="zeus"}[2h]) or deriv(v range-vector) for them instead.

Aggregating and ranking

sum by (job) ( rate(http_requests_total[5m]) ) adds series together keeping only the job label, and sum without (instance) (memory_consumption_bytes) is the same idea expressed by what to drop. Take the rate first and aggregate afterwards, or counter resets go unnoticed. count by (app) (instance_cpu_time_ns) counts series per group, topk(5, memory_consumption_bytes) keeps the five largest series with their labels, and histogram_quantile(0.9, sum by (job, le) (rate(http_request_duration_seconds_bucket[10m]))) estimates a percentile from classic histogram buckets, which must keep the le label when summed.

Checking configuration

absent(nonexistent{job="myjob"}) returns 1 when no series matches, which is how you alert on a job that has stopped reporting rather than one reporting bad values, while up == 0 catches targets whose scrape fails. On the operations side, promtool check config CONFIG-FILES validates the main configuration and the files it references, promtool check rules RULE-FILES validates recording and alerting rules, and promtool query instant SERVER EXPR runs a query from the shell for scripts and quick checks without opening the UI. A reload is then POST /-/reload (with --web.enable-lifecycle) or kill -s SIGHUP PID.

Questions

Which flag lets Prometheus reload its configuration over HTTP?

--web.enable-lifecycle. With it set, POST /-/reload re-reads the configuration and rule files and POST /-/quit shuts the server down gracefully. Without it, send kill -s SIGHUP PID to reload.

How can I check prometheus.yml before restarting the server?

Run promtool check config CONFIG-FILES, which also validates the files the configuration refers to. promtool check config --syntax-only CONFIG-FILES skips those referenced files, and promtool check rules RULE-FILES checks rule files on their own.

When should I use irate instead of rate?

irate(http_requests_total{job="api-server"}[5m]) uses only the last two samples, so it suits graphs of volatile, fast-moving counters. For alerts and slow-moving counters use rate(http_requests_total[5m]), which averages over the whole window.

Where did the retention flags go?

The 3.15 reference marks --storage.tsdb.retention.time and --storage.tsdb.retention.size as deprecated; set the storage.tsdb.retention.time and storage.tsdb.retention.size fields in the configuration file instead. With neither set, retention defaults to 15d.

Sources and checks

6 more source pages

Every row on this page was matched to a line in the documentation above on . A binding that could not be found there is not listed.

Found a wrong or missing key? Tell us and it gets checked against the source. How we verify describes the process and its limits.