Core Concepts

To facilitate the configuration and use of Prometheus, we will briefly introduce the data model, metric types, and concepts such as instance and job in Prometheus.

Data Model

The data stored in Prometheus is time series, which is uniquely identified by the name of the metric and a series of labels (key-value pairs). Different labels represent different time series.

• Metric Name: This name should be semantic and is generally used to represent the functionality of the metric. For example, http_requests_total represents the total number of HTTP requests. The metric name consists of ASCII characters, numbers, underscores, and colons, and must satisfy the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*.

• Labels: Provides different dimensions for the same time series. For example, http_requests_total{method="Get"} represents all HTTP requests of the Get request. When method="post", it is a new metric. The keys in the label are made up of ASCII characters, numbers, and underscores, and must satisfy the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*.

• Sample: The actual time series, each series includes a float64 value and a millisecond-level timestamp.

• Format: Such as http_requests_total{method="POST",endpoint="/api/tracks"}.

Metric Types

The Prometheus client library mainly provides four main types of metrics, which are as follows:

1. Counter

A kind of cumulative metric, typical applications such as: the number of requests, the number of tasks completed, the number of errors occurred, etc. For example, query http_requests_total{method="get", job="kubernetes-nodes", handler="prometheus"} returns 8, query again after 10 seconds, then return 14.

2. Gauge

A regular metric, typical applications such as: temperature, number of running goroutines. For example: go_goroutines{instance="10.9.81.55", job="kubernetes-nodes"} returns 147, and 124 after 10 seconds.

3. Histogram

Can be understood as a histogram, typical applications such as: request duration, response size. It can sample, group and statistic the observation results. For example, when querying http_request_duration_microseconds_sum{job="kubernetes-nodes", handler="prometheus"}, the result is as follows:

4. Summary

Similar to Histogram, typical applications such as: request duration, response size. It provides count and sum functions of observable values. Provide percentile function, that is, trace results can be divided by percentage.

instance & job

Instance: A single scrape target, usually corresponding to a process.

Jobs: A group of the same type of instances

For example, a job of the api-server can include 4 instances:

• job: api-server

  • instance 1: 1.2.3.4:5670
  • instance 2: 1.2.3.4:5671
  • instance 3: 1.2.3.4:5672
  • instance 4: 1.2.3.4:5673

When scraping targets, Prometheus will automatically attach some labels to this scrape time series to better distinguish, such as: instance, job.