Monitor with Prometheus And Grafana

Prometheus 是一套开源的系统监控、报警、时间序列数据库的组合。Prometheus 基本原理是通过Http协议周期性抓取被监控组件的状态，而输出这些被监控的组件的Http接口为 Exporter。
Grafana 是一个可视化仪表盘，它拥有美观的图标和布局展示，默认支持 CloudWatch、Graphite、ES、InfluxDB、Mysql、PostgreSQL、Prometheus、OpenTSDB等作为数据源。
我们可以将 Prometheus抓取的数据，通过 Grafana 优美直观的展示出来。

Prometheus安装

mac

prometheus下载地址：Prometheus

$ cd /usr/local
$ tar -zxvf prometheus-2.12.0.darwin-amd64.tar.gz
$ ln -s prometheus-2.12.0.darwin-amd64 prometheus 
$ cd prometheus
$ ./prometheus

premethus启动初始界面如下：

linux

下载安装包并解压

1
2
3

cd /opt/third
tar -zxvf prometheus-2.16.0.linux-amd64.tar.gz
ln -s prometheus-2.16.0.linux-amd64 prometheus

修改配置文件prometheus.yml
分别配置job_name为prometheus和pushgateway

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['192.168.90.88:8890']

  - job_name: 'pushgateway'
    # 覆盖全局的scrape_interval
    scrape_interval: 5s
    static_configs:
    - targets: ['192.168.90.88:8891']
      labels:
         instances: pushgateway

启动Prometheus，Prometheus服务监听8890端口

1 2	cd /opt/third/prometheus nohup ./prometheus --storage.tsdb.retention.time="2d" --web.listen-address=":8890" &

注意：使用nohup启动

exporter安装

mysql-exporter

mysql-exporter下载地址: mysql-exporter

准备连接mysql的配置文件my.cnf:

[client]
host=localhost
port=3306
user=root
password=123456

$ cd /usr/local/prometheus_exporter
$ tar -zxvf mysqld_exporter-0.12.1.darwin-amd64.tar.gz
$ ln -s mysqld_exporter-0.12.1.darwin-amd64 mysqld_exporter
$ cd mysqld_exporter
$ ./mysqld_exporter --config.my-cnf ./my.cnf

在/usr/local/prometheus/prometheus.yml中添加：

1
2
3

- job_name: 'mysql'
   static_configs:
   - targets: ['localhost:9104']

命令行启动mysql-exporter如下：

打开http://localhost:9104/metrics, 可以看到返回了mysql的瞬时metrics:

node-exporter

node-exporter下载地址： node-exporter

$ cd /usr/local/prometheus_exporter
$ tar -zxvf node_exporter-0.18.1.darwin-amd64.tar.gz
$ ln -s node_exporter-0.18.1.darwin-amd64 node_exporter
$ cd node_exporter
$ ./node_exporter

在/usr/local/prometheus/prometheus.yml中添加：

1
2
3

- job_name: 'node'
    static_configs:
    - targets: ['localhost:9100']

命令行启动node-exporter如下：

Prometheus界面搜索‘node_load1’，简单的图形化展示:

pushgateway

mac

pushgateway下载地址：pushgateway

$ cd /usr/local
$ tar -zxvf pushgateway-0.9.1.darwin-amd64.tar.gz
$ ln -s pushgateway-0.9.1.darwin-amd64 pushgateway
$ cd pushgateway
$ ./pushgateway

在/usr/local/prometheus/prometheus.yml中添加：

- job_name: 'pushgateway'
   static_configs:
   - targets: ['localhost:9091']
     labels:
        instances: pushgateway

命令行启动pushgateway如下：

Prometheus上展示的targets如下:

linux

下载安装包并解压

1
2
3

cd /opt/third
tar -zxvf pushgateway-1.1.0.linux-amd64.tar.gz
ln -s pushgateway-1.1.0.linux-amd64 pushgateway

启动Pushgateway，Pushgateway服务监听8891端口

1 2	cd /opt/third/pushgateway nohup ./pushgateway --web.listen-address=":8891" &

注意：使用nohup启动

其他exporter

kafka-exporter git地址：Kafka-exporter
jmx-exporter git地址：JMX-exporter

Grafana安装

mac

$ cd /usr/local
$ wget https://dl.grafana.com/oss/release/grafana-6.3.5.darwin-amd64.tar.gz
$ tar -zxvf grafana-6.3.5.darwin-amd64.tar.gz
$ ln -s grafana-6.3.5.darwin-amd64 grafana
$ cd /usr/local/grafna/bin
$ ./grafana-server

命令行启动grafana如下：

grafana启动初始界面如下：

linux

下载安装包并解压

1 2	tar -zxvf grafana-6.6.2.linux-amd64.tar.gz ln -s grafana-6.6.2 grafana

启动Grafana，Grafana服务监听3000端口
1
nohup ./bin/grafana-server &

注意：使用nohup启动

Grafana-dashboards

开源的grafana-dashboards模板：

1
2
3

git clone https://github.com/percona/grafana-dashboards.git
git clone https://github.com/finn-no/grafana-dashboards.git
git clone https://github.com/rfrail3/grafana-dashboards.git

在grafana界面添加premethus数据源：

上传dashboard的开源json模板，从上方的git工程中获取：

mysql监控的dashboard如下：

node监控的dashboard如下：

Flink metrics

1.7.2

基于flink 1.7.2版本做监控试验。

1.需要在 /usr/local/flink-1.7.2/lib 下添加 flink-metrics-prometheus_2.11-1.7.2.jar

2.修改flink配置文件: /usr/local/flink-1.7.2/conf/flink-conf.yaml：

metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
metrics.reporter.promgateway.host: localhost
metrics.reporter.promgateway.port: 9091
metrics.reporter.promgateway.jobName: myJob
metrics.reporter.promgateway.randomJobNameSuffix: true
metrics.reporter.promgateway.deleteOnShutdown: false

上传一个flink任务jar包：