Documentation: Migrate from reporting-task to native Prometheus metrics

[sbernauer]

The Reporting task was removed from NiFi 2.x.x in #708.

However, /nifi-api/flow/metrics/prometheus needs authentication- which might be a blocker :( Relevant Slack thread

The steps required for the migration/configuration of the metrics endpoint (auth, authz) and how to connect e.g. Prometheus, should be documented.

[sbernauer]

A relevant message on NiFi Slack (cross-posting it here, as NiFi Slack looses messages):

I have set up prometheus with the simple-nifi cluster in k8s https://docs.stackable.tech/home/stable/nifi/getting_started/first_steps/. But I cannot access the metrics endpoint given in the docs https://docs.stackable.tech/home/stable/nifi/getting_started/first_steps/ with this prometheus configuration, because I get an 401 Unauthorized HTTP status. Has anyone an idea to solve it ? Or is this a bug ?

apiVersion: v1 
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: monitoring
  name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: discoverer
rules:
- apiGroups: [""]
  resources:
  - nodes
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-discoverer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: discoverer
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus:v2.33.4
        ports:
        - containerPort: 9090
        volumeMounts:
          - name: config
            mountPath: /etc/prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-server-conf
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
  - port: 9090
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  namespace: monitoring
  labels:
    name: prometheus-server-conf
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'nifi-metrics'
        metrics_path: '/nifi-api/flow/metrics/prometheus'
        scheme: https
        static_configs:
          - targets:
            - 'simple-nifi-node-default-0.simple-nifi-node-default.default.svc.cluster.local:8443'
        basic_auth:
          username: admin
          password: admin
        tls_config:
          insecure_skip_verify: true
---
apiVersion: v1
kind: Service
metadata:
  name: my-service-prometheus
  namespace: monitoring
spec:
  type: NodePort
  selector:
    app: prometheus
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 30090

[Maleware]

docs pr: #786

[Maleware]

After our talks, I did some research. I haven't found any ticket related to this in nifi jira or gh.

However, I was looking through prometheus and found this prometheus/prometheus#4827

It's an issure originating in 2018 and has seen some comments more or less once a year. There was one try to implement it, but closed due to inactivity. Another try to solve it is still open.

I believe with Nifi 2.x.x. this will be a more solid issue at their end and it might be that it's going to be solved anytime soon.

We could push it by actively contributing and/or showing interest.

[soenkeliebau]

We found this in last weeks planning:
https://lists.apache.org/thread/jbccodltqg0g7tdrosd4yg4k58mxr82k
https://lists.apache.org/thread/lxqhkzr741olrcbs0k3mbmyhz6t46g89

[Maleware]

We have a working example of the ServiceMonitor

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: scrape-label
  labels:
    stackable.tech/vendor: Stackable
    release: prometheus
spec:
  endpoints:
  - port: https 
    path: 'nifi-api/flow/metrics/prometheus'
    scheme: https
    interval: 5s
    tlsConfig:
      insecureSkipVerify: true
    authorization:
      credentials:
        key: "nifi_token"
        name: "nifi-authorization-secret"
        optional: false
      type: "Bearer"
    relabelings:
      - sourceLabels:
          - __meta_kubernetes_pod_name
          - __meta_kubernetes_service_name
          - __meta_kubernetes_namespace
          - __meta_kubernetes_pod_container_port_number
        targetLabel: __address__ 
        replacement: ${1}.${2}.${3}.svc.cluster.local:${4}
        regex: (.+);(.+?)(?:-metrics)?;(.+);(.+)
  selector:
    matchLabels:
      prometheus.io/scrape: "true"
  namespaceSelector:
    any: true
  jobLabel: app.kubernetes.io/instance

Which makes things betterish. However, still you need to provide a Bearer token for each pod. Hence this is more useful then defining a static target yet it's not there to solve our problem completely.

This rather unsatisfying outcome is a consequence of how NiFi handles authentication, as the cluster pods are not trusting the same authority.

As I see it now ( already spend a decent amount of time debugging it ) this can only be fixed at NiFi by allowing less strong authentication or even none for the metrics endpoint ( which would be the correct implementation imo )

[Maleware]

Needs to wait for https://github.com/stackabletech/decisions/issues/54 as this will change the regex expression for the service-name of the headless service.