Skip to content

Helm chart fails to create the sidecar in the ruler pod in microservices (distributed) mode #16529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mwtzzz opened this issue Mar 4, 2025 · 4 comments · Fixed by #17231
Closed

Helm chart fails to create the sidecar in the ruler pod in microservices (distributed) mode #16529

mwtzzz opened this issue Mar 4, 2025 · 4 comments · Fixed by #17231
Labels
area/helm component/ruler type/bug Somehing is not working as expected

Comments

@mwtzzz
Copy link

mwtzzz commented Mar 4, 2025 •
edited
Loading

Describe the bug
In SimpleScaleable mode, the helm chart correctly deploys a sidecar inside the backend statefulset when the following parameters are set.

loki:
  rulerConfig:
    enable_alertmanager_v2: true
    alertmanager_url: http://alertmanager.services.stellar-ops.com:9093
    storage:
      type: local
      local:
        directory: /rules

ruler:
  replicas: 0
  maxUnavailable: 0
  extraArgs:
    - '-config.expand-env=true'
  extraEnvFrom:
    - secretRef:
        name: loki-dev-secrets

sidecar:
  rules:
    enabled: true
    folder: /rules/1
    logLevel: debug

But in Distributed mode, it fails to create the sidecar, which results in a ruler pod that does not work properly, with the following errors:

level=error ts=2025-02-21T21:11:16.53272825Z caller=log.go:216 msg="error running loki" err="mkdir /rules: read-only file system

I can fix the ruler by inserting a sidecar myself. But it would be nice if this were handled automatically by helm.

To Reproduce
Steps to reproduce the behavior:

  1. loki 3.4.2 with helm chart loki-6.27.0
  2. cat loki-dev-values-distributed | helm template loki grafana/loki --values=- --namespace ops > loki-01-distributed.yaml
  3. look inside loki-01-distributed.yaml and you will see there is no sidecar container in the loki-ruler statefulset

Expected behavior
I expect a sidecar container inside the loki-ruler StatefulSet, in the same way as taken care of automatically by helm in deploymentMode: SimpleScaleable

Environment:

  • Infrastructure: kubernetes cluster deployed with kubeadm
  • Deployment tool: helm

Screenshots, Promtail config, or terminal output
N/A above description gives reproducible steps
@mwtzzz mwtzzz mentioned this issue Mar 4, 2025
Merged
6 tasks
@Jayclifford345
Copy link
Contributor

Hi @mwtzzz, I spoke with the team again on this one. Their advice in microservice mode is to use object storage for rules. They understand your use case but believe rules that need to be uploaded in this way can also be added to object storage through automation.

We can leave this ticket open and if more support comes for this issue we can bump the sidecar issue up the list to revisit. If a community member also works on the issue and adds a PR I can look at fast-tracking as well. Apologies it's not the answer you are looking for at this time. :(

@mwtzzz
Copy link
Author

mwtzzz commented Mar 18, 2025

@Jayclifford345 Thanks for the update, no worries I appreciate you looked into it, yes please leave the ticket open, it would be useful, not only to ourselves, but to others too.

@madaraszg-tulip
Copy link
Contributor

I am also interested in migrating our loki-sc-rules sidecar to microservices mode, as we need to deploy rules from kubernetes automatically, and as far as I can tell there's no automation to deploy them to the object storage.

@mwtzzz
Copy link
Author

mwtzzz commented Mar 25, 2025

I am also interested in migrating our loki-sc-rules sidecar to microservices mode, as we need to deploy rules from kubernetes automatically, and as far as I can tell there's no automation to deploy them to the object storage.

No, there isn't. You'd have to create your own custom tooling to POST api requests, which is pretty kludgey. Instead of doing that, I opted to not deploy the ruler from helm, instead I'm using a manifest like this:

# Source: loki/templates/ruler/statefulset-ruler.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: loki-ruler-custom
  namespace: ops-services
  labels:
    helm.sh/chart: loki-6.27.0
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: loki
    app.kubernetes.io/version: "3.4.2"
    app.kubernetes.io/component: ruler
    app.kubernetes.io/part-of: memberlist
  annotations:
    reloader.stakater.com/auto: "true"
    ignore-check.kube-linter.io/non-existent-service-account: "Argo creates the service account with helm"
spec:
  replicas: 1
  serviceName: loki-ruler
  selector:
    matchLabels:
      app.kubernetes.io/name: loki
      app.kubernetes.io/instance: loki
      app.kubernetes.io/component: ruler
  template:
    metadata:
      labels:
        helm.sh/chart: loki-6.27.0
        app.kubernetes.io/name: loki
        app.kubernetes.io/instance: loki
        app.kubernetes.io/version: "3.4.2"
        app.kubernetes.io/component: ruler
        app.kubernetes.io/part-of: memberlist
    spec:
      serviceAccountName: loki
      terminationGracePeriodSeconds: 300
      containers:
        - name: loki-sc-rules
          image: "kiwigrid/k8s-sidecar:1.30.0"
          imagePullPolicy: IfNotPresent
          env:
            - name: METHOD
              value: WATCH
            - name: LABEL
              value: "loki_rule"
            - name: FOLDER
              value: "/rules/1"
            - name: RESOURCE
              value: "both"
            - name: WATCH_SERVER_TIMEOUT
              value: "60"
            - name: WATCH_CLIENT_TIMEOUT
              value: "60"
            - name: LOG_LEVEL
              value: "debug"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: sc-rules-volume
              mountPath: "/rules/1"
        - name: ruler
          image: docker.io/grafana/loki:3.4.2
          imagePullPolicy: IfNotPresent
          args:
            - -config.file=/etc/loki/config/config.yaml
            - -target=ruler
            - -config.expand-env=true
          ports:
            - name: http-metrics
              containerPort: 3100
              protocol: TCP
            - name: grpc
              containerPort: 9095
              protocol: TCP
            - name: http-memberlist
              containerPort: 7946
              protocol: TCP
          envFrom:
            - secretRef:
                name: loki-prd-secrets
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            readOnlyRootFilesystem: true
          readinessProbe:
            httpGet:
              path: /ready
              port: http-metrics
            initialDelaySeconds: 30
            timeoutSeconds: 1
          volumeMounts:
            - name: config
              mountPath: /etc/loki/config
            - name: runtime-config
              mountPath: /etc/loki/runtime-config
            - name: data
              mountPath: /var/loki
            - name: tmp
              mountPath: /tmp/loki
            - name: sc-rules-volume
              mountPath: "/rules/1"
          resources:
            {}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/component: ruler
            topologyKey: kubernetes.io/hostname
      volumes:
        - name: sc-rules-volume
          emptyDir: {}
        - name: config
          configMap:
            name: loki
            items:
              - key: "config.yaml"
                path: "config.yaml"
        - name: runtime-config
          configMap:
            name: loki-runtime
        - name: tmp
          emptyDir: {}
        - name: data
          emptyDir: {}

@madaraszg-tulip madaraszg-tulip mentioned this issue Apr 15, 2025
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/helm component/ruler type/bug Somehing is not working as expected
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(helm): Enable rules sidecar for the ruler pods madaraszg-tulip/loki
4 participants
@JStickler @Jayclifford345 @mwtzzz @madaraszg-tulip