feat(loki.source.podlogs): add tail_from_end argument to skip historical logs

[harshrai654]

PR Description

This PR adds a tail_from_end argument to loki.source.podlogs. When true, it starts reading only new logs for newly discovered pods, skipping any historical log backlog. This is useful for long-running pods to avoid processing old logs that might be rejected by Loki.

Which issue(s) this PR fixes

Fixes #3550

Notes to the Reviewer

Testing this feature requires interaction with the Kubernetes API, so it was tested manually against a local Minikube cluster.

1. A long-running pod was created in Minikube to generate a log history.
2. A PodLogs custom resource was applied to discover this pod.
3. A local Loki and Grafana stack was run via Docker Compose.
4. A local Alloy build was run with two configurations to verify the behaviour.

Test 1: tail_from_end = false (Default behaviour)

• Alloy correctly started tailing from the beginning of the log buffer.
• Logs confirmed that the stream start time was the Unix epoch.

ts=2025-06-22T12:30:34.814614Z level=debug msg="reconciling PodLogs" component_path=/ component_id=loki.source.podlogs.test key=alloy-test/test-podlogs
ts=2025-06-22T12:30:34.815355Z level=debug msg="found matching Pod" component_path=/ component_id=loki.source.podlogs.test key=alloy-test/test-podlogs pod=alloy-test/log-generator-2
ts=2025-06-22T12:30:34.815559Z level=debug msg="reconciling PodLogs" component_path=/ component_id=loki.source.podlogs.test key=alloy/test-podlogs
ts=2025-06-22T12:30:34.815721Z level=debug msg="reconciling PodLogs" component_path=/ component_id=loki.source.podlogs.test key=alloy/test-podlogs
ts=2025-06-22T12:30:34.815831Z level=debug msg="reconciling PodLogs" component_path=/ component_id=loki.source.podlogs.test key=alloy-test/test-podlogs
ts=2025-06-22T12:30:34.815876Z level=info msg="tailer running" target=alloy-test/log-generator-2:logger component_path=/ component_id=loki.source.podlogs.test
ts=2025-06-22T12:30:34.815938Z level=debug msg="found matching Pod" component_path=/ component_id=loki.source.podlogs.test key=alloy-test/test-podlogs pod=alloy-test/log-generator-2
ts=2025-06-22T12:30:34.829445Z level=info msg="opened log stream" target=alloy-test/log-generator-2:logger component_path=/ component_id=loki.source.podlogs.test "start time"=1970-01-01T05:30:00.000+05:30

Test 2: tail_from_end = true

• Alloy correctly started tailing from the current time, ignoring the log backlog.
• Logs confirmed the stream start time was time.Now()

ts=2025-06-22T13:19:49.316374Z level=debug msg="reconciling PodLogs" component_path=/ component_id=loki.source.podlogs.test key=alloy-test/test-podlogs
ts=2025-06-22T13:19:49.317023Z level=debug msg="found matching Pod" component_path=/ component_id=loki.source.podlogs.test key=alloy-test/test-podlogs pod=alloy-test/log-generator-5
ts=2025-06-22T13:19:49.3172Z level=debug msg="reconciling PodLogs" component_path=/ component_id=loki.source.podlogs.test key=alloy/test-podlogs
ts=2025-06-22T13:19:49.317319Z level=debug msg="reconciling PodLogs" component_path=/ component_id=loki.source.podlogs.test key=alloy-test/test-podlogs
ts=2025-06-22T13:19:49.317388Z level=debug msg="found matching Pod" component_path=/ component_id=loki.source.podlogs.test key=alloy-test/test-podlogs pod=alloy-test/log-generator-5
ts=2025-06-22T13:19:49.317398Z level=info msg="tailer running" target=alloy-test/log-generator-5:logger component_path=/ component_id=loki.source.podlogs.test
ts=2025-06-22T13:19:49.31746Z level=debug msg="reconciling PodLogs" component_path=/ component_id=loki.source.podlogs.test key=alloy/test-podlogs
ts=2025-06-22T13:19:49.323577Z level=info msg="opened log stream" target=alloy-test/log-generator-5:logger component_path=/ component_id=loki.source.podlogs.test "start time"=2025-06-22T18:49:49.317+05:30

• log-generator-pod specs:

apiVersion: v1
kind: Pod
metadata:
  name: log-generator
  labels:
    app: log-generator
spec:
  containers:
    - name: logger
      image: busybox
      args:
        - /bin/sh
        - -c
        - 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'

• PodLogs Custom Resource:

apiVersion: monitoring.grafana.com/v1alpha2
kind: PodLogs
metadata:
  name: test-podlogs
spec:
  selector:
    matchLabels:
      app: log-generator

Note:
A key part of this change was adjusting how the component detects a new pod without a saved position:

• The Problem: For a new pod, Positions.Get returns an offset of 0. The code then set lastReadTime = time.UnixMicro(0), resulting in the Unix epoch time (1970-01-01...). The check !lastReadTime.IsZero() was then used to see if a position existed. However, time.Time.IsZero() only returns true for the literal zero value (0001-01-01...), so the Unix epoch is not considered a "zero" time. This caused the logic to incorrectly assume a position existed, preventing the tail_from_end logic from ever running.
• The Solution: Added the condition to set lastReadTime value from offset only when it is not zero, So that lastReadTime remains a zero value if either there was an error getting the offset or offset value is returned as 0.

PR Checklist

• CHANGELOG.md updated
• Documentation added
• Tests updated
• Config converters updated

[clayton-cornell]

Minor doc suggestions