Goal: explain how app logs from KOB end up in Splunk, in plain language, no configs.
One‑sentence summary
Apps in KOB write their logs as usual → the PLX agent collects them on each node, adds Kubernetes context (service, pod, namespace), converts them into a clean JSON record, and sends them securely to Splunk where we search, alert, and report.
How it flows (5 steps)
- Your app runs in a container and writes logs (stdout/stderr or a file). No special SDK needed.
- PLX agent watches for new log lines on that node.
- Context is attached: cluster, namespace, pod, container, node, and basic metadata like timestamp and severity.
- Log is shaped into a JSON event (uniform keys so searches & dashboards work the same for every team).
- Event is delivered to Splunk over an encrypted channel; Splunk stores it in the right index and makes it searchable in seconds.
What you see in Splunk
- One place to search by service/namespace/request id.
- Dashboards: error rates, slow endpoints, noisy pods, release impact.
- Alerts on meaningful patterns (spikes in errors, timeouts, crash loops).
Why this matters
- Faster incident response: clear, structured logs cut time to root cause.
- Consistency: every team’s logs look the same; less ad‑hoc parsing.
- Compliance: retention, access control, audit trails handled centrally.
- Cost control: low‑value noise can be filtered before it hits Splunk.
What app teams do
- Keep logging to standard output, use levels (INFO/WARN/ERROR), avoid secrets.
- Add helpful IDs (e.g., request/trace id) to tie logs to a user action.
- Prefer structured messages (key=value in the message); the agent will wrap into JSON.
What platform team does
- Keep the PLX agent healthy on every node.
- Define the JSON schema and routing (which logs go to which index).
- Enforce redaction rules and retention, and maintain dashboards/alerts.
Guardrails
- Security: encrypted transport, token‑based access, namespace isolation.
- Privacy: block credentials/PII in logs; redaction at the edge.
- Reliability: buffering & retries so short Splunk outages don’t drop data.
Rollout snapshot
- Point apps to stdout; 2) enable PLX agent cluster‑wide; 3) agree JSON keys & naming; 4) publish shared dashboards & alerts; 5) add runbooks.
Bottom line
You build features and log sensibly; PLX moves those logs to Splunk as uniform JSON with Kubernetes context; managers get one truth source for search, alerts, and reports.