K8SEC / TETRAGON FIELD INVESTIGATION
Malware behavior is a sequence of qualified observations
A file hash, syscall hit or suspicious process name is a lead. The provided Tetragon operator draft models 31 observe-only policy candidates and eight cross-stream correlation candidates. This article extracts the transferable detection logic; it does not publish the complete operator bundle, assert that the policies are loaded on any production node, or treat its synthetic fixtures as measured recall.
Critical distinction: Tetragon Post emits an event. It does not quarantine, delete, block, or prove “system secure.” Enforcement requires a separately reviewed action and policy mode, and even a kill cannot reverse prior API writes or credential use.
Solid sequence = records present in the synthetic example. A displayed join is conditional; it does not prove causation. These are research diagrams, not live telemetry.
Interact with four evidence outcomes
Choose a sequence to change the graph and inspect its join conditions, counterexample, and defensible conclusion. All cases are synthetic.
R03 · staged execution is a hypothesis, not a malware verdict
- Observed
- In the synthetic trace, N03 sees an execve request from a writable path, N04 sees memfd creation, and N06 sees a process-name change in that order within 180 seconds.
- Join gate
- Require the same cluster and Pod UID, event-time container identity, then inspect exec IDs and parent/child lineage. The shared Pod alone cannot prove one process performed all three actions.
- Benign alternative
- A compiler or update worker can stage a temporary executable and allocate anonymous files. A Java process can rename itself for legitimate diagnostics.
- Next evidence
- Pair entry with result, inspect image digest and executable provenance, compare with the approved workload baseline, and establish whether the anonymous object was executed.
R01 · credential access and accepted cross-namespace placement
- Observed
- A Tetragon file-permission event is followed by an API audit record for an accepted, non-dry-run cross-namespace Pod create within 120 seconds.
- Join gate
- Resolve a unique source Pod at request time, verify direct source-IP attribution rather than proxy/NAT inference, and match the effective ServiceAccount. A shared SA name or close timestamps alone are insufficient.
- Still unknown
- File access is not proof that a token was copied or used by this process. An accepted API request is not proof that the Pod reached Running or that code executed there.
- Negative control
- If the source IP maps to several Pods, the caller is a controller, the API operation is denied, or the request is dry-run, do not promote this to the cross-plane candidate.
R05 · three denied requests do not make a container escape
- Observed
- A same-Pod sequence requests setns, mount and a BPF operation within 90 seconds. In this control fixture all results are denied.
- Decision
- Report boundary-sensitive probing and the negative results. Do not call host access, kernel compromise, or successful privilege escalation.
- Next evidence
- For a success claim, establish return values, effective capabilities, relevant namespace IDs, the host object reached, and an independently observed host-side effect.
- Coverage caveat
- N21 raw syscall mapping can duplicate these signals; duplicate events do not count as independent stages. A missing event may reflect hook or collector loss.
Approved build worker · an expected signal is a negative control
- Observed
- A signed, scheduled build job executes a temporary build helper under an expected parent and image digest.
- Decision
- The N03 signal still exists, but the owner-approved job and executable provenance explain it. Keep the raw event; suppress only the narrow workload context after review.
- Regression test
- Change the parent, image digest, namespace or time window. The same broad exception must no longer match. If it does, the exception is unsafe.
- Limit
- An approved identity does not make every later action benign; a successful sensitive API operation needs its own authorization review.
Sensor-policy anatomy: from hook to testable claim
| Candidate family | What the hook may show | What it cannot establish alone | Required control |
|---|---|---|---|
N03 · temporary executionsys_execve | Requested executable path and process context; correlate with process-exec and result. | That the payload ran to completion or was malicious. | Known build/update job, image digest, parent lineage, failed exec and pathname reuse. |
N04 · anonymous filesys_memfd_create | Creation request for a memory-backed file object. | That the object held code, was executed, or evaded disk forensics. | JIT and shared-memory workloads; object provenance and a later independently observed exec. |
N08/C03 · credential filesecurity_file_permission | A security-relevant access check on a projected credential file. | Token theft, token contents, or that this process made an API call. | Expected client use, actual access result, token audience and direct caller attribution. |
N11/C04 · runtime socketsecurity_socket_connect | Attempted connection to a container-runtime control surface. | The CRI method or whether another container changed. | Socket mount inventory, connection result, runtime logs and resulting workload state. |
N15/C06, N16/C07 · namespace and mountsys_setns, sys_mount | Boundary-sensitive requests. | Host escape or a successful mount. | Errno, effective capabilities, namespace IDs, target object and host-side effect. |
| C08–C10 · BPF, module, kexec | Kernel-control attempts under the policy’s supported hook. | Kernel compromise or even completed operation. | Node/kernel support, return values, authorized maintenance, module/BPF inventory. |
These names are from the supplied operator draft. Hook availability, argument types and event fields are version/kernel dependent. Confirm the installed Tetragon CRD and TracingPolicy reference before applying any policy. The draft uses Post; the table is not an enforcement recommendation.
One observe-only policy excerpt—and the tests it needs
This excerpt illustrates how a temporary-path execve candidate is selected. It is deliberately broad and not deployment-ready; scope it to a disposable workload and verify hook loading and field semantics against the installed Tetragon release. A path match can be noisy and does not establish execution success.
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: k8sec-temp-exec-observe-example
spec:
kprobes:
- call: sys_execve
syscall: true
args:
- index: 0
type: string
selectors:
- matchArgs:
- index: 0
operator: Prefix
values: ["/tmp/"]
matchActions:
- action: Post
Canary contract: one expected event for an authorized temporary helper, one for an unexpected helper, and no malware verdict for either without process/result context. Test a failed exec, an approved build image, an unrelated Pod, a renamed path, disabled hook or dropped event, and repeat after every sensor or kernel upgrade. Record policy load status, event loss, false positives per workload-hour and reviewer disposition.
Eight correlation candidates—not eight automatic “detections”
| ID | Required sequence | Adjudication boundary |
|---|---|---|
| R01 | Credential-file access → accepted cross-namespace Pod creation. | Unique, time-valid source-Pod attribution; verify effective SA and resulting Pod state. |
| R02 | Accepted API exec → shell or interpreter attempt in target Pod. | Target Pod UID and process lineage; approval and command result remain separate. |
| R03 | Temp exec → memfd creation → process-name change. | Same Pod is insufficient; reconstruct parent/child execution and benign builders. |
| R04 | Persistence-file permission check → file unlink. | Show actual mutation, consuming service and result; a check is not persistence. |
| R05 | setns → mount → kernel-control attempt. | Return values, namespace identity and host effect; denied probes are not escape. |
| R06 | Credential-file access → accepted Secret read. | R01 attribution plus independent transfer evidence before claiming exfiltration. |
| R07 | Permission change → OpenShift route or build action. | Effective API actor, permission diff and accepted object change; runtime corroboration optional. |
| R08 | Excluded-namespace SA → cross-namespace Pod create. | Audit-led fallback; credential namespace is not physical process origin. |
Deduplicate overlapping N/C policy families and N21 raw-syscall entries before counting stages. Preserve missing telemetry as unknown, never “clean.” The operator draft’s 120–600-second windows are teaching defaults; production windows require measured latency, clock skew and replay tests.
A real-world research contrast: AI infrastructure honeypots
Wiz’s 90-day AI-infrastructure honeypot study documents MCP gateway and configuration-test abuse, blind prompt-injection attempts that reached shell-capable agents, and environment-aware post-exploitation aimed at AI-service credentials. Their captured telemetry and published interpretation are a separate external case study, not observations from this K8SEC fixture or proof that the same campaign targeted your cluster. For a Tetragon deployment, the reusable detection question is: did the AI service spawn an unexpected descendant, did a kernel operation succeed, and can the action be attributed to a workload identity and a later API effect? Do not promote a DNS callback, deleted staging pathname or suspicious directory name into a confirmed exfiltration or successful compromise without independent outcome evidence.
References: Tetragon TracingPolicy schema · Tetragon execution observability · Kubernetes audit events · Wiz Threat Research, August 2026.
01Reverse engineering should produce a behavior contract
A suspicious ELF file in a container is an artifact. A process changing memory permissions is an event. A Kubernetes API request is an action under an authenticated identity. None, alone, proves the entire incident. The investigator’s job is to connect these observations without inventing the links between them.
This guide follows a fictional compromised application workload from binary triage to process memory, kernel operations and cluster evidence. The diagrams and replay records are synthetic teaching material, not a captured malware campaign. No malicious executable, exploit, real memory dump or credential is distributed or executed by this page.
Four different conclusions need different evidence
Write a compact behavior contract before writing a detector: subject → operation → target → prerequisites → result → uncertainty. Add the benign alternative you must rule out. “May allocate executable memory” is a static hypothesis; “permission change succeeded for this mapping” is a dynamic observation; “malware executed from it” requires additional evidence.
02Inspect the specimen without granting it your environment
Preserve the original acquisition context: image digest, registry or volume origin, file hash, collection time, operator, source workload and any transformations during export. An image tag is mutable; a sample filename is not identity. Keep a read-only original and work on a separately hashed copy with restricted access.
Static triage: questions before tool output
Identify the ELF architecture, interpreter, entry point and loadable segments. Review dynamic dependencies, exported and imported symbols, strings and cross-references. Then follow data flow around relevant call sites: where do a target PID, pathname, destination or requested memory permission originate? An imported symbol may never run; a missing import does not prove the capability is absent. Statically linked code and direct syscall use can change the evidence available.
sha256sum -- specimen.elf
file -- specimen.elf
readelf --file-header --program-headers --dynamic -- specimen.elf
strings -a -n 8 -- specimen.elfThese are parser operations, not a guarantee of safety: malformed input can attack analysis tools. Use maintained tools in a disposable analysis VM without production credentials, mounted container-runtime sockets or access to the cluster. Do not run ldd against an untrusted executable; use non-executing dependency inspection instead. Sources: GNU readelf, ldd safety warning.
A disassembler/decompiler such as Ghidra helps reconstruct control flow. Treat reconstructed C as an analyst aid, not the original source. Record function offsets, conditions, supporting instructions and confidence. Keep “possible,” “observed in the lab” and “observed in the incident” as different labels.
Dynamic analysis: reproduce the behavior, not the damage
Real specimen execution belongs in an approved malware-analysis environment with explicit egress containment, synthetic data and a recovery plan. A normal application container is not a sufficient malware sandbox. For initial detector development, use harmless behavior emulators and the JSON fixtures below instead of executing unknown code.
Define each run’s inputs, architecture, kernel, policy, clock, network simulation and sensor versions. Collect a baseline without the suspicious behavior, then a controlled positive case and a denied negative control. One quiet run does not establish benignness: the relevant branch may depend on missing inputs, timing or environment. Do not turn that uncertainty into a guide for bypassing analysis.
03Understand the process memory before calling it malicious
Virtual addresses belong to a process address space; they are not direct physical addresses or globally unique objects. The following is a schematic, not a fixed ordering or scale. ASLR, architecture, loader and runtime behavior affect actual placement.
Open a region to inspect its security meaning. This is not a memory dump.
ELF code / shared librariesOften r-x
Executable mappings can be file-backed. Verify their origin, image provenance and actual mapping permissions; do not assume all executable pages came from the expected image.
Data, heap and anonymous regionsOften rw-
Writable data is ordinary application state. Its presence is not malware; investigate which identity or operation changed a region and why.
Generated-code regionRW → RX (example)
A JIT or an unexpected loader can request a transition. Permission change, execution of bytes and malicious purpose are three separate claims.
Thread stacksCommonly rw-
Each thread has stack-related state. Actual layouts and permissions vary; an unusual permission needs context rather than automatic attribution.
Kernel-managed page tables / physical pagesNot a readable userspace region
The kernel and hardware manage translation and access. A process virtual address is not a physical address or a container-to-host escape.
PROCESS ADDRESS-SPACE BOUNDARY
Another process’s mappings require a separate identity and authorized observation.
/proc/<pid>/maps describes current mappings, permissions and backing information, subject to access checks. A blank pathname can indicate anonymous memory; a deleted file can remain associated with a mapping. A mapping snapshot is not a history of writes or executed instructions. Correlate address ranges with process lifetime and collection time, not merely a PID. See the maps format.
RW → RX is an observation, not a malware signature
mprotect requests protection changes for pages in an address range. A successful result establishes that the request succeeded under that interface’s semantics; it does not prove that the CPU later executed those bytes. Writable-and-executable memory and transitions to executable permissions deserve context, particularly in workloads that should not generate code. JIT runtimes can legitimately change memory protections. Architecture-specific permission behavior also matters. Source: mprotect semantics.
memfd_create produces an anonymous file object with a descriptor. It can support legitimate shared-memory and loading use cases. Its creation is not evidence of execution, absence of forensic traces or kernel compromise. Track object lifetime, writes, executable use and process identity; a descriptor number may later refer to a different object. Source: memfd_create.
| Observation | Defensible statement | Still unproven |
|---|---|---|
| Anonymous executable mapping | A region is executable at collection time. | Its origin, contents, execution history or malicious purpose. |
| Successful cross-process memory write | The specific interface reports a transfer into another process. | That control flow was redirected or the written bytes executed. |
| Deleted mapped executable | A mapping references a file no longer named at that path, with format caveats. | That deletion was malicious; legitimate upgrades can replace files. |
| High CPU and repeated memory activity | The workload consumes resources and performs those observed operations. | Cryptomining attribution; batch processing and compilation are alternatives. |
Mapping identity is difficult: regions split, merge, unmap and reuse addresses. A production collector needs interval and lifecycle handling, not a permanent identity inferred from one start address. The exercise uses an explicit synthetic mapping ID to keep that requirement visible.
04Translate capability into kernel behavior and telemetry
Malware typically requests ordinary kernel services through the same interfaces as legitimate software. The kernel does not receive a “malicious intent” flag. Detection depends on the operation’s arguments, subject, object, outcome and workload context.
| Capability hypothesis | Observe and correlate | Negative control |
|---|---|---|
| Memory-backed execution | Object creation and provenance, executable transition or successful image execution, lineage and workload purpose. | Approved JIT or launcher; a permission request that fails. |
| Process inspection or injection | Specific ptrace operation or cross-process transfer, source/target identities, return result and subsequent effects. | Authorized debugger; unrelated target; zero or failed transfer. |
| Credential discovery | Resolved sensitive object and successful access under the correct mount namespace; subsequent independently established use. | Expected workload credential use; attempted access denied by policy. |
| Persistence or policy tampering | File or resource mutation, prior/after state, responsible identity and maintenance approval. | Authorized deployment or configuration management; failed mutation. |
| Host or kernel access | Actual boundary reached, effective capabilities, mounts/devices, relevant policy decisions and resulting host effect. | Capability absent, operation denied, or object confined to the container. |
Syscall entry records a request. A paired completion supplies a result. Tracepoints and function probes expose selected kernel activity; LSM hooks operate at security-relevant points. An eBPF observer is not automatically an enforcement policy, and a traced function is not a portable ABI. Review sensor attachment and coverage on each supported node image. See event tracing and BPF LSM programs.
Do not infer that file contents left the cluster just because a process opened a file and later connected to a destination. Confirm object access, process or explicit IPC lineage, connection outcome and evidence supporting actual transfer. Likewise, shell execution is useful context but neither necessary nor sufficient for compromise.
05Build a Kubernetes detection architecture with explicit joins
Solid arrows represent conceptual data flow, not a proven attack chain.
Workload execution plane
- UserspaceContainer process + memoryELF, mappings, credentials, process lifetime. Container isolation is not a separate kernel.
- User / kernel boundaryKernel services + policyMemory, VFS and network operations; relevant permissions, seccomp and LSM decisions.
- Observation boundaryNode runtime sensorSupported hooks and event outcomes. Sensor privilege and coverage need separate review.
Independent control-plane evidence
- API requestAuthenticated callerA workload or operator requests a resource operation. A shared ServiceAccount is not a unique process ID.
- AuthorizationAPI server + audit policyRelevant request stages and result. Audit scope depends on policy; it is not a syscall recorder.
- Evidence collectionOff-node audit storageControlled access, retention and integrity. Do not collect Secret bodies by default.
↓ Independent runtime records + event-time metadata · API audit records ↓
Candidate cross-plane relationship: INFERRED until supported by evidence.
Correlation and investigation
Join stable identities, explicit request/object relationships and bounded time. Preserve missing data, competing explanations and source references. Process memory → API action is not an automatic causal edge.
The node collector must resolve host process and container-runtime identities to the pod UID, container ID, namespace, owner and image digest at event time. Pod names can be reused; containers restart; shared process namespaces and sidecars complicate assumptions. Preserve original identity fields when enrichment arrives late, and mark unresolved context rather than inventing a workload match.
Kernel operations and API actions are separate evidence planes
Kubernetes audit records describe API requests according to the configured audit policy and stages. They do not record every syscall, every file read or all in-container activity. Avoid collecting Secret request/response bodies or unnecessary sensitive content. Source: Kubernetes auditing.
Record audit identity, verb, object reference, authorization/result context and event time. A ServiceAccount can be shared by multiple pods; its name alone cannot attribute an API request to one Linux process. A source IP may identify a proxy or translated address. Treat a join from runtime activity to API activity as a hypothesis until stronger request, credential-provenance or network evidence supports it.
namespace=training
pod=payments-demo
kubectl get pod "$pod" -n "$namespace" -o jsonpath='{.metadata.uid}{"\n"}{.spec.nodeName}{"\n"}{.spec.serviceAccountName}{"\n"}{range .status.containerStatuses[*]}{.name}{"\t"}{.containerID}{"\t"}{.imageID}{"\n"}{end}'This retrieves current metadata, not historical attribution or Secret values. A terminated/replaced pod may no longer be recoverable through this query. Capture runtime metadata continuously under an approved retention policy rather than relying on a post-incident name lookup.
Choose sensors for coverage, not branding
Falco fields support rule construction from available events and context; availability depends on source and version. Tetragon provides eBPF-based security observability and policy-dependent enforcement capabilities. Neither product name proves your required event is collected or blocked. Maintain a tested kernel/architecture/runtime/sensor matrix, attachment health checks, event-loss metrics and a controlled canary workload.
K8SEC Agent contributes read-only resource and RBAC graph context. It does not collect process memory, Secret values, execute in pods, mutate resources or exploit findings. Keep any separately approved node sensor and forensic acquisition authority distinct from the production K8SEC Agent.
06Engineer detections that survive benign explanations
Detection A · unexpected memory-backed execution
Falco’s upstream rules include a memory-backed execution analytic using proc.is_exe_from_memfd. It concerns executable provenance, not every use of anonymous memory. The original condition below is a candidate local rule, dependent on Falco’s default macros and field support. It has not been validated against a running Falco installation here; load and validate it with the matching release in an isolated lab before rollout. Reference: upstream Falco rules.
- rule: K8SEC Container Memory Backed Execution Review
desc: Review memory-backed executable provenance in a container
condition: spawned_process and container and proc.is_exe_from_memfd=true
output: >-
Memory-backed execution requires review
(pid=%proc.pid container=%container.id executable=%proc.exepath)
priority: WARNING
tags: [k8sec, process, container]Test the rule with an approved benign emitter, ordinary disk-backed execution, an expected legitimate launcher and missing metadata. Verify that required macros are loaded, fields are populated and alerts route correctly. Exceptions must be independently managed and scoped to workload identity, executable provenance, purpose and expiry; a process name is not an authorization boundary.
Detection B · executable-memory transition with context
The interactive investigation evaluates a different, narrower signal: a normalized RW-to-RX permission transition. It is not the Falco rule above and does not claim memory-backed execution. Compare an unexpected transition, an approved JIT, a denied request and lost telemetry.
One memory transition · four evidence outcomes
Observed means present in this synthetic fixture. Approval and collector coverage are independent teaching inputs, not verified by the page.
All synthetic cases are available below. No process memory is accessed.
Unexpected executable mapping
- Process / mapping identityRequest RW → RXmprotect request against mapping-42 in an identified synthetic process and container.
- Kernel resultObserved · RW → RXThe outcome must match the invocation, mapping and process lifetime.
- InterpretationInferred · requires reviewObtain executable provenance, workload expectations and independent evidence of code execution. Do not call a permission transition an injection.
- Evaluator result · not a malware verdict
- review-transition: An unexpected executable mapping warrants investigation; execution and malware are not established.
Inspect this normalized fixture
{
"id": "unexpected",
"label": "Unexpected executable mapping",
"approval": "unapproved",
"coverage": "complete",
"events": [
{
"phase": "request",
"timeMs": 10,
"callId": "unexpected-call",
"process": "node-a/boot-7/pid-820/start-9",
"podUid": "pod-uid-a",
"containerId": "container-a",
"mappingId": "mapping-42",
"operation": "mprotect",
"before": "rw-",
"requested": "r-x"
},
{
"phase": "result",
"timeMs": 12,
"callId": "unexpected-call",
"process": "node-a/boot-7/pid-820/start-9",
"podUid": "pod-uid-a",
"containerId": "container-a",
"mappingId": "mapping-42",
"operation": "mprotect",
"result": 0,
"after": "r-x"
}
]
}Approved JIT workload
- Process / mapping identityRequest RW → RXmprotect request against mapping-42 in an identified synthetic process and container.
- Kernel resultObserved · RW → RXThe outcome must match the invocation, mapping and process lifetime.
- InterpretationContext · approved JITValidate the approved workload, operation and expiry. The same low-level transition does not justify a global exception for the process name.
- Evaluator result · not a malware verdict
- expected-transition: Approved context explains this transition, not the safety of every action by this process.
Inspect this normalized fixture
{
"id": "jit",
"label": "Approved JIT workload",
"approval": "approved",
"coverage": "complete",
"events": [
{
"phase": "request",
"timeMs": 10,
"callId": "jit-call",
"process": "node-a/boot-7/pid-820/start-9",
"podUid": "pod-uid-a",
"containerId": "container-a",
"mappingId": "mapping-42",
"operation": "mprotect",
"before": "rw-",
"requested": "r-x"
},
{
"phase": "result",
"timeMs": 12,
"callId": "jit-call",
"process": "node-a/boot-7/pid-820/start-9",
"podUid": "pod-uid-a",
"containerId": "container-a",
"mappingId": "mapping-42",
"operation": "mprotect",
"result": 0,
"after": "r-x"
}
]
}Permission change denied
- Process / mapping identityRequest RW → RXmprotect request against mapping-42 in an identified synthetic process and container.
- Kernel resultObserved · EACCESThe outcome must match the invocation, mapping and process lifetime.
- InterpretationDisproven · this requested successThe modeled request failed; attribute the denial through policy evidence. Do not infer that all other paths are blocked.
- Evaluator result · not a malware verdict
- denied-attempt: This normalized request was denied. The responsible control needs separate evidence.
Inspect this normalized fixture
{
"id": "denied",
"label": "Permission change denied",
"approval": "unapproved",
"coverage": "complete",
"events": [
{
"phase": "request",
"timeMs": 10,
"callId": "denied-call",
"process": "node-a/boot-7/pid-820/start-9",
"podUid": "pod-uid-a",
"containerId": "container-a",
"mappingId": "mapping-42",
"operation": "mprotect",
"before": "rw-",
"requested": "r-x"
},
{
"phase": "result",
"timeMs": 12,
"callId": "denied-call",
"process": "node-a/boot-7/pid-820/start-9",
"podUid": "pod-uid-a",
"containerId": "container-a",
"mappingId": "mapping-42",
"operation": "mprotect",
"result": -1,
"after": "rw-",
"error": "EACCES"
}
]
}Collector coverage gap
- Process / mapping identityRequest RW → RXmprotect request against mapping-42 in an identified synthetic process and container.
- Kernel resultMissing · result recordThe outcome must match the invocation, mapping and process lifetime.
- InterpretationUnknown · actual outcomeA lost result cannot be reconstructed from intent. Restore coverage and seek independent evidence without labeling the request safe or successful.
- Evaluator result · not a malware verdict
- unknown: Incomplete coverage or event pair.
Inspect this normalized fixture
{
"id": "gap",
"label": "Collector coverage gap",
"approval": "unknown",
"coverage": "loss",
"events": [
{
"phase": "request",
"timeMs": 10,
"callId": "gap-call",
"process": "node-a/boot-7/pid-820/start-9",
"podUid": "pod-uid-a",
"containerId": "container-a",
"mappingId": "mapping-42",
"operation": "mprotect",
"before": "rw-",
"requested": "r-x"
}
]
}Detection C · runtime anomaly plus cluster action
runtime lead = unexpected behavior + stable process and workload identity
api lead = sensitive API action + authenticated actor + recorded result
candidate join = bounded time + supported workload/request relationship
if only ServiceAccount name or timing matches:
mark relationship INFERRED; request corroboration
if telemetry is incomplete:
mark coverage DEGRADED; retain known observations
if action is independently linked and outside approved purpose:
open investigation with raw event references and alternativesNever turn this into “memory anomaly → cluster-admin compromise” by default. A denied API action is not a successful resource change. A permitted API action is not automatically malicious. Assess the request, authenticated authority, resource scope, outcome and independent approval.
07Replay a detection contract and challenge its assumptions
The downloadable JavaScript evaluator accepts a synthetic request/result pair. It requires matching invocation, process, pod UID, container and mapping IDs; finite ordered timestamps within a one-second teaching window; and consistent requested/observed permission states. It outputs review-transition, expected-transition, denied-attempt or unknown. None means “malware confirmed.”
The fixture uses normalized wrapper-style results. A production ingestion adapter must test its sensor’s errno representation, process/mapping lifetimes, partial-range semantics, event ordering and lost-event behavior. The approval and coverage fields are supplied teaching context, not facts this evaluator can authenticate.
node --input-type=module -e '
import {readFileSync} from "node:fs";
import {evaluateMemory} from "./memory-behavior-v1.mjs";
const fixture = JSON.parse(readFileSync("./memory-behavior-fixture-v1.json", "utf8"));
for (const trace of fixture.cases) console.log(trace.id, evaluateMemory(trace));
'Download memory evaluator v1 · JavaScript · Download synthetic memory evidence v1 · JSON
Student challenge and reviewer standard
- Explain why the unexpected and approved traces have the same low-level transition but different contextual conclusions.
- Change only the result record’s process identity. The evaluator must not join it to the original request.
- Introduce a different pod UID, container ID or mapping ID. Explain which boundary the invalid join would hide.
- Remove the result or declare collector loss. Show why UNKNOWN is more defensible than “blocked.”
- Present the denial alongside the synthetic success case. State what additional policy evidence would attribute the denial.
- Write a one-paragraph finding that reports the transition without claiming instruction execution or exfiltration.
Before production use, add a trusted sensor adapter, interval-aware mapping tracking, deduplication, bounded late-event handling, signed or access-controlled policy context and operational evaluation. Measure false positives per workload-hour and investigation time. Evaluate recall only against a stated corpus with known labels; passing four fixtures is not broad malware-detection coverage.
08Preserve evidence, contain deliberately and validate the fix
Memory acquisition can interrupt a workload and capture credentials, tenant data and unrelated processes. Obtain authority, choose a supported acquisition method and document its effects before collecting. Do not deploy an improvised privileged collector or load a kernel module into production merely to complete a checklist. Avoid public malware-analysis upload services for sensitive samples or dumps unless disclosure is explicitly approved.
For offline Linux memory analysis, Volatility 3 requires matching symbol information; Linux kernel banner/configuration differences matter. “Same major kernel version” is not an adequate compatibility check. Preserve acquisition hashes, tool versions and symbol provenance, and record unsupported analysis as a limitation rather than silently guessing structures.
Containment follows the demonstrated boundary
If the compromise is confined to one workload, coordinate evidence preservation, service continuity, isolation and replacement from a trusted digest. If credential misuse is established, rotate or revoke the affected credentials through their supported lifecycle and review related actions. Deleting a pod alone does not reverse completed API operations or invalidate every copied credential. If node integrity is suspect, escalate to node-level incident response and trusted recovery; do not rely solely on that node’s clean telemetry.
Reduce recurrence through least-privilege workload identities, restricted host exposure and runtime confinement. The Kubernetes Restricted Pod Security Standard provides admission constraints such as non-root execution, capability restrictions and seccomp requirements. Admission policy is not runtime malware detection, and applying a label does not prove existing workloads were remediated. Validate the running pod and the relevant negative controls.
Reverse engineering
Hash, provenance, architecture, imports, control/data flow and untested branches. Static capability is not observed impact.
Memory
Backing object, permissions, mapping lifetime and process identity. Executable permission is not proof of execution.
Kernel
Request, policy decision, result and independent effect. Record attachments and event loss.
Kubernetes
Pod UID, container ID, image digest and node context. API audit is a separate plane; shared identity is not unique attribution.
Detection
Benign JIT, denied action, missing event and mismatched identity controls. Keep uncertainty visible.
Response
Evidence authority, privacy, containment, credential lifecycle, trusted recovery and tested remediation.
Investigator cheat sheet
Import: capability lead. RW → RX: permission change. Successful exec: program-image transition, not intent. Connection: network state, not confirmed exfiltration. ServiceAccount: API identity, not Linux UID. Missing events: unknown coverage. Root in container: not automatically host or kernel compromise.
0 controls verified
The checklist is a local reading aid, not an automated compliance score. References were reviewed October 17, 2026. Automated checks exercise synthetic logic and website behavior; no live malware, kernel sensor or Falco deployment was tested for this article.
Related K8SEC research: kernel boundaries and OpenShift security, Kubernetes audit-log investigations, Docker image layers and runtime confinement.
I do not want a detector that calls every JIT malware or every container process a host compromise. I want an investigation that explains the sample’s capability, the operation the kernel accepted, the identity that performed it and the evidence connecting it to an actual effect.
Reverse engineering supplies hypotheses. Runtime evidence decides which ones survived contact with the system.
Riad DAHMANI — K8SEC Security Research
Investigate the whole path.
Use documented infrastructure relationships to guide the next question, then validate each boundary with evidence.
Explore related K8Map research