Dynamic validation

How the set is measured with the YARA engine and how problematic rules are isolated.

What is measured

dynamic_gate.py builds a safe corpus (random files, text, system files and detection canaries), compiles the approved set and runs it with YARA against every corpus file measuring CPU, peak memory and time (on Linux with the Ubuntu YARA; in the macOS job with the engine MacGuard ships, 4.5.8). It compares the candidate with the previous release in the same job, alternating runs.

sequenceDiagram
  autonumber
  participant G as dynamic_gate.py
  participant B as bench.py
  participant Y as yarac and yara
  participant C as compare()
  G->>B: build_corpus (random, text, system files, canaries)
  loop runs, alternating baseline and candidate
    G->>B: measure_set(previous release)
    B->>Y: yarac one namespace per file
    B->>Y: yara -C per corpus file, timeout 10 s
    Y-->>B: cpu, max RSS, wall, exit status
    G->>B: measure_set(candidate)
  end
  G->>C: baseline runs vs candidate runs
  alt regression, crash or timeout
    C-->>G: verdict regression
    G->>G: ddmin over new or changed files
    G->>G: culprits to quarantine, rebuild the set
    G->>B: measure the reduced set again
  else within thresholds
    C-->>G: verdict ok
    G->>Y: canaries must all be detected
  end

sequenceDiagram
  autonumber
  participant G as dynamic_gate.py
  participant B as bench.py
  participant Y as yarac and yara
  participant C as compare()
  G->>B: build_corpus (random, text, system files, canaries)
  loop runs, alternating baseline and candidate
    G->>B: measure_set(previous release)
    B->>Y: yarac one namespace per file
    B->>Y: yara -C per corpus file, timeout 10 s
    Y-->>B: cpu, max RSS, wall, exit status
    G->>B: measure_set(candidate)
  end
  G->>C: baseline runs vs candidate runs
  alt regression, crash or timeout
    C-->>G: verdict regression
    G->>G: ddmin over new or changed files
    G->>G: culprits to quarantine, rebuild the set
    G->>B: measure the reduced set again
  else within thresholds
    C-->>G: verdict ok
    G->>Y: canaries must all be detected
  end

Decision

flowchart TD
  S["approved set"]:::val --> B{"previous release<br/>available?"}:::val
  B -->|no| A["absolute limits<br/>p95, RSS, compile time"]:::val
  B -->|yes| CP["compare medians with<br/>the baseline noise"]:::val
  A --> V{"ok?"}:::val
  CP --> V
  V -->|no| I["isolate culprits<br/>delta debugging"]:::qua
  I --> Q["culprits to quarantine"]:::qua --> R["rebuild and measure again<br/>max 3 rounds"]:::val --> V
  V -->|yes| K{"all canaries detected?<br/>measured set equals set to publish?"}:::gate
  K -->|yes| P["publishable"]:::sec
  K -->|no| X["not publishable"]:::qua
  classDef src fill:#cfe3ff,stroke:#2f6df0,color:#0b1b33
  classDef val fill:#ffe9b8,stroke:#b8860b,color:#2b1d00
  classDef sec fill:#cdeedd,stroke:#1b6e3c,color:#08210f
  classDef pub fill:#e4d7ff,stroke:#6a3fd1,color:#1b0b3b
  classDef qua fill:#ffd4cc,stroke:#c0392b,color:#33100b
  classDef gate fill:#ffffff,stroke:#111111,stroke-width:3px,color:#111111

flowchart TD
  S["approved set"]:::val --> B{"previous release<br/>available?"}:::val
  B -->|no| A["absolute limits<br/>p95, RSS, compile time"]:::val
  B -->|yes| CP["compare medians with<br/>the baseline noise"]:::val
  A --> V{"ok?"}:::val
  CP --> V
  V -->|no| I["isolate culprits<br/>delta debugging"]:::qua
  I --> Q["culprits to quarantine"]:::qua --> R["rebuild and measure again<br/>max 3 rounds"]:::val --> V
  V -->|yes| K{"all canaries detected?<br/>measured set equals set to publish?"}:::gate
  K -->|yes| P["publishable"]:::sec
  K -->|no| X["not publishable"]:::qua
  classDef src fill:#cfe3ff,stroke:#2f6df0,color:#0b1b33
  classDef val fill:#ffe9b8,stroke:#b8860b,color:#2b1d00
  classDef sec fill:#cdeedd,stroke:#1b6e3c,color:#08210f
  classDef pub fill:#e4d7ff,stroke:#6a3fd1,color:#1b0b3b
  classDef qua fill:#ffd4cc,stroke:#c0392b,color:#33100b
  classDef gate fill:#ffffff,stroke:#111111,stroke-width:3px,color:#111111

• Each set is measured 5 times and medians are compared against the baseline noise; a regression needs to exceed 3 times the noise and at least 25 %, 0.5 s of CPU or 25 MB.
• Without a previous release absolute limits apply: scan p95 2.0 s, 1500 MB of memory scanning, 3000 MB compiling and 240 s of compile time.
• Timeout per file: 10 s. Every canary must be detected.
• If anything fails, delta debugging (ddmin) isolates the new or changed files responsible, they go to quarantine and the set is measured again, up to 3 rounds.
• The measured set must be exactly the one that is published (set_root equals the Merkle root); publish.yml checks it again before signing.

On macOS

A job on a macOS runner installs the engine shipped with MacGuard and runs macguard_smoke.py with the candidate set. The exact values are in config/policy.json.

LIMIT GitHub macOS runners are virtual machines (Apple M1 Virtual, hv_vmm_present=1, no Hypervisor.framework). There is no independent macOS virtual machine: dynamic.json declares it as not_run: independent-macos-vm. That needs a self-hosted physical Mac.

LIMIT The Linux gate uses Ubuntu YARA (4.5.0) while the app ships 4.5.8. Static classification of the whole set gave the same result with both engines (checked 2026-10-18), but dynamic measurements may differ between engines and systems; see Limits and pending work.

No real malware is run at any point: the corpus is made of synthetic data, text and benign files.