Testing Every Engine Slot Against Known-Wrong Inputs Before Promotion

Why positive unit tests are not enough: every AgentFoundry conformance kit pairs known-good fixtures with known-wrong controls and keyword-matched benign twins.

A common failure in AI-generated test suites is the vacuous pass: a test or verifier function that checks only whether an output object exists (assert result is not None) without testing whether broken or unsafe outputs are actually rejected. On getagentfoundry.si, no component pack or engine adapter can be promoted unless its verifier.py passes a three-part conformance kit:

• Known-good fixtures (must_accept): Valid boundary and nominal inputs that the engine and verifier must accept with zero errors.
• Known-wrong adversarial controls (must_refuse): Deliberately corrupted meshes (open boundary edges, inverted normals), physics tunneling trajectories, or exploitative contract clauses that the verifier must reject with the exact expected refusal_code.
• Keyword-matched benign twins (must_clear_without_false_alarm): Inputs that share surface vocabulary with hazards (for example, an employer-paid cross-border Manila-to-Hong Kong contract) that must pass without triggering a false refusal.

# Conformance Kit Meta-Test: Proving the Verifier Rejects Known-Wrong Controls
def qualify_slot_verifier(verifier_fn, kit: dict) -> dict:
    for good in kit["known_good"]:
        res = verifier_fn(good["candidate"])
        if res["status"] != "passed":
            return {"qualified": False, "reason": f"rejected_known_good:{good['id']}"}

    for wrong in kit["known_wrong"]:
        res = verifier_fn(wrong["corrupted_candidate"])
        if res["status"] == "passed":
            return {"qualified": False, "reason": f"vacuous_verifier_accepted_known_wrong:{wrong['id']}"}
        if res.get("refusal_code") != wrong["expected_refusal_code"]:
            return {"qualified": False, "reason": f"wrong_refusal_code:{res.get('refusal_code')}"}

    return {"qualified": True, "good_checked": len(kit["known_good"]), "wrong_rejected": len(kit["known_wrong"])}

When one of our early skill cards caused a -0.149 score regression on glm-5.3-flash, adding known-wrong controls and non-regression gates caught the failure immediately and blocked promotion until the card beat the unassisted baseline.