What a Clean Control Test Really Proves
Résumé
Pixel shares an analysis of what a passing leaver-process test can—and cannot—prove. It explains how samples drawn from existing tickets may miss unrecorded departures, and why control decisions must account for the objective, population, test, and period examined.
DEV Community
The Sample Passed. What Did We Actually Prove?
Why a clean control test can still leave a decision maker exposed By Manfredi Marrocco The report says every sampled access removal was completed on time. The dashboard turns green. A manager asks whether the company can now rely on its leaver process. There is a missing question: did the sample come from a complete list of leavers? If the test began with tickets already created in the identity system, it may have excluded precisely the cases where nobody created a ticket. Ten well-processed tickets can demonstrate that those ten tickets were well processed. They cannot, by themselves, establish that all departures entered the process, or that the same process worked in every subsidiary. This distinction has practical consequences. A control assessment does not merely collect proof that an activity happened; it establishes how far that proof can support a decision. The boundary depends on the control objective, the population, the test and the period examined. A pass without those boundaries can be more misleading than an explicit limitation. Return of the Obra Dinn as a decision lens In Return of the Obra Dinn, the player investigates a ship and reconstructs events through observation and deduction. The official description establishes that investigative premise. The connection to control assessment is the discipline of asking what an observation can support. It is not a claim that a fictional investigation follows an audit standard. The useful connection is the boundary of a deduction. A ticket constrains the account of a recorded departure; it cannot establish the complete departure population. The game offers a designed mystery. An organisation must also account for evidence that is unavailable and conclusions that remain unresolved. Start with the claim, not the screenshot Consider a fictional organisation, Northstar Distribution. Its policy says that relevant access for departing employees must be removed within a defined period after departure. The assessment team receives ten closed tickets, each with a timestamp and an apparent approval. All ten meet the organisation's internal deadline. The example is hypothetical: neither the deadline nor the sample size represents a universal requirement. What has been shown? For the ten selected records, the ticket system contains information consistent with timely completion. That is a useful observation. It is not yet the same as showing that every departure was captured, that all relevant accounts were disabled, or that an account could not be reactivated later. The claim can be broken into questions: Coverage: Were all in-scope departures identified, including contractors where the control applies? Execution: Were the required actions completed by the applicable deadline? Completeness of action: Did the actions cover the relevant systems and privileged accounts? Persistence: Did access remain removed, or did a later event restore it? Period and scope: Which entities, services and dates were actually examined? Each question calls for different evidence. A ticket timestamp may speak to execution. It is weaker evidence of coverage when the ticket list is the only source of the population. A screenshot of a disabled account may demonstrate its state at a particular time, but not necessarily when it changed or whether other accounts existed. NIST's definition of a security control assessment distinguishes correct implementation, operation as intended and the desired outcome against security requirements. Those are related but separate questions. NIST SP 800-53A provides tailorable assessment procedures and planning guidance; it is a US framework, and its use here is methodological guidance, not a claim that every organisation is legally bound by it. The population is part of the test A sample can only be interpreted against the population from which it was selected. If Northstar tests only identity-system tickets, its population is “tickets recorded in that system,” not automat...

Powered by Readybot.io