The Challenge
Australian public sector agencies face a significant compliance timeline issue. The Information Security Manual (ISM) defines 1,081 security control requirements across 22 guideline chapters. Demonstrating compliance to achieve an IRAP assessment outcome typically requires months of preparation, evidence gathering, and testing. For organizations building on AWS, this creates a bottleneck: you're deploying cloud infrastructure faster than your compliance documentation can keep up.
The gap isn't just documentation. It's the manual work of mapping technical implementations to ISM control requirements, proving those implementations are operating as designed, and maintaining evidence as your environment evolves. When gwi.digital conducted an independent assessment of the Landing Zone Accelerator on AWS (LZA), they needed to evaluate 1,081 ISM controls. Without automation, that's a months-long exercise in evidence collection and control-by-control validation.
Realistic Constraints
The assessment operated under constraints that mirror what Australian organizations face. gwi.digital deployed LZA in a greenfield AWS environment in the ap-southeast-2 (Sydney) Region, positioned within Phase 2 of the ASD Cloud Security Assessment and Authorization Framework. This phase focuses on cloud consumers building on already-authorized AWS services, specifically the PROTECTED level services assessed by CyberCX last year.
Scope limitation is crucial. Of the 1,081 ISM controls, only 256 fall within what infrastructure automation can meaningfully address. The remaining 825 controls cover physical security, personnel vetting, organizational governance, and classification-level exclusions that sit outside any technical deployment tool's scope. This three-tier shared responsibility model (AWS-provided, LZA-enabled, customer-responsibility) clarifies where automation ends and organizational policy begins.
LZA's Universal Configuration deploys nearly 200 security controls automatically, based on AWS Well-Architected pillars and AWS security practices. But proving those controls meet ISM requirements demanded more than deployment scripts. It required systematic validation across account structure, identity and access, network configuration, logging, encryption, and backup.
The Approach
gwi.digital combined manual assessment with automated validation tooling. They mapped the 256 addressable ISM controls to LZA's technical implementation, categorizing coverage as Full, Partial, or Not Addressed. For controls requiring evidence, they used the Controls Acceptance Testing Suite (CATS), an automated compliance validation engine that runs purpose-built tests against LZA's security baseline.
CATS executed over 3,600 individual tests across six AWS accounts during the assessment. Each test validates a specific configuration element against the expected state. For ISM assessments, gwi.digital developed a conversion script that maps CATS output to ISM control references, enabling auditors to interpret results from an ISM perspective. The output includes OSCAL (Open Security Controls Assessment Language) format for interoperability with assessment tools.
The testing wasn't a one-time validation. CATS enables repeatable execution, identifying when configurations deviate from the validated baseline between assessment cycles. This shifts compliance validation from periodic manual audits to continuous automated assurance.
Results and Metrics
LZA achieved Full or Partial coverage for 234 out of 256 ISM controls within its addressable scope. That's 91% coverage for the technical infrastructure controls a deployment solution can implement. The 22 controls not addressed require customer-specific decisions (data classification schemes, retention policies aligned to business requirements) that no automated baseline can predetermine.
The evidence generation numbers demonstrate the scale of automation. Over 3,600 individual tests across six accounts produced machine-readable results that replace weeks of manual evidence collection. For Australian customers preparing for IRAP assessments, this compresses the evidence-gathering phase from months to days.
The assessment also identified quick wins. gwi.digital found configuration adjustments that elevate specific control ratings from Partial to Full with minimal effort. Increasing the default password length from 14 to 15 characters aligns with ISM requirements. Adjusting log retention to match Australian Federal Disposal Authority standards closes another gap. These recommendations have been communicated to the LZA team for inclusion in future ISM-specific guidance.
Future Improvements
The assessment revealed opportunities for ISM-optimized configuration from deployment. Rather than adjusting password length and retention policies post-deployment, future versions of LZA could include ISM-specific configuration options at provisioning time. This would push Full coverage higher without requiring customers to modify the baseline.
The three-tier responsibility model analysis proved valuable but came late in the process. Starting with explicit categorization of the 825 out-of-scope controls into physical security, personnel, governance, and classification subcategories would have accelerated scoping. Future assessments should establish this shared responsibility breakdown before diving into technical validation.
CATS currently operates as a private beta solution accessible through AWS Professional Services. Broader availability would let customers run continuous validation internally, catching drift before assessment cycles. The ISM-enriched reporting that gwi.digital developed should be integrated into CATS directly, not maintained as a separate conversion script.
Takeaways for Your Team
If you're preparing for IRAP assessment, the LZA approach offers three immediate advantages. First, pre-determined control applicability saves weeks on scoping. The LZA Compliance Workbook (available on AWS Artifact) maps ISM requirement identifiers to security implementation statements, giving you a documented starting point rather than a blank spreadsheet.
Second, independent third-party validation provides assurance you can share with assessors. The gwi.digital report isn't an official IRAP authorization, but it's a professional evaluation based on evidence. When your assessor asks how you're meeting ISM control requirements, you're handing them documentation of 3,600 automated tests across your baseline configuration.
Third, continuous evidence generation changes your compliance posture between assessment cycles. Traditional approaches treat compliance as a point-in-time event. You gather evidence, pass assessment, then drift until the next cycle. Automated validation tools like CATS detect configuration drift as it happens, replacing periodic manual audits with ongoing assurance.
The broader lesson extends beyond ISM and Australian organizations. When you're facing complex compliance frameworks with hundreds of technical controls, the combination of automated deployment and continuous validation compresses timelines while improving reliability. Manual evidence collection doesn't scale. Automated testing that maps directly to control requirements does.
Contact your AWS account representative or reach out to the LZA team if you want access to CATS or have questions about the ISM assessment approach. The report and compliance workbook are available now on AWS Artifact.
