Overview
Custom Automated Tests (CAT) enable customers to create custom tests against integration resources. Secureframe supports CAT across all integration resources, including any additional resources connected through the Custom Data Platform.
Secureframe will be adding support for other types of resource categories and fields over time. If you have a need for other categories to be supported, please let your Customer Success Manager (CSM) know and we will prioritize support accordingly.
Workflow
Here's a loom walkthrough of a example test: Video
Create a custom automated test
Navigate to the tests page and click "Add test" and select "Create custom automated test".
Create test scope
Select the type of resource to test.
Choose from these 2 options:
Cloud Resources, then choose AWS
Assets, then choose from
Devices - Physical machines (i.e. Mac or Windows computers). Note: CAT does not have a dedicated device platform or type filter. To exclude specific device types from an MDM integration (e.g., filtering out iPhones/iPads from Kandji), use the Model field in the scope conditions. You can also combine Model with OS Version for more precise filtering.
Repositories - Version-controlled collections of files and metadata that represent a project
Tickets - Tracked unit of work that represent a task, bug, or feature request
As an optional step, narrow down the resources that should be tested. In most cases, you can proceed to the next step after selecting a resource type; however, if you only want to test a subset of resources for a given type, you can add more conditions:
Create test evaluation criteria
Depending on what is more intuitive, specify if resources should pass or fail when conditions are met:
Specify the conditions in which resources returned by the query should pass or fail
Once satisfied with the conditions, proceed to the next step.
Set test details
Fill out basic test information, including
Title (required)
Description
Remediation Guidance
Owner
Vendor (defaults to the vendor initially selected)
Resource type
Domain
Function
Click "Enable test" once satisfied.
Mapping tests to controls
Map test to applicable controls. The test will not evaluate until at least one control is mapped.
Viewing evidence
View evidence, inspect json, and export evidence just like Secureframe's default tests.
Editing a test
Navigate to the "Query logic" tab and click "Edit". You can adjust the test logic on demand.
Custom Automated Test Use Cases
Secureframe's default test evaluation logic is too rigid
Example: Secureframe's default "Daily automatic RDS backups (AWS) test requires that backups be retained for at least 6 days. An organization may not want to retain data that long due to cost or other concerns. An organization could create an equivalent custom automated test against "AWS RDS Instance" where "Backup retention period" greater than / equal to a retention period that is aligned with their needs. Organizations no longer need to resort to disabling tests or passing a test via upload.
Organizations want to implement with security requirements that exceed whats required for compliance
Example: An organization may have an internal policy that requires all password to be at least 12 characters in length. Secureframe's default "Passwords require minimum length (AWS)" test only checks for at least 8 characters in length. The organization could create a custom automated test for AWS IAM Password Policy" where "Minimum password length" is greater than or equal to 12.
Secureframe's default tests don't account for compensating methods to address compliance
Example: Secureframe's default tests attempt to account for as many edge cases as possible; however, there are scenarios where organizations need custom logic to address compliance requirements. As an example, an organization could have a very unique custom access policy that isn't yet compatible with our default tests. The organization could create a new access test that better suits their needs. Organizations no longer need to resort to disabling tests or passing a test via upload.
Secureframe's default test scope is too broad
Example: Secureframe's integration tests examine ALL resources of a given resource type across all accounts, regions, etc. An organization may have resources that are in audit scope; however, shouldn't be in scope for a particular test. S3 public access is a practical example. An organization may have S3 buckets that are intentionally public but are still in audit scope. The organization could create a new public access test to only examine s3 buckets with a certain account id or arn. Organizations no longer need to ignoring test results.
Organizations lack automated test support for custom controls and frameworks
Example: An organization using custom controls and frameworks due to internal requirements or niche regulatory requirements may not be able to find applicable, default Secureframe tests that fit their needs. An organization could create new tests that are bespoke to these custom requirements.
Filtering specific device types from an MDM integration (e.g., excluding iPhones/iPads from Kandji)
Example: An organization using Kandji as their MDM may want to exclude mobile devices (iPhones and iPads) from a test that should only apply to Mac or laptop computers. Since CAT does not have a dedicated device platform or type filter, use the Model field in the scope conditions to match and exclude specific device models (e.g., filter out devices where Model contains “iPhone” or “iPad”). Combining Model with OS Version can provide even more precise filtering when needed.