Automating Security Questionnaires
Secureframe’s Questionnaire Automation tool learns from your company’s previously completed questionnaires and helps you to automatically complete new ones. It also reads from the data stored in your Secureframe account to generate answers. While the Knowledge Base is the primary source the machine learning pipeline uses to formulate answers, it also draws from your Policies, Tests, Controls, and your company metadata (name, address, etc.).
If you’d prefer a video-based walkthrough that includes context on the latest improvements to the tool, please view the Loom below:
Otherwise, this article will step through how to add question and answer pairs to your Knowledge Base, how to manage your content, and how to process your first real questionnaire.
Getting started with Questionnaire templates
If you’re using the Questionnaire Automation tool for the first time, start by building your Knowledge Base (KB). The KB is the first place that the machine learning pipeline looks when processing a new questionnaire to see if there is related content to answer similar questions. It also helps you ensure that all answer content is up to date. You can set an owner on each question/answer pair and schedule content reviews to occur on a specific frequency.
The best way to quickly build your KB is by uploading prior questionnaires in Excel format, but you can also Download our Starter Pack to answer common questions and upload them in bulk:
-
Click on the Knowledge Base tab on the left panel
-
If there is no content, you can optionally download the Starter Pack by clicking the Download starter pack. It’s an Excel spreadsheet containing some of the most common kinds of security compliance questions you’ll encounter. Fill this out and upload it using the methods covered in the following steps.
-
If you have one or more questionnaires that you’ve previously filled out, you can upload them by clicking Start uploading in the card at the top or Add to library button on the top right.
-
Select your document and click Upload. Please note: This bulk upload process works best if your question and answer pairs are in a simple format. We recommend avoiding spreadsheets with macros or other kinds of automation that conditionally display content as these may cause errors upon import.
-
You should now see the tagging screen, which looks like the screenshot below. This is a pared down preview (no formatting or images) of your uploaded spreadsheet. You can view the tabs on the top and see the content rendered in column and row format.
-
We need to point out which items are the questions and answers we’d like to import into the KB. For this, we’ll start by tagging questions and then answers. There are a few options: You can highlight an entire column by clicking the column letter (e.g. “A”) and then mark these as “Question.” You can also click and drag individual cells to make your selection and tag.
-
Once we’ve identified questions, we’ll mark the columns/cells that contain the associated answers. You can also click and drag or select the column containing each answer type. The tool gives you a few different options to tag answer types: Yes/No, True/False, and Free-Form. Use Yes/No or True/False on binary answer types and Free-form for longer-form free text responses.
-
Optional: Use the Auto tag function to tag questions and answers while ignoring blank spaces and header rows to save time. To do this, we must tag a few example questions and associated answers.
-
Click the Auto tag button, confirm, and let the tool run. This process may take a moment. You should see your sheet correctly tagged with questions and answers. We recommend that you scan through the sheet and confirm.
-
Once you’ve tagged your questions and associated answers on all sheets within your uploaded questionnaire, click Process to have the tool add them to your KB.
-
You should now see your content in your answer library. (You may have to refresh the page if you don’t see anything at first).
Tag and organize Knowledge Base content
Use tags to filter and sort your content by categories that you create. Examples include: tags for answers pertaining to specific compliance frameworks, tags for answers about certain infrastructure (e.g. AWS, Google Cloud), tags for answers by locale (e.g. US, APAC, EMEA).
To add tags:
- Select the applicable content using the checkboxes and click the Add tags bulk action on the top bar.
- Select a tag(s) or create a new tag by click the Search tags box and typing your new tag name.
- Click Save.
Viewing past Knowledge Base bulk uploads
In the Knowledge Base, you can view which questionnaires you’ve uploaded by clicking the “Documents” tab to the right of the “Library” tab and download them to inspect the content.
Managing the freshness of your Knowledge Base content
To ensure your KB content stays fresh and accurate, we recommend you do two things:
- Set an owner for all applicable content
- Set a review period for all applicable content
“Applicable content” in this context would be any answers you know will change over time. For example, this could include the results of penetration tests or auditor findings that you note in your answers. Or, if your answers reference infrastructure or company policies that are liable to change. These are all good candidates to incorporate into a review process.
When content comes up for review on the frequency that you select, the owner will receive an email from Secureframe to review the content. They will also see a prompt at the top of the KB screen with a link to view content in need of verification.
To set an owner:
-
Select or bulk select question/answer pairs by checking the box next to the content
-
Click Set owner on bulk select actions
-
Select a person from your organization and click Save
To set a review period:
-
Select or bulk select question/answer pairs by checking the box next to the content
-
Click Set review period
-
Select your frequency and then click Save
-
You should see the review frequency appear on each piece of content. Hovering over the icon on the Reviewed on column will show the next, upcoming review date
Verify that content is reviewed by clicking the Mark as reviewed button on the content or use the Mark as reviewed bulk action to mark a list of question/answer pairs as reviewed.
Publish policies to make them available to the machine learning pipeline
To ensure that answer suggestions can be sourced from policies pertinent to security compliance questions you receive on questionnaires, ensure that all applicable policies are in the Published state.
Optionally opt out of answer rephrasing or generative AI suggestions
Secureframe uses OpenAI to assist in portions of the Questionnaire Automation pipeline. You can view OpenAI’s policy on information storage using its API platform (which is the method that Secureframe uses to interact with OpenAI) here. At the time of this writing (Q3, 2024), none of the data that Secureframe sends to OpenAI is used for OpenAI’s model training.
Opting out will likely reduce the quality of answers that the tool is able to generate on your behalf, as well as the amount of automation coverage you will see on a given questionnaire.
If you do not wish send your security questionnaire or compliance information to OpenAI you can opt out by clicking on cog icon to access Questionnaires settings:
- Switch off “Large language model authorization” to prevent any answer content from being sent to OpenAI. Switching this off will also disable Knowledge Base content rephrasing.
- Switch off “Knowledge Base content rephrasing” to speed up questionnaire processing. Having this feature switched on improves the quality of your answers by adapting useful content from your Knowledge Base to similar questions that require a differently-worded reply.
Supported questionnaire formats
At this time, Questionnaire Automation tools supports Microsoft Excel-formatted questionnaires (.xls, .xlsx). You may also fill in web portal-based questionnaires using the Secureframe Knowledge Base Chrome Extension.
Learn more about the Chrome extension in this article.
How to process your first questionnaire
-
Navigate to the Questionnaires page in the Secureframe application.
-
Click Upload questionnaire on the top right.
-
Optionally edit the questionnaire name, type, owner, client name, and due date. Questionnaire type is for internal reference only. Setting the due date lets you filter questionnaires by date and also provides an internal reference for prioritizing responses.
-
Click Upload on the modal
-
Tag your questionnaire in the same fashion as you did when importing previously completed questionnaires into your Knowledge Base (documented in the sections above): Identify which cells are questions and which cells should contain the corresponding answers.
-
Once you’ve completed tagging, click Process and confirm (this invokes the machine learning pipeline and the Questionnaire Automation tool will attempt to generate answers from a variety of sources in the Secureframe platform)
-
You may have to wait for a few minutes for the questionnaire to finish processing (the time will lengthen with larger questionnaires). If you would like to navigate away from this page while the questionnaire processes, you may do so and return later.
-
When it is finished processing, the tool will bring you to the answer verification screen (which looks like the screenshot below). Here, you can view all answers that the tool attempted to fill. Answers are divided into “Unanswered” (the models were not confident enough to answer these), “Draft” (answers the model attempt to give but which require human review), and “Complete” (answers about which the model had high confidence; these should be ready for the export at the end).
-
To provide or edit an answer, click on the blue text of the question to open the verification modal. If you would like to edit an existing Complete answer you must click Re-open on the bottom right.
-
On the left side of the modal you can see the question and its answer types (binary or freeform). The AI models will attempt to select binary options and generate free-form answers. The right side of the modal shows “Suggested answers” sourced from data on the Secureframe platform. “Trust AI” responses are the final result of generatively-created answers. For these, you can view the sources the AI drew from by clicking See sources. Note: If the question text is an exact match with a question in your KB, the application will use the exact text of the corresponding answer from the KB. In all other cases where there is not an exact match but a similar response in the KB, Knowledge Base content rephrasing will attempt to adapt the content from that answer to respond to this question.
-
You can use a suggestion on the right by clicking Use this answer, which will replace the free-form text on the left.
-
You can optionally use the Refine with Trust AI feature to change the style or character of an answer in a dialog format.
-
You can optionally assign reviewers to the answer from your organization by clicking Reviewers. Any users from your company who are assigned will receive an email prompting them to log into Secureframe to review answer content in this questionnaire.
-
You can also optionally attach documents such as data flow diagrams or policies to the answer by clicking Attachments. These will export with the finalized, filled-in questionnaire Excel sheet in a zip file when you complete your review of the answers.
-
When you are satisfied with your answer click Complete to mark it as done.
-
When all the answers have been marked Complete, export your completed questionnaire and all attachments by clicking Complete review
-
This will bring you to a screen where you can download the completed questionnaire. If you have included no attachments, it will download an Excel sheet. Otherwise the download will appear in .zip format. Note: If you’re wondering what happens to the formatting on the original spreadsheet, the exported questionnaire sheet will retain all of the formatting and imagery from the uploaded version. The Questionnaire Automation tool only attempts to fill in the text.
Frequently Asked Questions (FAQ)
I tried to import my questionnaire spreadsheet, but only some of the questions uploaded?
- One of the most common reasons why some answers would be uploaded but others were not is formatting.
- As shown below, if a cell is blank the uploader can assume the line below is the end of that sheet. In this case, best to check for empty columns and rows to ensure the full sheet is uploaded.
I have noticed that some of my questionnaires are corrupted or not complete when I download?
- To download a questionnaire, the status of the questionnaire must be marked as Completed.
- If the status is still in "Verify", this will only download the original file that was uploaded.
I have purchased a set number of questionnaires, how do I know what counts towards my paid limit?
- If you have purchased a specific alotment of questionnaires, then only those that are answered automatically via Trust AI will count towards your limit. If you answer the questions manually, they do NOT count toward your limit.
- If you have purchased an unlimited package, then you have unlimited use of Trust AI automated answers or manual answers.
- Learn more about how to upgrade by speaking with our Account Management team at accountmanagement@secureframe.com
Comments
0 comments
Article is closed for comments.