What is an AAR? Who can access it?
What is an AAR?
At the end of a crisis sim exercise, an After Action Report (AAR) will be available to Crisis Sim Managers. This will inform them on:
- Overall Performance
- Inject Breakdown (Metrics relating to decision points)
- Participant Breakdown
- Next Steps (through related scenarios and lab collections that can be assigned)
How is a new AAR different from previous reports (View Results)
AAR is in addition to and complements the ‘View Results’ Report. AAR equips you with actionable insights and metrics on performance, whereas the ‘View Results’ report outlines the specific decisions made by participants and allows you to export these to CSV. Use ‘View Results’ to examine specific responses; use the AAR to determine next steps.
AAR also includes support for reporting by teams in single-player mode. Managers can assign exercises that include team reporting options by region, department, or enterprise when an exercise is created or edited.
Who can view AARs?
Anyone designated as Crisis Sim Manager can access and view AARs. This role is typically allocated to those responsible for crisis management in your organization (e.g., Incident Response Manager).
Are AARs available for all exercise types (e.g., Presentation, Single-Player, and Drills)?
Yes, as long as either response confidence or ranking is enabled in the scenario settings (this is the default case for all Immersive Labs scenarios). However, support for reporting by teams is only available in single-player mode.
The report will become available 30 minutes after the first completion of the exercise.
Why are AARs valuable?
The report provides the insights crisis managers need to effectively plan next steps for teams and individuals to improve their crisis response readiness.
What are some typical use cases for reporting by teams?
- Assign team exercises that scale across the organization.
- Compare teams’ performance against one another to prioritize exercising strategies.
- Assess organizational cyber readiness and risks by specific teams.
How to access AARs
How do I view an AAR?
At the end of an exercise, Crisis Sim Managers can follow these steps:
- Navigate to ‘Exercise’ in the main navigation menu and select ‘Immersive Crisis Simulations.’
- Search for the specific exercise in the ‘Managed by Me’ or ‘All Exercises’ area: use the search bar and/or filters on the left-hand side to bring up the exercise you want to view results for.
- Select the exercise and then the ‘View After Action Report’ button.
Note: the AAR button will be accessible 30 minutes after the
first completion.
Are AARs available retroactively (i.e., for exercises ended before its release)?
Yes. All exercises (including exercises that ran in the past) will have AARs available. If you don’t have an AAR appearing, jump to the next FAQ.
Why didn’t I get an AAR for an exercise?
The report will become available 30 minutes after the first completion. Contact our Support Team if a report isn’t generated, and we’ll investigate.
What data points can I see in the report?
What is visible depends on what you have chosen to measure in your exercise.
All data points will be available if your exercise is set up to measure response confidence and ranking of inject options. We measure both by default in all Immersive Labs scenarios.
If you create your own custom scenarios and want to obtain After Action Reports, ensure you’ve enabled these options when creating your content. Response confidence can be found in the ‘Analysis’ area of the content creator, which is just below ‘Ranking.’
If your exercise isn’t measuring response confidence, the data points related to this, such as decision confidence by inject, won’t be available.
Similarly, if you haven’t enabled ranking inject options, the data points related to this, such as Decision Score by Inject, won’t be in your report.
Note: data points relating to reporting by teams are only available in a single-player mode.
Why doesn’t my AAR for an exercise include a High Performers and Low Performers section? This feature is available in AARs for other exercises I’ve run.
If you did your exercise in Presentation mode, the AAR wouldn’t include these two sections in the Participant Breakdown area of the report. There will only be data for one participant in this area because, in Presentation exercises, only one participant selects answers: the facilitator.
Can I export/download an AAR?
Future releases will allow you to export the report to PDF format.
Terminology FAQs
What is a good overall score?
This is located In the Overall Performance section.
Based on our experience with IL clients today:
>= 75% – Excellent
>= 50% – Good
>= 25% – Fair
>= 0% – Needs improvement
The scoring guidelines apply to Immersive Labs scenarios. Consider this when interpreting the score if you have created your own scenarios or heavily customized option rankings within Immersive Labs scenarios.
What are the related scenarios and collections in ‘Next Steps’ based on?
These are chosen from the ‘attack vector’ of the exercise’s scenario.
What is an inject?
We call every decision point in scenarios an inject; scenarios are made up of injects. Injects (decision points) comprise inject options that rank weak, okay, good or great.
What is a playthrough?
A playthrough is a completion of the exercise.
Calculation FAQs
How is the Overall Score calculated?
The score on the exercise is an average of participants’ decisions (selection of inject options), with an adjusted element to account for confidence levels.
For example, selecting a weak option with a high level of confidence is a weaker decision than choosing a weak option with a low confidence level. This is because a high level of confidence could lead to the participant swaying others in a crisis or not checking with team members before acting, which could translate to higher risk for the organization.
We use the Dunning-Kruger curve to adjust for overconfidence bias in the score. If the exercise does not measure participants’ response confidence, the overall score will be the Decision Score (see next FAQ).
How is the Decision Score calculated and ranked?
It is an average of an individual’s scores based on their selected inject option. Inject options are ranked:
- Great – 100%
- Good – 75%
- Okay – 50%
- Weak – 25%
For example, in an exercise with five injects, where a participant selected the Great option for three injects and the Okay option for two, the Decision Score would be calculated as: (100+100+100+50+50)/5 = 80.
How is Decision Confidence calculated?
This is found in the Participant Breakdown section. It is the average of the response confidence scores.
- Very confident – 100%
- Confident – 75%
- Somewhat confident – 50%
- Not very confident – 25%
- Not at all confident – 0%
How are the Strongest and Weakest Injects determined?
We consider the strongest injects to be the highest-scoring answers as voted by the participants. The weakest injects are the lowest-scoring answers as voted on by the participants.
How are High and Low Performers determined?
High performers are individual participants (Drill exercises) or teams (Single Player exercises) with an overall score equal to or higher than 50%.
Low performers are those individuals or teams that have scored lower than 50% in the exercise.