After Action Reports (AARs) include the following sections:
|
Notes:
Download reports to PDF formatFind a download PDF button on the right corner of each report. |
Scenario
This section of the report provides contextual information on the cybersecurity topics the scenario covers and the capabilities it is set out to exercise.
Overall performance
This includes the following high-level metrics:
Completion rate
The percentage of participants who completed the exercise (out of those assigned.
Overall score
If the exercise is a presentation exercise, it is the average decision score of the facilitator. If it is a single player or drill exercise, it is the participants' score on the exercise with an adjustment relating to the confidence of their response, if applicable.
Overall score: how is this calculated?
For presentation exercises, the overall score reflects the facilitator's average decision score.
For single-player and drill exercises, the overall score represents the participant's average decision score, adjusted to account for their confidence levels. This adjustment reflects the understanding that a high degree of confidence in a suboptimal decision can be riskier than a weak decision made with less certainty. The scoring incorporates the Dunning-Kruger effect to mitigate overconfidence bias. If the exercise doesn't include a measure of confidence, the overall score is simply the average decision score.
Decision score:
This indicates how well the participants have responded to injects.
Decision confidence:
The average confidence level of the participants' decision throughout the exercise.
Decision score and decision confidence: how are these calculated?
Decision score: an average of an individual's scores based on their selected inject option. Inject options are ranked:
Great : 100%
Good: 75%
Okay: 50%
Weak: 25%
Decision confidence: this is the average of the response confidence scores. These are scored as below:
Very confident: 100%
Confident: 75%
Somewhat confident: 50%
Not very confident: 25%
Not at all confident: 0%
Exercise playthroughs by day
This table shows the number of participant playthroughs by day from the exercise start date. A playthrough refers to any time a participant completes an exercise.
Questions and conclusions
Overall performance indicates how critical it is to take follow-up action.
- How can the overall score be interpreted?
Based on our experience with Immersive clients:
>=75%: Excellent
>=50%: Good
>= 25%: Fair
>=0%: Needs improvement
Note: the above score interpretations apply to Immersive scenarios. If you have created your own scenarios or have heavily customized option rankings within Immersive scenarios, take scoring factors into account when drawing conclusions.
- What does the score indicate about strengths/weaknesses in a particular topic?
- Does it indicate anything about individuals' strengths/ weaknesses?
- What does the relationship between the quality of the decision and the level of confidence indicate?
- Did participants complete the exercise straight away or following reminders? Is this an indicator of their level of engagement with the exercise?
Follow-up actions:
- If results largely demonstrate a high quality of response but low confidence:
- Provide positive feedback and reinforcement
- Exercise more often to increase confidence while simultaneously feeding back on results
- If results demonstrate overall low quality of response but high confidence, then participants are likely to have made higher-risk decisions in the exercise
- Schedule another exercise within a set time frame
- Explore the inject breakdown area which could allow you to highlight specific topics for further development or allow you to assign exercises that test those capabilities
- The exercise playthroughs by day area indicates participant engagement. This could inform how to increase engagement and drive momentum in future exercises.
- If high engagement is observed following reminders, plan a series of reminders in future exercises.
Inject breakdown
This section includes the following:
Capability by inject
This chart demonstrates the average decision score against the confidence score per inject, allowing visualization of competence at-a-glance.
Low scores selected with high confidence (bottom right of the chart) are the most problematic decisions as they indicate a lack of knowledge about the topic and unawareness of this could translate to higher risk.
Decision score by inject
The average decision score for all participants per inject.
Decision confidence by inject
The average confidence score for all participants per inject.
Inject summary
This displays the inject option feedback for the three strongest injects and the three weakest injects.
The strongest injects are the highest-scoring answers that most participants voted for.
The weakest injects are the lowest-scoring answers that most participants voted for.
Questions and conclusions
- Are there any particular strengths displayed? These'll be the injects that have been answered with a high score and high confidence.
- Are there any particular weaknesses displayed? These'll be the injects that have been answered with a low score but high confidence.
- What does the inject summary indicate?
Follow-up actions
- Review injects associated with high-risk responses: what topics are they covering and how have the injects been created?
Participant breakdown
For drill and presentation exercises, this section displays a breakdown of individual participants.
For single-player exercises, this section displays a breakdown of team capabilities.
Capability by participant/team
The average decision score against confidence by participant/ team, enabling you to visualize risk at-a-glance.
High performers
A list of your highest-performing participants/teams with details, such as average decision quality and confidence.
Low performers
A list of your lowest performing participants/teams with details such as average decision quality and confidence.
How are high and low performers calculated?
High performers are those participants or teams with an overall score equal to or higher than 50%.
Low performers are those that have scored lower than 50% in the exercise.
| Note: if your exercise was a presentation exercise, there won't be high or low performers sections. The capability by participant section will only show average decision against average confidence for one participant. This is because in presentation exercises there is only one participant selecting answers: the facilitator. |
Questions and conclusions
- Can you spot any participants/teams performing particularly well?
- Are there individuals/teams with gaps in their knowledge?
- What are these strengths/ weaknesses?
- What actions can you take to support individuals' cyber capabilities?
Follow-up actions
- Drill down into specific participants' or teams' performance by navigating to the 'Actions' column of the tables. You can then understand performance by inject and their corresponding decision quality or confidence.
- Do individuals have gaps in their knowledge that can be seen through their responses to questions on a similar topic? This could indicate that the team or organization needs to do more on those topics and/or relevant policies.
- Does an individual not know how to respond to situations directly relevant to their role? In this case, you may want to further exercise the individual on this topic by setting an asynchronous exercise, such as a drill or single-player exercise.
Next steps
Related scenarios
Access a list of scenarios related to the scenario the current exercise is based on. Related scenarios are determined by the threat and attack vectors of the exercise completed.
Related collections
Find lab collections with similar threat and attack vectors to the scenario the exercise is based on.
Follow-up actions
The 'Action' column of both tables takes you straight to the related scenario or lab collection, where you can assign this to individuals and/or teams. The decision of which collection/scenario to choose and who to assign this to will depend on the exercise's results and can be decided by delving into the inject breakdown and participant breakdown sections above.