Platform role required: Workforce Manager
This article includes the following sections:
Glossary: scenario v. exercise
- Scenario - A scenario is a story that acts as a template for an exercise.
- Exercise - An exercise is a playable instance of a scenario that can be assigned.
How to schedule an exercise
1. From the main navigation menu, select Exercise > Workforce Exercising.
2. Click Schedule Exercise in the upper right-hand corner.
3. From the Setup Exercise step, select a scenario from the catalog.
You can select one of our pre-populated off-the-shelf scenarios designed by experts at Immersive Labs or any custom scenarios you've created yourself.
Once you select a scenario, you can preview the scenario before finalizing your choice by clicking the Preview button.
4. Click Select to use the selected scenario:
5. From the Details panel, provide the following information for the Exercise:
- Exercise Title - defaults to the published name of the scenario
- Start Date (optional) - indicates the date that those who are assigned the exercise can begin working on it. Note that you will be able to specify a specific time here.This is based on your own timezone, as set in the platform.
- Due date (optional) - indicates when the exercise must be completed
6. From the Assign Exercise panel, select the team(s) or individual(s) to whom you want to assign this exercise.
- Assign to organization - select this option to assign the exercise to all members of your organization who are registered on the platform.
- Automatically assign to new users - select this option to assign all newly licensed users. This could be a brand new user or an existing user that has been given a new Workforce Exercising license. This option will automatically assign the exercise to any new users who join the organization before the scheduled end date of the exercise (if an end date is specified).
Note: A team assignment includes all participants that are members of the given team, at the moment the exercise is assigned. It is a static assignment and will not automatically assign new team members to the exercise, or remove the assignment if individuals leave the team.
7. From the Set Pass Threshold panel, you can define the percentage score users must achieve to pass each risk area of the exercise.
For more information on the Pass Threshold functionality and how it can automatically assign upskilling content, please refer to the Pass Threshold Details section of this article.
8. Review the exercise settings and make any amendments.
9. Publish the exercise.
This will trigger email notifications to those assigned, alerting them of the assignment and any due date set.
Top tip: When assigning exercises, provide additional communications to participants via your internal communication channels to maximize engagement. Include how completing the exercise will benefit the organization, as well as them as individuals. Many of these topics are just as relevant to people's home lives!
Pass Threshold Details
When using Workforce Exercising across an organization at scale, it's important that you can still easily identify and address where strengths and weaknesses exist without the requirement to drill down into individual response data. Our automated assignment and exercise performance functionality addresses this challenge.
How do I set a pass mark on a Workforce exercise?
When scheduling an exercise, you can now choose to set a percentage-based pass threshold. To use this functionality, you need to:
- Have rank options switched on in the scenario that you assign. This means that response options within the scenario have been categorized as either 'Great', 'Good', 'Okay', or 'Weak', with specific scores attached to each response type. Great are considered an optimal response, good a beneficial response but not ideal, okay are considered to not create further risk but also to not address the issue (i.e., a neutral behavior), and weak are considered a high risk or problematic behavior (i.e., what we absolutely do not want people to do in that scenario!).
- Have at least one risk area selected within the scenario content. This is because pass threshold scores are applied separately across different risk areas, enabling specific strengths and weaknesses to be identified so that more targeted recommendations for further learning can be provided.
All of our workforce catalog scenarios contain risk area tagging already assigned and you can also tag your own custom exercise when you create each part by selecting a risk area from the optional 'Pick a risk area' dropdown box.
More information on selecting risk areas can be found in these Help Center articles:
Pass thresholds are currently defaulted to 80%, but you can adjust this threshold when setting up your exercise to make it higher or lower to meet your organizations requirements and standards.
How are thresholds applied to individual risks in a scenario?
Pass thresholds are applied individually to risk areas within a scenario so that strengths in one area do not overshadow weaknesses in another and appropriate follow-on content can be recommended.
For instance, if your scenario contains only one risk area (e.g., all of the questions within the scenario are tagged to the 'device security' risk area) then learners will need to get 80% overall to pass the exercise. However, if your scenario contains more than one risk area (e.g., some questions within the scenario are tagged to the 'device security' risk area and some are tagged to the 'social engineering' risk area) then learners will need to get 80% in each risk area to pass the exercise. If they get a score of 90% for the device security questions and 70% for the social engineering questions and you have set a pass threshold of 80%, then they will still fail the exercise overall and we would recommend that further learning on social engineering is then completed.
In setting a pass threshold, it may be helpful to understand the scoring mechanisms that are used in workforce exercise rankings, particularly if you are using shorter exercises or those with only one question in a risk area. If a scenario with a single question is used, scoring for answers is as follows:
- A 'great' answer = 100% score.
- A 'good' answer = 75% score.
- An 'okay' answer = 50% score.
- A 'weak' answer = 25% score.
What is quick fail?
You can choose to turn on the 'Quick fail' option in addition to a pass threshold. This means that even if someone scores higher than the pass threshold for an area overall, they will still automatically fail the exercise if they select a single 'Weak' answer. Enabling this option prevents people from passing an overall threshold, whilst still demonstrating a very high risk behavior in a specific area that it would be beneficial to address.
How do I auto assign upskilling content where needed?
If you have the pass threshold turned on, you can also choose to auto assign further upskilling content to those that fail the exercise. This upskilling content is currently Immersive Labs created lab collections that have been designed to focus explicitly on the risk areas assessed within exercises. When you choose to auto-assign upskilling content, it will identify what risk areas are assessed within the scenario you have selected and will display the associated lab collections that will be assigned based on performance in these areas.
The automatic assignment will be based on exercise performance, with only those who fail the exercise being assigned upskilling content, and that content being linked directly to those areas that the individuals failed.
For example, imagine you choose a scenario that includes two risk areas (e.g., 'device security' and 'social engineering' risk areas), choose a pass threshold of 85% and do not turn on quick fail:
- Learner A completes the exercise and scores 90% on both the device security and social engineering questions. They pass the exercise and no further content is assigned.
- Learner B completes the exercise and scores 75% on the device security questions and 90% on the social engineering questions. They fail the exercise and are automatically assigned the 'Device Security' lab collection only. They are NOT assigned the 'Social Engineering' lab collection.
- Learner C completes the exercise and scores 75% on the device security questions and 75% on the social engineering questions. They fail the exercise and are automatically assigned BOTH the 'Device Security' lab collection and the 'Social Engineering' lab collection.
It is not currently possible to select alternative upskilling content (e.g., alternative or custom lab collections, additional exercises).
Do I need to set different deadlines for completion of exercise and upskilling content?
When scheduling your exercise using automated assignment functionality, you will be required to choose separate deadline dates for completion of the initial scenario/exercise and the follow-on upskilling content that will be assigned if learners fail the exercise. You will be guided through this process as part of the exercise scheduling.
We recommend that the deadline date provided for the upskilling content is at least 3-4 weeks after the initial scenario deadline, to ensure that learners who complete the initial scenario close to the deadline and fail have sufficient time to complete the assigned labs. If your scenario contains multiple risk areas and learners fail all of these, they will be assigned multiple lab collections, so it's worth considering how much time you realistically want to allow for people to complete this content.
What happens to those who do not start or complete the initial exercise prior to the deadline?
Learners who do not start or do not complete the exercise prior to the deadline will not receive any associated upskilling content. Automatic assignments will only be assigned to those who complete the initial exercise.
You will be able to download a list of users who have not completed the exercise from the exercise results area. This user list is provided in a format that will enable you to bulk upload these users into a specific group within the platform in the future should you wish.
What about those who do not start or complete their upskilling content prior to the deadline?
Learners who do not start or complete the upskilling content prior to the deadline will not complete the overall exercise assignment and will not show as complete in the Audit reporting.
What will I see in my assignments?
You will need to be a workforce manager to use the auto assignment feature. You will see your assigned exercise in the 'Managed by me' area of the platform. You will also see any assigned upskilling labs in this area of the platform.
As a learner, you will see the exercise assignment, and any automatically assigned lab collections, in the 'Assigned to me' area of the platform.
What will I see in my results?
As a manager, you will be able to view a summary of your exercise results by selecting the exercise assignment and then selecting 'View results'. In this area, you will see a breakdown of user progress, including:
- The proportion of users who have not yet completed the initial exercise.
- The proportion of users who have failed the exercise.
- The proportion of users who have passed the exercise.
If you have set-up auto assignment of upskilling content, you will also see:
- The proportion of users who have been assigned upskilling content.
- The proportion of users who have completed this upskilling content.
If no further learning has been assigned due to all learners passing the exercise, this will also be displayed.
You can also select 'View Exercise Results' within this page to be taken to more detailed performance reporting, including a downloadable 'audit table'.
You can also access this area of the reporting directly from 'Reports' - 'Capability by Area' - 'Workforce Exercising'. You will then need to select the individual exercise using the 'Exercise Name' dropdown field.
You will need to be a workforce manager to access this information.
As a learner, you will see a breakdown of your performance across all risk areas included in the exercise (this will be a minimum of one risk area and up to a maximum of eight risk areas) and links to any automatically assigned upskilling collections.
What information is included in the Exercise Audit Table?
The exercise audit table currently includes the following information:
| Column | Definition |
|---|---|
| user_email | The email set against the account of the assigned user. |
| workforce_exercise_name | The name of the workforce exercise. |
| workforce_exercise_closed_at | The UTC datetime (precision in seconds) the workforce exercise was closed, either manually by manager or by the pre-configured end date being reached, or by all assigned users completing the exercise. |
| workforce_exercise_assigned_at | The UTC datetime (precision in seconds) the workforce exercise was assigned to the user. |
| workforce_exercise_completed_at | The UTC datetime (precision in seconds) that the workforce exercise was completed by the user. |
| pass_threshold_enabled | True if the pass threshold toggle was enabled on the exercise, false if not (default). |
| pass_threshold_value | The pass threshold value configured if pass threshold enabled, null if pass_threshold_enabled = false. |
| pass_threshold_quickfail_enabled | True if the quickfail toggle was enabled on the exercise, null if pass_threshold_enabled=false. If this is set as true the user will fail the risk area if they select the weak option as an inject response. |
| pass_threshold_auto_assignments_enabled | True if auto-assignments are enabled on the workforce exercise, null if pass_threshold enabled = false. |
| number_of_failed_risk_areas | If pass_threshold_enabled, the count of risk areas failed in the user's workforce playthrough against the pass_threshold_value and quickfail setting. If pass_threshold_enabled=false this value is null. |
| number_of_auto_assignments | If pass_threshold_auto_assignments is enabled, the number of assignments of follow-on content to the user from their exercise playthrough results. If pass_threshold_enabled = false then this value is null. |
| number_of_completed_auto_assignments | If pass_threshold_auto_assignments is enabled, the number of assignments of follow-on content completed by the user. If pass_threshold_auto_assignments=false this value is null. |
| all_tasks_completed_at | The UTC datetime (precision in seconds), that the user completed all tasks related to the workforce exercise. If pass_threshold_auto_assignments_enabled = true, this is when the user completed the final auto-assigned content if they were auto-assigned content, otherwise it is the timestamp they completed the workforce exercise. |
What if a user is assigned the workforce exercise but the user has their access revoked?
The user will still appear in the table and assignment.
What if a user is assigned the workforce exercise but all their licenses are revoked?
The user will still appear in the table and are still part of the assignments, but the workforce assignment will not appear for the end user. The user now does not appear in the platform (e.g. in the users area).
What if a user is assigned the workforce exercise but their workforce exercising license is revoked?
The user will still appear in the table and are still part of the assignments, but the workforce assignment does not appear for the end user.
What if a user is assigned the workforce exercise via a team assignment but is then removed from the team?
The user will still appear in the table as they are not removed from the assignment.
What if a user is assigned the workforce exercise but removed before the exercise is closed and before they started a playthrough of the exercise (removed via any method)?
The user will not appear in the table as they are removed from the assignment.
What if a user is assigned the workforce, completes the workforce exercise and then is removed (via any method)?
The user will appear in the table as a user who has completed the workforce exercise cannot be removed.