This data challenge focuses on developing AI approaches that can turn multimodal physiological measures into accurate quantitative assessments of cognitive state/cognitive workload. Such technology can be applicable to ground training, in-flight training, and any task that’s cognitively demanding and has the potential for optimizing the training outcome.

There are two predictive modeling tasks for you to complete in this data challenge. Given the physiological data measured from a subject during a run:

• Challenge Task 1: Predict the difficulty level of the run (there are 4 difficulty levels, thus this is a multiclass classification)
• Challenge Task 2: Predict the performance error of the run

Not for this data challenge. But solutions that leverage open source resources are preferred.

You can always reach the challenge organizers by sending an email to: cogpilot@mit.edu.

You can also join the CogPilot Data Challenge 2.0 Slack channel and post your questions and comments there. It’s a community for participants to discuss data challenge related topics.

Anyone interested in the intersection of AI, human performance, physiology, cognition, and flying! All participants are welcome.

The data challenge will begin in the middle of October and continue until Spring 2023. Registration will remain open for the entire Data Challenge.

This challenge will be hosted fully virtually.

Please have all members register individually and list their team’s name.

No, but usually a team of 4-8 is recommended.

The dataset is freely available on Physionet: https://doi.org/10.13026/azwa-ge48.

Please check out the reference folder that comes with the challenge dataset download. It contains a wealth of information for this data challenge.

The dataset includes a PerfMetrics.csv that includes, for each run, both the difficulty level label (“Difficulty”) for Challenge Task #1 and the total flight performance error (“Cumulative_Total_Error”) for Challenge Task #2.

Trainees are first outfitted with a suite of wearable sensors. Then, they perform a 5-10 minute practice run on the easiest difficulty level to get familiarized with the scenario. The experimenter loads each scenario to the same starting location, aircraft state, and aircraft attitude, with the simulation “paused” so there’s no aircraft movement. The sequence of difficulties is known to the experimenter but unknown and “random” to the subject. The experimenter then starts the data recording. The only differences between levels of difficulty are changes in weather (visibility, wind, turbulence, and height of cloud ceilings). The start of the data recording happens slightly before the start of the flight. In practice, the simulation is paused, the data logging begins, the simulation is then un-paused, and the participant begins the run. One could plot the aircraft airspeed to see that the initial few points have zero velocity, but it jumps to ~115 knots once the simulation is un-paused. After the trainee lands the aircraft (or crashes), the experimenter asks the trainee to take their hands off the controls and then ends the recording of the data. The experimenter asks the subject to state the level of subjective workload they experienced by completing a Bedford workload assessment at the end of each run. At this point, the next run is loaded. Please see the CogPilot datasheet contained in the dataset for more detailed information.

There is a 5-min Rest period before the 12 runs and after. During the rest periods, trainees sit quietly with all the sensors recording data. The rest periods may provide information regarding the physiological baseline of an individual.

Runs are approximately 7 to 10 mins and may depend on the speed of flight, or errors in movement. Sometimes, a novice subject may crash during the virtual flight, so the run may be shorter. Each of the 12 runs is one of the four difficulty options.

For each modality, the data and time vectors are aligned, and the linking time point is listed in the first column. The time listed in the first column is universal across modalities. However, the first timestamp between two different modality files (e.g., Subject001_EDAfile vs Subject001_EMGfile) may not be the same. One modality stream might start slightly ahead of the other. Nevertheless, the timestamps will be the truth to identify the slight time difference.

No, because of head movement and rotation, the position of the instrument panel may change in the VR space relative to the person’s eye tracking gaze.

We use Trial and Run interchangeably. In the data files, we use “runs”.

The entire dataset available for download is for you to develop your AI models. You can partition the data however you like for the goal of model development. A typical approach is to use cross validation. We have an independent dataset outside of the downable dataset for evaluation.

The starter code has been written in Python. However, you are welcome to use any language that you are comfortable with.

While there are good ML courses online, here we recommend a ML course taught by one of the MIT co-PIs of the CogPilot project. Course Link: https://tamarabroderick.com/ml.html.

Please refer to the Data Challenge Description Tab for details.

If you’d like to be a subject, please email cogpilot@mit.edu (data collections occur at MIT Campus in Cambridge, MA)

We use the Professional version of the X-Plane 11 simulation engine at https://store.steampowered.com/app/269950/XPlane_11/. We use the HTC Vive Pro Eye headset, but X-Plane 11 should be compatible with other types of VR headsets. We use a premium (paid) model of the T-6A Texan II aircraft, which is the aircraft used in the first flying phase of Air Force pilot training. This model is developed by FliteAdvantage and is available for the public to purchase here: https://www.fliteadvantage.com/product/t6a-virtual-model/.