Although the terminology may sound lofty, the idea here is relatively simple. When attempting to monitor a complex instrument panel, like the one you might find in a cockpit, there may be dozens of information streams that must be monitored. Fuel, altitude, pitch, yaw, airspeed, and on and on. It’s only possible for a person to monitor one data stream at a time, so the system operator (the pilot in this example) has to try to sequentially direct attention to each of the relevant instruments. This must be done often enough to have a current and comprehensive understanding of the current system states, but not so often as to interfere with a primary task (like watching where you’re going, in the case of the pilot).
One problem becomes clear: because each instrument is constantly streaming information, the system operator will not be able to attend to the vast majority of the information the system is producing. Essentially, it’s like trying to catch the water from multiple faucets at the same time, using a single cup. This means potentially important information could go unreceived.
My proposal to overcome some of these concerns involves the use of an eye-tracker to track the operator’s gaze, monitoring the fixations made on each instrument, including when they were made and the instrument’s value at the time. The eye-tracker can then pass this information back to the instrument panel. This serves essentially to compare notes, where the instrument panel says “I sent X, Y, and Z, but I see you only received X and Y, so you likely don’t know about Z”. This establishes a rudimentary Theory of Mind regarding the knowledge state of the operator, and is detailed below.
Once this Theory of Mind regarding the operator’s likely knowledge states is established, it is possible to intelligently direct the operator’s attention to high-value items- instruments that haven’t been sampled in a while, or have changed significantly since last fixated. This allows for the augmentation of Situational Awareness.
Situational Awareness refers to not only the perception of the relevant stimuli in your environment, but having a holistic understanding of the implications of the status of those stimuli such that you may make the best possible decision. Lapses in Situational Awareness are one of the most highly implicated causes of aviation accidents.
With this proposed data exchange between the operator and instrument panel, there are a host of applications we hope to explore in the near future. These include:
- Maintenance and augmentation of Situational Awareness
- Reduction of working memory and prospective memory demands, offloading some of these to the network
- Scan path training, in order to teach the operator how to most efficiently scan the instrument panel
- Checklist monitoring, ensuring checklists are properly completed
Patrick Lawson, PhD 