Enhanced pilot environment

The outcome should contribute to:

• European platforms for enhanced combat pilot technologies tests and demonstrations to welcome joint or national tests and demonstration needs.
• Consolidation of a sector of excellence in the EU for enhanced combat pilot based on innovative technologies.
• Generation of inputs for the mid-term and long-term development of next generation air combat cockpit HMI.
• The development of novel cockpit HMI technologies.
• Increase mission capability, efficiency, effectiveness, and performance in air combat missions (e.g., safer for pilot and helping to limit the collateral damage) exploiting the emerging technologies.
• Provision of a potential starting point for developing EU guidelines in the frame of advanced HMI design for managing systems-of-systems operations.
• Provision of an opportunity for cross-ministries of defence and cross-industries exchanges in the subject of cockpit design and pilot operating procedures.
• Strengthen EU industry in advanced air combat cockpit technologies independent of third countries.
• Quick wins identification to be implemented on current or upcoming systems.

Future warfare is likely to be largely characterised by the networking of combat systems, including unmanned autonomous assets, and a high degree of automation of many systems. As a result, a large number of actors, sensors and effectors may be connected, generating an astonishing collection of information and data.

The challenge is therefore to provide the pilot with the appropriate situational awareness, understanding of system modes and status, and the ability to act and react in a timely manner to ensure mission success.

This requires the development of new equipment and associated software that can take advantage of new technologies such as wearables, optics, haptics, voice command, virtual operator assistants, augmented reality, and 3D holography.

Based on appropriate implementation concepts, this would free crew members from repetitive tasks, allowing them to focus their resources on high-value areas of action, and also support in mission execution, thereby improving combat effectiveness.

Specific objective

From the point of view of the human-machine relationship, the new generation of military aircraft involved in this collaborative air combat is likely to require a new generation of human-machine relationship that allows ergonomic cooperation between the crew and the machine, effective and safe flight, as well as cooperation with other assets, including unmanned ones. The new technologies would make it possible to gain a tactical advantage by assisting the crew as a real teammate, responding to requests, suggesting tactics and procedures, and adapting interfaces to the pilots' and/or operators' status and needs.

The definition of a novel design and interaction principles for managing automated and autonomous aircraft cooperating with System-of-Systems (SoS) teammates, including adaptive interfaces can be defined as Human-Machine Teaming (HMT).

Taking into account the new paradigm of human-machine teaming in future collaborative and connected air warfare, this call topic aims to address the following areas:

• New or disruptive Human-Machine Interface (HMI) technologies, such as displays, wearables, vocal dialogue, augmented reality, stereoscopy.
• Pilot status monitoring in relation to the mission and systems status.
• Assisted decision-making support based on advanced techniques like Artificial Intelligence (AI) not excluding other approaches.

Preliminary analyses show that, in order to meet these challenges, future European air combat systems must be equipped with an innovative cockpit offering the pilot groundbreaking display and interaction capabilities. In this context, it seems clear that new products (e.g., head-down, eyes-out, interface modalities, virtual assistant) have to be developed.

With a view to contributing to the development of new generations of air combat and training aircraft and systems in the EU, including existing manned and unmanned air platforms, or to upgrading those currently in service, the proposals must:

• Mature the required cutting-edge technological and engineering solutions for future enhanced pilot environments, including cockpits, through evaluation and demonstration in representative operational scenarios.
• Produce common high-level “platform agnostic” specifications and guidelines for HMI design for future cockpit equipment.
• Specify and perform, where applicable, prototyping activities for cockpit equipment incorporating these maturated solutions.

Proposals must therefore address the following four areas:

• Adaptive human system collaboration to improve tactical situational awareness and enable ergonomic crew-machine cooperation for safe flight and high performance in cooperation with both manned and unmanned assets. Adaptive collaborative HMIs are required with a view to enhanced human-machine/human-human/machine-machine teaming for operations in a distributed environment with multi-platform assets. Novel design and interaction principles are also required for the management of automated/autonomous aircraft functions and collaboration with SoS teammates, including adaptive interfaces.
• Visualisation: both visualisation products and advanced pilot information presentation capabilities, including 3D presentation, and other novel presentations, through for instance but not limited to:
  • Augmented reality, large area displays (free form, multi touch, auto-stereoscopy), 3D holography and implementation concept.
  • Helmet Mounted Display (HMD) solutions which are crucial for the next generation cockpit.
• Crew monitoring system (CMS) to monitor in real time the physiological and cognitive states of the crew, through systems and techniques, enabling the adaptation of the HMI in a way to support and assist the aircrew in performing the flight and mission control in demanding operational environments.
• Interaction modalities to address both the modalities of interaction as well as their combination through innovative HMI technologies, such as but not limited to wearable, optics, haptics, vocal command, and virtual operator assistant.

Types of activities

The following types of activities are eligible for this topic:

Accordingly, the proposals must cover at least the following tasks as part of mandatory activities:

• Studies:
  • In the area of adaptive human system collaboration:
    • Explore and mature innovative HMI principles for cross platform mission management considering (human-machine/human-human/machine-machine teaming, not including the specific functional algorithms).
    • Explore and mature adaptive HMI mechanisms (e.g., based on CMS outputs and in accordance with the specific operational context).
  • In the area of visualisation:
    • Explore and mature technological solutions for increasing technical characteristics in terms of presentation field and functional capabilities, such as but not limited to:
      • Digital integrated night vision.
      • Primary flight display function.
      • Enhanced synthetic vision system (including live virtual constructive visual integration).
      • Target designation and view through the cockpit.
      • HMD wireless link, also considering the control of inertia characteristics (i.e., mass and centre of gravity of the HMD carried by the pilot's head).
  • In the aera of CMS:
    • Explore and mature solutions for monitoring the physiological and cognitive states of the crew, for operational embedded systems as well as to training systems (embedded or on ground) from all relevant sources of information, through for instance but not limited to:
      • Support for CMS sensors,
      • Operator incapacity (e.g., G-LoC, hypoxia, spatial disorientation),
      • Hypo-vigilance, including mind-wandering and surprise effect,
      • Attentional tunnelling, including visual/auditory tunnelling,
      • Mental workload and mental fatigue,
      • Stress, including mind-blocking,
      • Engagement level and ability to collaborate,
      • Situational awareness.
    • Validate mature CMS models, considering a Pilot Behavioural Knowledge Base (PBKB) that needs to be contextualised in accordance with the diversity of humans, missions, and tasks, including through AI- and Machine Learning (ML)-based techniques.
  • In the area of interaction modalities:
    • Explore and mature solutions for both the modalities of interaction as well as their combination, through:
      • Sound, in terms of input/outputs: voice command, natural language processing, in a very constrained environment such as that of a fighter, voice synthesis and advanced audio functions such as 3D sound.
      • Vision: eye-tracking that is used as a CMS sensor to be dedicated to interaction, coupled with another modality such as voice, with a view to increasing efficiency for target designation in an eyes-in or eyes-out use.
      • Touch: study multi-touch (up to 5 fingers) technologies to interact with the displays.
      • Gesture controls.
      • Haptic/tactile display of information.
• Design:
  • In the areas of adaptive human system collaboration, visualisation, CMS and interaction modalities:
    • Perform demonstrations of the solutions developed with physical, digital mock-up and/or simulations that is relevant for future fighter cockpit environment, based on operational use-cases.
  • Conduct an iterative implementation of findings to continuously optimise the performance of the demonstrations.
• Prototyping:
  • Considering that prototyping is a mandatory activity, prototypes of the developed solutions must be built, where applicable, and providing a sufficient maturity is reached, in order to allow for a timely integration in other national or multinational development projects related to next generation aircraft.

In addition, the proposals may:

• Refine operational use-cases where needed in the context of SoS architectures and identify and elaborate on structuring dimensioning elements.
• Evaluate and assess increased mission capability and impact on pilot/crew workload.

The proposals must substantiate synergies and complementarity with foreseen, ongoing or completed activities in the field of air combat, notably those described in the call topic EDF-2021-AIR-D-EPE related to Enhanced pilot environment.

Functional requirements

The proposals should fulfil the following requirements:

(1) in order to support the future air collaborative combat:

• Take into account the new paradigm of HMT in the future collaborative and connected air warfare and adaptive cooperation between all systems, either manned or unmanned, involved in multi-assets operation.
• Ensure muti-modularity to provide greater security, resilience, and accuracy by removing ambiguity about the operators’ intentions.
• Ensure flexibility and adaptability of the HMIs to meet the demands of future combat systems.

(2) in order to improve human-machine performances:

• Human-machine performance should be evaluated according to different criteria to be defined in the proposals.
• Human factor aspects should be considered to develop the technologies, especially physical and cognitive ergonomics.
• Physical ergonomics should fit the air crew anthropometrics data to ensure that physical interfaces are adapted to any crew.
• HMI should allow:
  • Strengthened and adaptive cooperation between all systems, either manned or unmanned, involved in an operation.
  • Human supervised delegation of tasks to increasingly autonomous systems.
  • Intelligent assistance to provide the crew with system proposals and to adapt interfaces.
  • Piloting performance monitoring.

(3) Specific technical requirements:

• The technologies should be scalable for existing fighters or future fighters in order to apply the “quick win” principle.
• Developed technologies, concepts, solutions guidelines, specifications should be platform agnostic.
• Each technological building and capability blocks should be evaluated and demonstrated through physical or digital mock-ups and simulations, based on representative use-cases.
• The treatments of data collection about humans to build models or algorithms must be compliant with the EU General Data Protection Regulation (GDPR).