CFPs: Privacy-Preserving Human-AI Dialogue Systems – Organisation of a Technological Challenge

Objective

• Human-AI dialogue systems offer impressive results but are still prone to errors of various types. Moreover, there is no established metric to measure system performances. In order to ensure trustworthiness and steer progress, these systems should be submitted to common tests using shared data and clear metrics and protocols.
• The goal of this call topic is thus to set up a testing environment and organise a technological challenge to evaluate the performances of such systems for defence use cases, including their abilities to manage classified information and to justify their answers. The challenge should be open to research teams supported through another call topic (EDF-2025-LS-RA-CHALLENGE-DIGIT-HAIDP) and possibly by other sources of funding. Representative defence users should be involved to contribute to the definition of the use cases and associated data, to test the demonstrators produced by the participating teams, and to provide feedback.

Scope

• The proposals should address the organisation of a technological challenge on human-AI dialogue based on the preliminary evaluation plan provided as part of the call document (cf. Annex 4). This includes the collection, annotation and distribution of data, the elaboration of evaluation plans and metrics, the measurement of system performances, and the organisation of debriefing workshops.

Funding Information

• Total Budget: €7,000,000
• Funding Type: EDF Lump Sum Grants.

Eligible Activities

• The following types of activities are eligible for this topic:
  • Activities that aim to create, underpin and improve knowledge, products and technologies, including disruptive technologies, which can achieve significant effects in the area of defence (generating knowledge) – Optional
  • Activities that aim to increase interoperability and resilience, including secured production and exchange of data, to master critical defence technologies, to strengthen the security of supply or to enable the effective exploitation of results for defence products and technologies (integrating knowledge) –  Mandatory
  • Studies, such as feasibility studies to explore the feasibility of new or upgraded products, technologies, processes, services and solutions) – Optional
  • Design of a defence product, tangible or intangible component or technology as well as the definition of the technical specifications on which such design has been developed, including partial tests for risk reduction in an industrial or representative environment – Optional
• The proposals must cover at least the following tasks as part of mandatory activities:
  • Integrating knowledge:
    • Setting-up of the infrastructure for testing human-AI dialogue systems in the framework of the technological challenge.
    • Elaboration of data annotation guidelines, collection and annotation of data, quality assessment, distribution and curation of databases.
    • Organisation of the evaluation campaigns, and in particular.
      • coordination of the exchanges with the participating teams and any other relevant stakeholders on the evaluation plans and elaboration of these plans.
      • management of the experimental test campaigns and of the objective measurements of the performances of the systems submitted to the tests by the participating teams according to the protocols and metrics described in the evaluation plans.
      • organisation of the debriefing workshops.
  • The proposals should include descriptions of work packages, tasks and deliverables that enable a clear assessment of work package completion. These should include the production of detailed evaluation plans agreed upon by all stakeholders, the production of the annotated databases needed for the evaluations, the production of measurements for all systems submitted to the tests by the participating teams following these plans, and the organisation of the needed events.

Expected Impact

• The outcome should contribute to:
  • Standardisation of testing for dialogue systems.
  • Enhanced clarity on the performances of dialogue systems for all stakeholders, including system developers, funders, and users.
  • Community building at the European defence level.
  • Trustworthy dialogue systems that enhance operational decision-making.
  • Availability of databases to further develop dialogue systems.

Eligibility Criteria

• In order to be eligible, all applicants (beneficiaries and affiliated entities) must cumulatively:
  • be legal entities (public or private bodies)
  • be established in one of the eligible countries, i.e.:
    • EU Member States (including overseas countries and territories (OCTs))
    • non-EU countries :
      • listed EEA countries (‘EDF associated countries’)
  • have their executive management structure established in eligible countries
  • not be subject to control by a non-associated third country or non-associated third-country entity.

For more information, visit EC.