Open call for VOXReality Initiative

VOXReality is an initiative that aims to facilitate the convergence of Natural Language Processing (NLP) and Computer Vision (CV) technologies in the Extended Reality (XR) field. They will develop innovative Artificial Intelligence (AI) models that will combine language as a core interaction medium supported by visual understanding to deliver next-generation applications that provide comprehension of users goals, surrounding environment and context.

VOXReality will support a minimum of 5 external institutions as beneficiaries throughout a 1-year programme to extend application domains, providing with 200K EUR equity-free funding. The goal is to integrate VOXReality models into new XR applications, thus advancing immersive experiences across various sectors.

Objectives

• Improve human-to-machine and human-to-human XR experiences
• Extend and improve the visual grounding of language models
• Widening multilingual translation and adapting it to different contexts
• Provide accessible pretrained XR models optimized for deployment
• Automating the generation of virtual agents using multimodal information
• Demonstrate clear integration paths for the VOXReality pretrained models

What are the challenges?

• Integration challenge
  • Integrate VOXReality models into new XR applications, expanding their functionality and applicability.
    • Integration of at least one VOXReality model into an XR application.
    • Development of a fully functional XR application.
    • User testing with at least 30 participants and a comprehensive report on outcomes.
• Extension challenge
  • Extend VOXReality models to new languages, directions, or tasks, enhancing their capabilities and performance.
    • Utilisation of VOXReality models as pre-trained models.
    • Literature investigation on selected tasks for model extension.
    • Training and adaptation of new models with benchmark testing and comparison.
• Full-cycle challenge
  • Extend VOXReality models while integrating them into new XR applications, pushing the boundaries of both research and integration.
    • Proof-of-concept demonstrating XR solutions in new application domains.
    • Integration of VOXReality models into innovative XR applications.

What does VOXReality offer?

VOXReality programme offers an opportunity designed to empower visionaries like you to redefine the boundaries of extended reality (XR) experiences. With the technology and support, you can unleash your creativity and bring your boldest XR ideas to life. Here’s a closer look at the technologies offered:

• Elevate your XR applications with immersive visual experiences. The visual language models provide spatial descriptions, image captioning, and question answering functionalities, enabling interaction with RGB images.
• Break language barriers effortlessly. From audio transcription to contextual translation, the  tools ensure seamless communication across multiple languages, including consortium languages such as: English, Dutch, German, Spanish, Italian, and Greek.
• Enhance user engagement and immersion with conversation agents tailored for conference-related information and training instructions. The agents provide intent recognition, navigation assistance, and program-related information retrieval, ensuring seamless user interactions in XR environments.
• Furthermore, they will also provide developer tools, publicly shared AI models and deployment guidelines tailored to help you navigate the intricacies of immersive experiences.
• Developer Tools: Simplify your development process with the suite of developer tools. From model optimisation to seamless deployment, the tools streamline every aspect of XR application development, allowing you to focus on innovation.
• Publicly Shared AI Models: Join the thriving community of innovators on Hugging Face, where all the AI models are publicly shared. Collaborate, iterate, and innovate with fellow developers to unlock new possibilities in XR technology.

Who are they looking for?

• Are you an innovative thinker eager to shape the future of extended reality (XR) experiences? You can apply if you are:
  • Single entity: Micro, small and medium-sized enterprises (SMEs); or
  • Consortium of maximum of 2 entities: Micro, small and medium-sized enterprises (SMEs).
• If this is you then you are what they are looking for!

For more information, visit VOXReality.