VidEgoThink

egocentric video understanding

Except for the watermark, they are identical to the accepted versions; the final published version of the proceedings is available on IEEE Xplore. EGOILLUSION comprises 1,400 videos paired with 8,000 human-annotated open and closed-ended questions designed to trigger hallucinations in both visual and auditory cues in egocentric videos. Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in complex multimodal tasks. In this framework, we can implement temporal reasoning as a hierarchical graph neural network that learns progressively coarsened representations https://aicurated.com/ of the input video.

We plan to expand the benchmark and develop egocentric foundation models Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks for robotics. VidEgoThink is the first to propose four tasks for assessing egocentric video understanding in MLLMs for Embodied AI. VidEgoThink underscores the limitations of current MLLMs in handling first-person perspective data, thereby indicating directions for future research and advancements Although the performances of MLLMs are poor, we believe these tasks have a significant impact on downstream tasks in Embodied AI and deserve more attention. The prompts for both inference and evaluation are shown in Appendix A. Furthermore, we summarize the experimental results for different tasks, and their corresponding case studies are illustrated in Appendix B. Then, we use GPT-4o to generate feedback from different aspects for negative instances based on step annotations and the extracted keyframes.

  • EgoMimic trained on 2 hours robot data + 1 hour hand data (Blue) strongly outperforms ACT trained on 3 hours of robot data (Orange) in the Continuous Object-in-Bowl task.
  • (i) Effective feature representations of egocentric videos that benefit downstream tasks of egocentric videos, such as episodic memory, future anticipation; (ii) Novel techniques to transfer/translate between egocentric videos and exocentric videos; (iii) Improvement to retrieve ‘moments’ from past videos using a category, sentence or an object; (iv) Improvement to identify speaking faces in an egocentric video and summarize the speech.
  • For example, learning to recognise human actions can give clues as to which objects are being manipulated or what will happen next.
  • For future work, we plan to explore the possibility of generalizing to new robot embodiments and entirely new behaviors demonstrated only in human data.
  • We validate EgoPack on Action Recognition (AR), Object State Change Classification (OSCC), Point of No Return (PNR) and Long Term Anticipation (LTA) from Ego4D.

The interest of the computer vision community into the egocentric paradigm has been arising slowly entering the 2010s and it is Pegasus 1.6 Targets the Data Work Behind Robot Training rapidly growing in recent years, boosted by both the impressive advances in the field of wearable technology and by the increasing number of potential applications. Subsequently, wearable cameras were used for health-related applications in the context of Humanistic Intelligence and Wearable AI. This architecture of EgoPack is structured around a two phase training.

💾 VidEgoThink Benchmark

egocentric video understanding

While natural language is effective for human communication, it cannot be directly translated into low-level actions or grounded in the real world. Since our real world is inherently dynamic and humans frequently process substantial amounts of video data, it is crucial to evaluate the video understanding capabilities of MLLMs. VQA/HP/VG/RM indicate visual question answering, hierarchy planning, visual grounding, and reward modeling. Various egocentric benchmarks have emerged to evaluate the capabilities of MLLMs from a first-person perspective. ArXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Egocentrism typically fades as teenagers mature and develop more realistic perspectives about themselves and others.

Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding – The Association for the Advancement of Artificial Intelligence

Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding.

Posted: Sat, 14 Mar 2026 07:00:00 GMT source

To do so, we adopt a graph-based shared model and the goal is to outperform single and multi-task baselines adapted to our scenario. Different egocentric video tasks can provide different, A Survey of Large Language Models and possibly complementary, perspectives on what the user is doing. Other applications include navigation/assistive technologies for the blind, monitoring and assistance of industrial workflows, and augmented reality interfaces. More recently, egocentric cameras have been used to study human and animal cognition, human-human social interaction, human-robot interaction, human expertise in complex tasks. A collection of studies published in a special theme issue of the American Journal of Preventive Medicine has demonstrated the potential of lifelogs captured through wearable cameras from a number of viewpoints. Following this idea, the first computational techniques for egocentric analysis focused on hand-related activity recognition and social interaction analysis.

4 Reward Modeling

egocentric video understanding

Here we list the prompts for API-based models to assess the performance for some tasks. On the one hand, our real world cannot be mapped to virtual simulators exactly the MIT introduction to deep learning same way. Two key areas for the future of Embodied AI are egocentric video and multi-modal large language models.

Replanning and robustness to object and robot perturbations

To share the same model across different tasks, we propose to model video as graphs whose nodes correspond to temporal segments of the video, edges connect temporally close nodes and egocentric video tasks can be represented as different graph operations. You can use Ego4D CLI to get the original egocentric videos of Ego4d GoalStep. ALL author names match the snapshot above—including middle initials, hyphens, and accents. Finally, a set of task-specific heads project the nodes in the output space of each task. To share the same model across different tasks, we propose to model video as graphs whose nodes correspond to temporal segments of the video, edges connect temporally close nodes and egocentric video tasks can be represented as different graph operations In the second stage, the model is fine-tuned on a novel task using egopack’s cross-task interaction mechanism

Leave a Reply