PI: Aparna Bharati
Co-PI(s): Maryam Rahnemoonfar
University: Lehigh University
Industry partner: Celebr8 Life, Inc.
Organizing vast amounts of visual content is significant for various personal and institutional needs. While there are existing techniques to cluster and summarize visual content, most of them do not account for user behavior such as taking multiple very close-looking pictures (near-duplicates) and subtle user preferences. Celebr8 Life, Inc., a startup located at the Ben Franklin Tech Ventures building on the Lehigh campus, is pursuing to enhance the existing capabilities of their Intelligent Video Engine to provide better functionality for their user. Their product aims to deliver digital memories from photo and video content gathered across multiple devices such as Apple or Android mobile phones, GoPro digital cameras, and other digital sources. The goal of this project is to advance state-of-the-art clustering and summarization using a hybrid of user and AI-driven processes such that the user retains some flexibility without having to manually triage hundreds of media files. This includes developing a semantic clustering technique that is aware of near-duplicate images and subtle user preferences and a multi-modal (input is a mix of images and videos) summarization technique which not only takes into account visual changes between consecutive input elements but also highlights provided by the user.
This will help suggest sequencing to best render in a highlight reel format. Once suggested sequences are selected by the user, this feedback will be used to fine-tune the deep learning model for personalization with the expectation to continuously improve the suggested sequence. This project is a collaboration between research, and industry through a partnership between the Computer Science and Engineering Department at Lehigh University and Celebr8 Life, Inc. The results of the project will enable Celebr8 Life to gain the user-friendly capabilities needed for commercialization.