Sixth Sense Retrieval: A Sneak Peak at the Finished Prototype

Earlier this year, we introduced you to Sixth Sense Retrieval, a DNI prototype project dedicated to analyzing and indexing vast amounts of video content in order to unlock media archives. In the meantime, 6SR has been completed, and everybody at MindFlux and DW is quite happy with the results. Within the project scope, we could implement all envisioned features, which can now serve as a basis for a future, more sophisticated tool. Here's a quick look at what the platform looks like after 6 months of development and testing.

The finished 6SR prototype is able to import all currently available DW video content and analyze it with the help of Google's Cloud Video Intelligence.

An easy-to-use interface allows content producers and archivists to search for

  • an object ("tree")
  • a concept ("nature") that emcompasses a number of related objects that can be predefined ("plant", "tree", "forest")
  • several more or less unrelated objects ("tree", "man", "shirt")

Objects and concepts are color coded and represented in a timeline right below the individual video bits. The darker the color, the more likely it is 6SR has found an exact match.

Search results can be displayed in different ways:

  • "Labels" will display all search terms found in a specific video, starting with the most common; a timeline with markers also tells users where (or better yet: when) to find an object in a video
  • "segments" will display all uninterrupted scenes (no cuts) along with the objects found in them (and a timeline and a key frame)

6SR's powerful semantic search is based on word embedding and the WordNet lexical database. The platform helps users find relevant video content even when there are no direct object matches–without supplying false, confusing results.

Just a small part of the platform’s comprehensive food vocabulary

At DW, the 6SR prototype received nothing but positive reactions. Our test users could instantly see the benefit the tool would have for content curation and creation once it is properly trained, scaled up, and integrated. Producers would be able to archive their content in a sustainable way. Archivists, often overwhelmed with requests, would be able to dramatically speed up their workflows, especially since 6SR would also be great for reindexing legacy content.

With the support of additional funding, we would be able to further develop 6SR and work towards implementation in the media sector. The ultimate vision is a platform that provides the advantages of all current computer vision technologies (face recognition, optical character recognition, location detection, saliency detection) and analyzes all video data in a flash.

Drop us a line for more information on 6SR.

Alexander Plaum
Olga Kisselmann