XR & Immersive Journalism

V4Design: Processing Traditional Media Assets for Interactive Immersive Experiences

Wouldn’t it be great to experience the Beatles play their first Hamburg gig from the perspective of a fan who danced in the front row? Or witness the fall of the Berlin Wall in a truly immersive way? The bad news: That’s still kind of tricky. The good news: It’s not impossible, and the V4Design consortium has been working on technical solutions for a full year now.

V4Design in a nutshell

Horizon 2020 project V4Design started in January 2018. One of its main goals is to recycle all sorts of 2D visuals for reuse in virtual environments. Use cases focus on architecture, design, and–this is where DW comes in–documentary film making and language acquisition. The consortium is currently developing plug-ins for Rhino 3D (a computer graphics and CAD application software) as well as add-ons for Unity (a cross-platform game engine). These extensions connect with V4Design media library items, templates, filters, and other assets that allow producers to create immersive experiences.

The original V4Design conceptHammer & Tusk

From Gendarmenmarkt to Bauhaus

The first DW use case, realized with the help of software engineering experts Nurogames, is about creating an immersive documentary:

In a first step, we virtually rebuilt the Gendarmenmarkt in Berlin and sifted through our 2D video and picture archives where we hoped to find enough interesting objects and characters to rehash–and thus bring the square to life in VR, in different time periods.

Although the prototype was completed successfully and the engine per se worked fine, the overall results didn't meet our high quality standards: That's because a lot of the footage (and basically all of the material recorded until the late 2000s) proved difficult or impossible to recycle. There wasn't enough camera movement, there was a lack of depth, a lack of resolution, and no user generated content (UGC) available.

Hence we decided to adjust our use case: We kept the basic setup (and the original pilot), but started looking for a "fresh" topic/setting with significantly more and better coverage. Eventually, we started to focus on the 2019 Bauhaus anniversary. So far, it seems to offer all the media assets we need–and it perfectly reflects the mix of journalism, architecture, and design at the core of our project. Diving into era-defining Bauhaus surroundings is both compelling and technically impressive. Stay tuned for a couple of teasers and demos.

A participant of the 2018 Reseachers’ Night in Thessaloniki testing a V4Design prototype at the CERTH exhibition stand.V4 Design Consortium

From 200 episodes to one immersive game

Our second use case, also a joint-venture with Nurogames, aims to turn a special kind of telenovela into a VR experience.

Nicos Weg is a self-study German language program featuring a combination of regular TV episodes and interactive exercises. Created by DW/DW Akademie and the Federal Employment Agency, it follows Nico Gonzalez, a young man who recently arrived in Germany. He would like to learn the local language–and understand the German way of life.

Adapting Nico for VR isn't a walk in the park, either. On top of the "flattie goes immersive" transformation challenge, we had to deal with an enormous amount of Nico content: There are more than 200 episodes, a lot of settings, and a pretty complex didactic concept. Lessons include some 14.000 exercises and 19.000 audio files.

Our solution this time: Focus on one setting (Nico's apartment), partly re-shoot, and do laser scans. Focus on essential language exercises. Create a reduced complexity VR serious game that is both educational and entertaining–just like the original telenovela. The Nico demo is coming a long fine and should be ready for evaluation later this year.

Scene from Nicos WegDW

First official V4Design review

The efforts of the entire V4Design consortium, which also includes The Centre for Research and Technology-Hellas (CERTH)–Information Technologies Institute (ITI), KU Leuven, Universitat Pompeu Fabra (UPF), Herzog & de Meuron (HdM), The Aristotle University of Thessaloniki (A.U.Th.), Solaris Filmproduktion GbR (SFd), ArtFilms Ltd (AF), and Europeana Foundation (EF), was reviewed at the DW headquarters in Bonn just a couple of weeks ago. EU H2020 Reviewers were impressed with how the project is tackling the major challenge of content repurposing for VR. There was also praise for the smart legal concepts with regard to intellectual property and privacy–and for the overall team work.

The V4Design teamV4 Design Consortium

V4Design has now successfully started into its second year, which will see a lot of internal and external evaluation and–quite possibly–a first prototype of the envisioned authoring tool for Unity.

It might take a while until you're able to meet Nico at the Indra Club in 1960 or Checkpoint Charlie in 1989 (with a couple of Breuer chairs and Wagenfeld lamps in the background)–but we're getting there!

To learn more about V4Design, check out the project's official website and Twitter account. V4Design is looking for volunteers: If you're interested in joining a user or focus group, send an email to eva.lopez@dw.com.