SERMAS Open Call: 3Dify and 3DforXR Impress with Smart Generation of XR-ready 3D Models
,As a part of its cascade funding program, the EU-funded XR/AI innovation action SERMAS recently ran two open calls. Their aim was to expand both the development and integration of the project's core outcome (i.e. the SERMAS toolkit) – and find external parties to carry out the job. In this post, we'll take a closer look at two results of the first open call (dev) that are particularly interesting for stakeholders in the sphere of digital media, journalism, and content creation: 3Dify and 3DforXR. Both tools focus on the smart generation of XR-ready 3D models, and everybody's welcome to make free use of them.
Now what exactly can media people do with these tools?
3Dify is basically an avatar generator. It converts portraits (or selfies) into fully rigged, low-poly 3D avatars ready for real-time applications. Using AI to infer facial features, 3Dify generates an optimized mesh with UVs and bone rigging. It then exports the result as FBX and GLB files, allowing users to directly use them in tools like Unity, Unreal Engine, Blender, or any WebGL toolkit, even without prior 3D modeling expertise. 3Dify avatars are ready to use in any real-time graphics application, e.g. an XR app or a video game.
3DforXR is basically an object generator. It allows users to create exportable, XR-ready 3D models, no previous technical knowledge needed. There are three input types: Models can be created from a single image, a set of images, or a textual description (a prompt).
Use cases and target audiences
Who did the developers build this for?
3Dify is designed for anyone who needs to quickly create an avatar, but doesn't have 3D modeling skills. Indie game developers and studios can use RESTful APIs to integrate the system and generate characters on the fly. XR content creators and educators can use it to create engaging virtual instructors or guides. Corporate training and e-learning platforms can also use 3Dify to create socially compelling virtual avatars for simulations and tutorials, thereby enhancing learner engagement through personalized digital personas.
3DforXR is designed for non-expert users and XR enthusiasts who want to easily create a 3D representation of a single object. The tools caters to a diverse audience, including digital creatives, cultural heritage professionals, indie game developers, but also architects and engineers.
XR innovation
What are the most interesting features?
3Dify's most innovative feature is its ability to infer facial features from a single photo, reconstructing a user's likeness without multi-view capture.
The web editor, built in Unity and delivered via WebGL, lets users adjust facial proportions, hairstyles, clothing and accessories. Users can also preview eight built-in animations in real time. Topology generation, UV mapping, and rigging (including facial bones) are all automatic, so anyone can upload a photo, adjust parameters, and export a ready-to-use avatar without using a 3D modeling tool.
3DforXR stands out for combining more established 3D reconstruction methods with cutting-edge AI generation of models. Everything is wrapped in a clean, easy-to-use web application that also allows for the editing of models – and their instant export for any XR story or experience.
A look behind the scenes
Who did develop this, how long did it take, what were the challenges?
3Dify was developed by ISISLab, the research laboratory at Università degli Studi di Salerno (UNISA), who cooperated with the Centro Regionale Information Communication Technology (CeRict), and several digital artists. It took them roughly 12 months.
Although ISISLab had a strong foundation in XR applications and cloud computing, they initially faced two major challenges: First of all, mapping facial features from a single image required the creation of a new end-to-end pipeline for generating animation-ready topology since no prior academic or open-source work covered the entire process. Secondly, validating avatar likeness and user acceptance proved to be an inherently subjective process, as people's perceptions of resemblance vary widely.
3DforXR was developed by up2metric, an innovation studio based in Athens, Greece, also over the course of one year.
The biggest challenges proved to be the selection of the right algorithms for 3D model generation. They had to be efficient in terms of quickly and smoothly generating a broad range of 3D objects. Another challenge was designing an intuitive web application. Its 3D viewer had to be as lightweight as possible and allow model viewing and editing for mobile devices. Finally, the 3DforXR platform required a robust backend able to deal with latency and scalability issues.
User feedback
Who has tested this so far – and what do they think?
ISISLab, who also ran a 3Dify hackathon, had the system tested by roughly 100 people, many of whom also submitted online feedback and/or engaged in private sessions. The overall System Usability Scale (SUS) score averaged around 61.5, a decent result that indicates slightly above-average usability – and quite a bit of room for improvement. Users consistently praised the graphical fidelity of the tool: The avatars actually resemble the input photos. They also value the intuitive customization workflow. The most common critique centered on avatar build times, with several testers suggesting performance optimizations to speed up processing. Testers at DW also pointed out a couple of UX/UI issues and challenges when trying to host the 3Dify system on a web server.
3DforXR has only been tested by a small number of XR developers, computer vision academics and media professionals so far. The general feedback has been very positive: Based on aggregated responses, the web application scored an average of 4.5/5. Test users appreciated 3DforXR's overall performance and quality while also pointing out minor shortcomings and untapped potential.
Code, documents, and apps
Where can innovators find more info – and test the tools?
A detailed documentation of 3Dify as well as example avatars are available on the official project website. In case you'd like to self-host or extend the system, you can access the complete source code and deployment guide via 3Dify on Github.
3DforXR has a web-application up and running: Follow this link to create a free account and take the tool for a spin. Short tips are provided via UX/UI copy. For comprehensive documentation and 3DforXR deep dives, check out this page.
Feedback appreciated
Even though the main development cycles are over and the SERMAS project itself is coming to and end later this year, feedback regarding 3Dify and 3DforXR is much appreciated and will be taken into account by ISISlab and up2metric.
At DW, we'd also love to hear from journalists and XR creatives: Are you planning on featuring 3Dify avatars in your story? Have you created 3D objects with the help of 3DforXR and plan on using them in an educational piece? Have the tools helped with the heavy lifting and sped up your workflows? Can you think of a new R&D project that builds on these specific outcomes of SERMAS? Please let us know.
