Recursive and Vioso turn the point of sale into an immersive experience

The project, that merges the virtual and physical world, It was made in a fashion store in Shenzhen and for its development Vioso software was used and six 4K laser projectors were installed. 21.000 lumens on walls and floor.

Vioso has helped the audiovisual consultant Recursive to transform the store of a high-end fashion brand in Shenzhen (China) and turn it into an immersive experience, based on real-time projection.

Recursive has a long history of developing audio and video hardware and software-based design solutions for a number of high-profile brands around the world.,. When a high-end fashion client decided to create a fully immersive, projection-based real-time rendering experience in their Shenzhen store, Recursive was the choice.

The client, who had seen the work done at the Puma store on Fifth Avenue in New York, wanted a multimedia experience that would take its customers to “an enchanted journey of discovery” through a magical and ethereal landscape with real-time interactions between the virtual and physical world.

“We were tasked with developing the architecture of the system and interpreting a creative idea to translate it into a physical solution.. It wanted users to experience four different worlds based on four central themes in the natural world., “triggered by social interaction and adapted to the movements of each user within the space.”, comments David Yates, CEO of Recursive.

To carry out this project they needed a real-time rendering environment and this would require a method to adapt the content to the shape and layout of the physical space., which had a somewhat complex design, with both convex and concave surfaces. The team also identified that the final resolution and canvas size had to be very large. To achieve their objectives, they installed six 4K laser projectors from 21.000 lumens on walls and floor.

The real-time nature of this representation meant that the load would have to be split across multiple machines acting together as one., communicating over the network with each other. This led Recursive to use the nDisplay system of Unreal Engine, as it allows the environment to be rendered and load distributed across multiple workstations in a way that is quick to set up and easy to manage.

The team also made use of Vioso technology to scan the space and generate the necessary files, in the required formats. Besides, helps the Unreal system understand the layout and shape of nDisplay views.

“The projection surface that we had to work within the given space is deeply irregular. We needed a reliable and flexible projection mapping solution that would support complex projection mixing and be able to synchronize multiple workstations in real time.. We use multiple cameras to scan the space and then go through a conversion process to arrive at a single unified MPDCI output”, Marshall explains., audiovisual consultant.

MPCDI files were brought to nDisplay, and then used within your configuration to assign the outputs to the projectors that were connected to each node.

“In addition to the basic requirement of real-time rendering, It was also necessary for other physical elements of the facility to communicate”, adds Richard Smith, associate director of Recursive. “NDisplay enables this through Unreal planes and network-based protocols like OSC and UDP, allowing us to control dynamic lighting, Nano winches, Hazers and real-time audio systems”.

Vioso system offers equipment automated multi-camera and multi-projector calibration, allowing you to generate the files necessary for nDisplay configuration.

With Visoso software they were able to scan the space, align it with that of the 3D model and output the data set in an MPCDI v2 format which could then be used by nDisplay. The team achieved the required level of monitoring and control by developing its own internal system based on multiple depth cameras.. The system uses a point cloud approach to view the user from multiple angles by interpreting depth data in real time..

“We also needed to be able to use skeleton tracking to handle real-time graphical objects within the Unreal environment.”, says George Roboz, content production manager. “Usamos Microsoft Azure Kinect, that translates the user's skeletal movements into data via Touch Designer. This drives the nDisplay engine through Unreal's blueprint system.. Once the client enters the room, handles complex particle systems, deforms objects, Interacts with flora and fauna by reaching out and changing its position in physical space, while a magical story unfolds around them”.

Real-time interactions both virtually and physically make this space unique in both the user experience, as in the design and implementation challenges.

Due to the complexities of the installation and the need to ensure that the designs would work in reality, the team created a full-size mock-up space in the UK, where they could test and optimize the workflow and prepare for potential pitfalls.