Twin cities: harnessing gaming technology for immersive digital twins

Digital twins are a game-changer for Buro Happold, literally. The consultant’s geoscience and digital twins team is using computer-game engine software to enable it to create immersive, collaborative and interactive 3D digital replicas of physical entities, such as cities.

‘A big part of using a gaming engine is enabling user immersion, because interaction and experience are a big part of a digital twin,’ says Miraj Patel, associate graphical information systems (GIS) and digital twin developer at Buro Happold. Another major benefit, according to Patel, is the almost photo-realistic rendering capability that game engines bring to enhance the immersive experience. ‘Depending on how you build it, with a VR [virtual reality] environment users can jump into it and almost feel it,’ he says. 

The innovative use of game engines to enable the creation of immersive digital twins was one of the reasons Buro Happold was crowned Digital Champion at the CIBSE Society of Digital Engineering Awards in December. Game engines enable Buro Happold’s digital twin models of cities, for example, to incorporate vast datasets of geographical and environmental characteristics, to create highly accurate and comprehensive virtual replicas of cities. 

It is the ability to combine information within the digital twins, and use it to run simulations, that reduces decision-making time and enhances collaboration, says Patel. It also enables those using the model, such as urban planners, to make more informed decisions. 

With this enhanced level of detail, municipal bodies are able to use the digital twin to monitor infrastructure performance, or run scenarios to assess the impact of a road closure, say, or flooding. Environmental organisations can assess the impact of developments and their alignment with sustainability goals, while engineering firms can adapt and generate infrastructure designs more effectively.

Levelling up

Buro Happold’s digital twin journey started five years ago, when the team set out to ensure all the data it had and was collecting was saved in a format that was usable universally. To enable all of its various data sources to be used, the engineer put in place a framework and protocols to ensure its data management systems were sound. Patel describes these as ‘technology/software agnostic’ robust data standards.

This data management provides the foundation for Buro Happold to use its data in the development of digital twins for a wide variety of projects, scenarios and applications.

Patel says engineers generally come to the digital team with a problem statement/challenge that needs a digital solution. ‘We call it computational engineering; the idea is that a lot of the work we do can be aided by computation to improve our performance, by enabling us to challenge a problem in a lot more detail,’ he says. ‘The idea is to create a digital replica of a complex environment, a building or a system that allows you to challenge it virtually, to change it easily, and even to break it.’

Some of the areas where computational engineering solutions have been implemented include: masterplanning; facilities/asset management; design review and integration; construction management and progress tracking; scenario simulation; community engagement; building fabric analysis; and automated district energy network design and optimisation

When Buro Happold developed the community heat digital twin for Barcombe (see ‘Heating made to measure’, CIBSE Journal, January 2022), one of the big benefits of having a digital model of the village was that it gave the engineers and the community a comprehensive image of what was happening. For the engineer, it was a way to test various heating scenarios without impacting homes. It also helped enhance community involvement. 

‘We were able to use the digital twin to show “what if” scenarios within a virtual environment,’ says Patel. ‘We could say, “here’s your house, and you can see the predicated energy use and carbon output”.’

Initially, Buro Happold digital twins, such as the one for Barcombe, were developed as 2D and 3D web-based models, using open source QGIS and ArcGIS software to enable geospatial data to be visualised. The digital team works backwards from the outputs the model needs to deliver, and through the engineering steps, to establish the data sets and software required to build the model. ‘You work backwards to define your data requirements, then you have to see where you can source that data to start building up the model,’ Patel explains.

A major constraint with these web-based digital twin models, however, is that they are limited graphically and in the level of interactivity. ‘The web is great when you have a 2D to 3D platform for letting you see 2D and 3D shapes, but it’s not immersive, it’s not interactive, and the renderings are not realistic – so users cannot get the full experience of what they are seeing and what they are building,’ Patel says.

With the digital sector continuing to evolve rapidly, Buro Happold’s digital team talked to Epic Games (the maker of Fortnite) about using its Unreal Engine software to create real-time 3D content for its digital twins. ‘It would allow users to jump into the model and be in a place virtually, which would make it a really useful communication tool for clients and the different engineering disciplines on a project,’ says Patel.

The benefit of using gaming software is that it allows the model to be adapted in real time. ‘We didn’t want to do what everyone else does – preload data into the model, which makes it a huge application. We wanted the data to flow from our database and we had already built the mechanism to enable that,’ says Patel.

‘This gives you a whole new level of useability. It makes the digital model useful, not just for the design or construction stage, but also for asset management, because you can bring in different live data sources by connecting in IoT [internet of things] devices.’ This methodology allows for live-operation digital twins and dashboards to manage a city, for example.

Tomorrow’s epic cities

Epic Games could see the potential in Buro Happold’s proposals and awarded it an Epic MegaGrant. This supported the digital team while it spent time navigating the technical challenges of using a gaming engine to stream geospatial data into an application in real time, then render it and make it interactive. 

‘The MegaGrant allowed us to build up the functionality and to develop new texturing techniques within Unreal to make the buildings look realistic,’ explains Patel. ‘Every time we had a conversation with Epic, we showed them what we were trying to do and what we’d done. They were very interested to push it further; some of the stuff we showed them they had not seen before.’

Buro Happold’s gaming engine-driven digital twin development has progressed to the stage where it is now capable of building advanced interactive digital models for numerous applications. 

‘There are so many possibilities. When a user gets into the VR digital twin, they are able to interact with the data, so they can move buildings around, delete buildings and draw new ones; they can add comments for review and even tag people,’ Patel enthuses. ‘What’s more, everything is coming from a single source of truth: all the data is in one place and all the engineers work from it and update it, so the model is always live and always up to date.’

The London city model data shown on these pages comes from Photorealistic 3D Tiles from Google Maps Platform through Cesium. The level of detail varies based on the source data used to build up the digital twins. BIM, CAD or any other data can be integrated into the digital twin. Buro Happold has created digital twins using the Unreal Engine but currently has non-disclosure agreements with these clients so cannot yet share any images

The generalised methodology Buro Happold has developed allows it to generate digital twins for cities worldwide. Patel uses a masterplanning project to redevelop a city (he is unable to say which one) to explain how these models are being used. 

‘Before the client set about redeveloping the city, they wanted a digital twin of how it exists at this point in time, including assets such as utilities, buildings, trees – even lampposts – so they could overlay the redevelopment masterplan to analyse its impact,’ he says. 

To create the model, Buro Happold takes GIS, planning and budget data for various scenarios, and processes it according to its protocols, to ensure it can be integrated into the Unreal platform and then develop the various functionalities the users need. 

‘A big part of this was being able to take the existing city model and look at where the new buildings would be going, what the scheduling might look like, what roads would be affected, and things like that,’ says Patel. 

‘Using the digital model, you can see what the impact on the surrounding environment might be of closing a road, to help judge what the best option would be, rather than actually closing a road and then having to deal with what happens!’ 

It is not just towns and cities that can be modelled; the Unreal Engine provides the model with enhanced levels of granularity. ‘When you look at a development in a web-based digital twin, you cannot go inside the buildings. Using Unreal enables users to look at the development from outside, with a third-person view, but they could also go inside the buildings,’ Patel says. 

‘In theory, people could walk inside to see what the design would look like, then do a flow analysis to see how people move about, which you could visualise. You could even do a façade analysis to look at the impact of reflections on the space, for example.’ 

Each interactive digital twin is developed as a bespoke model. ‘There are core components, which make it easier to use again and again, but we don’t subscribe to a one size fits all,’ Patel says. 

Development of the digital twin starts with what Patel describes as ‘design thinking’ sessions and user interface (UI) and user experience (UX) workshops, to identify individual user needs and define the model’s functionality. Each user has an account that defines their level of access. ‘We construct personas, story maps and wireframes to design and assess UI/UX, because no-one wants to click through seven pages to get to the information they need,’ he says. 

The generalised methodology Buro Happold has developed means it is not tethered to a single technology, so it can easily develop interfaces and systems that cater to the unique requirements of each project. This adaptability ensures that the digital twin remains scalable and future-proofed, capable of evolving with technology and the data. 

So, what of the future? Patel has just taken delivery of VR glasses and AI-powered smart glasses to help the team develop Buro Happold’s next generation of immersive digital twins.