By making these models available to residents and local businesses, cities can also gain feedback on their plans. Public users are also looking for tools that let them access the data information in a more modern and immersive way. Our solution is based on CIM (city information modeling) and GIS (geographic information system), which contain crucial information for city planners and public users. With CIM, users can display 3D city environments where data is overlaid on the top of it, acting as an informative tool for various city layers. Developing interactive solutions that can generate fully digital, virtual and mixed reality experiences is still a challenge, but now we are proposing this as an open-source solution.
The uses of VR and MR vary widely. Here, Tomasz Zawadzki, former / senior postdoctoral researcher at the National Centre for Geocomputation (NCG), Social Sciences Institute (MUSSI) at Maynooth University in Ireland, discusses the work he did in building a 3D VR and MR model of Dublin.
Describe the city system that’s part of this project.
Building City Dashboards (BCD) is a bigger project in which one part of it is a multiplatform solution for desktop, virtual reality and mixed reality (BCD Desktop/VR/MR). Here, I focus more on BCD VR/MR, which can be described by a new term, XR (extended reality) as an umbrella for virtual, augmented and mixed reality.
We decided to develop a mixed-reality instead of an augmented-reality experience because it does not overlay virtual objects only, but rather, anchors them to the real world, which is crucial in building interactive city systems. The project already has XR support and is City Information Modeling (CIM) oriented. It supports three platforms and five navigation modes including BCD Desktop – Street view/Bird’s eye view, BCD VR – Street view/Bird’s eye view and BCD MR Bird’s eye view only.
There is no BCD MR Street view version as the device being used (Microsoft HoloLens) would have had performance issues if we had tried to display a bigger scale model. Despite the fact that the city system contains the same Dublin 3D city model, the overall experience varies for each platform from the 3D flat screen, through the immersed virtual world into mixed reality (real view with holograms).
Each user interface (UI) had to be designed for a specific platform. That means for each platform, we used dedicated modules and components. The same set of initial features had to be rebuilt each time for new platforms. Different interaction techniques (mouse click, left- and right-hand controllers and gestures) distinguish one platform from another. The city system was designed to be one multiplatform project wherein the same tools and features are available for each platform.
How long have you been working on it?
The project was developed in three years (excluding preparation of the Dublin 3D model, which started before the BCD project) to evolve and reach the current state.
What purpose does it service – it, what can it be used for?
The Building City Dashboards project is focused on delivering tools and features for city planners, thereby engaging public and potential stakeholders in Dublin. The City System can facilitate the decision-making process by pre-planning techniques based on adding buildings to the city, simulating shadow and visibility, flood resilience, and exploring currently available data sets. The public can easily access the data via City System – statistical (employment rate, etc.) or in real time (mainly from the sensors located in the city).
Based on the data, people can make decisions where in the city is better to live (lower noise level/pollution), investors can decide about the areas for investment (property price rental data), the city council can plan the strategy in the city (land use zones), etc. BCD VR has a very useful tool for city planners called real-time procedural building generator. We can choose roof type (whether there is an attic or not), the initial foundation shape and the number of floors. This tool is very useful for city planners to simulate landscape in the city by adding new concepts and analyzing visibility and shadows with an estimation of how new buildings will influence the existing city landscape.
What MR-VR tools are used?
Unity game engine was used to build the project and facilitate multiplatform distribution. Additional packages were necessary for the virtual environment (HTC Vive Input Utility 1.10.7) and MR (Mixed Reality Toolkit – MRTK 2.3.0). They are dedicated assets, modules and libraries necessary to create XR experiences. In addition to that, the project had to be deployed to Microsoft HoloLens using Visual Studio 2019 Community, while the VR experience was built directly from the Unity editor (to run from EXE file). The VR package contains a VR camera and Canvas Raycast Target script, which could recognize standard UI elements – buttons, sliders, toggles – in VR. The MR package contains the MR camera and MR UI elements, which were completely different than the standard ones in Unity.
What MR-VR platform/headset is used?
For the BCD VR experience, the HTC Vive first generation was used. For the BCD MR experience, Microsoft HoloLens first gen was used. There is a second gen of the devices for VR/MR available that has better device performance.
What is the next step for the city system?
The next step in project development is to integrate BIM (building information modeling) in CIM (city information modeling). That means looking at the city not only from a farther perspective, but to be able to focus on details – specific buildings. Each building can represent many data sets attached to it. Another interesting approach can be to use AI to simulate traffic and crowds, which might be correlated with existing traffic conditions in Dublin city.
Tell me about the MR (holographic) model of Dublin.
BCD MR is a holographic representation of the 3D Dublin city model with UI built-in mixed reality on the top of it. The model was created in the form of an interactive table where all available tools and features were located behind the 3D model to easily access them. Initially, the table should be placed in any open space where is enough space for exploration. Interaction with the model base is through predefined gestures. Tap is representing a mouse click in a 3D environment, and this was used to toggle city elements (including vegetation, river, and roads) and data layers. Another way of interaction is using Tap-Hold-Drag in a sentence, which is corresponding in 3D to mouse click, hold and move. This interaction was used to change the sliders and handles values – move them up and down, simulate shadow by rotating the sun, increase flood level and moving predefined 3D buildings from the library into the table. To exit from application, you need to perform Bloom gesture. Another interesting fact is that physics was added to the buildings, and they can be moved in the real world and placed on a holographic table and will not fall as the table has colliders to keep them on the surface.
How did you create that model?
Generally, the process started from importing the 3D Dublin city model to an empty Unity scene project with all attached textures. In the next step, the model was enriched by shaders (reflective water texture, double side tree shaders). After that, flood resilience and shadow simulation with real sun movement on trajectory were added. Other components and tools were created from standard Unity UI elements: buttons, sliders, toggles.
Many times, the guides and advice on how to solve a specific problem were partially available on the Internet (game developer forums, etc.). Based on that, slowly new features were added to the project. When the desktop version was almost ready, there was a need to build the VR and MR experience. Initially, 3 VR libraries were tested: SteamVR, Vive Input Utility (VIU) and Newton VR. It was decided that the best at this stage will be to use VIU. After that, the project was rebuilt using new components for the VR platform. The same situation happened with MR, but there was one library to choose from only (MRTK).
Please outline how the model was generated.
The whole currently available 3D model of Dublin city was created by a D3D company based in Dublin (http://www.d3d.ie/index.html). For the BCD project, a small part of the city model was delivered as an open-source data (2 square km of a real area). The company is an owner of the whole model, and the model was built for Dublin City Council (city promotion purpose) in last years. The model is very high quality (15 cm precision) and was built from aerial photography using photogrammetry technique to 2D mesh representation (each building or tree was extruded based on heights). In the second stage, 3D objects were created from 2D mesh representation (wireframes were transformed into 3D objects) in 3D software (Autodesk 3ds Max). The last stage involved applying previously prepared (in Adobe Photoshop) textures taken directly in Dublin city.
What was the purpose of this model?
The main purpose to build this model was to prepare animation about Dublin city for the Dublin City Council. At this stage, BCD team members were not included. After engaging potential stakeholders D3D company (owner of the model), BCD team (research institution), Dublin City Council (public institution), it was agreed that the small part of the model might be distributed to work on for research purposes, and potentially in the future, the project can be used by city authorities.
How large is the model? Does it include the entire area of Dublin?
Currently, the model available in 3D/VR/MR is 2 square kilometers. We need to take into consideration performance issues especially for mixed-reality experiences. Ultimately, the model will be larger with two high-quality areas (one described here), while outside of it will be lower-quality city landscape. We don’t have access to the whole 3D Dublin city as it is owned by the D3D company.
What was the most challenging aspect of the model creation?
The most difficult part was to create the user interface, tools, features, simulations and data integration initially in 3D and transform it into a VR/MR environment. Each tool was created in a 3D environment in the Unity game engine, and this was a base for further development. Core parts from 3D environment were taken into VR/MR and included/adjusted into new components (for instance, a slider in 3D for flood resilience needed an extra script to be recognized in VR and had to be completely rebuilt in MR using slider dedicated to this platform).
Who worked on this with you?
The BCD 3D/VR/MR model was created by: me, who was responsible for UI, tools, features and simulations in 3D/VR/MR; as well as D3D team members, who provided the initial 3D Dublin city model without UI; Dr. Oliver Dawkins, BCD project member who developed interactive data layers; and professor Rob Kitchin, PI who was coordinating the project.
How long did the model generation take?
The whole Dublin 3D city model was built manually by D3D team members in the last five years. On top of that, there were three years of UI development for Desktop/VR/MR platforms.
What materials did you use as a reference for the Dublin model?
An open-source part of the model already contains textures, and there was no need to work on it. In the Unity game engine, the model was improved by adding photorealism – reflective water shader with physics, double-sided material for trees while the building textures remained untouched.
Acknowledgements: This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number 15/IA/3090.