A Designer’s Guide for 3D Content in Mixed Reality
Mixed Reality is the newest trend right now. What brings value is having your custom Holograms in the experience. Why does learning this matters? It is similar as creating a picture in Photoshop and then you don’t know how to put it in your app or share it.
How I landed in 3D
I came to Microsoft to work in designing a music streaming application. It was so fun, but as it often happens in corporations, a re-org happened. I was placed in a group where they worked with a device called HoloLens. I was nervous! I had no idea how to design for it. I had not even seen that device before nor knew exactly what was the difference between AR and VR.
So, I had to tackle this the best way I could. Everybody was pretty friendly in the new team. I asked fellow designers on my new team.
What is Mixed Reality?
The application we were designing was called Microsoft Dynamics 365 Layout.
It helps you visualize your space using the HoloLens and with your own 3D models. Viewing 3D in a 2D screen is a different experience. One of the advantages of HoloLens applications is you can see 3D in context.
You get a sense of the scale of the real world and what these objects would really look like — with out the effort and time of moving physical objects around in space.
How many of us have had that moment with the sofa, or book shelf trying to find the “right spot” in your apartment? This app could help you visualize how it can look, and most importantly, if it will fit.
Where to start
Since I was scared to start designing for the 3D experience, I volunteered to design the PC app where the users would upload their 3D models into our HoloLens app.
I did not know anything about 3D models, but I imagined it shouldn’t be that hard. I started to be on customer calls and understand how their 3D content workflow and what was needed for our application.
Oh boy.. I did not know I had chosen one of the most technical difficult problem to solve: how to bring your custom 3d models.
Why are 3D models important
Understanding 3D content
Think about how in 2D the output of your work in Photoshop is a 2D image, the output of a CAD program is a 3D model in the 3D world.
Get your 3D model that works as a Hologram.
You have to follow these steps:
What tools create 3D models?
You must know which program you are using. Depending on the program, you will be using parametric geometry vs. polygonal geometry. The 3D standards use real-time rendering applications that require models to be represented as a triangulated polygon surface. In human words for it to become a Hologram it has to be made as a Vector using polygonal geometry. Some CAD programs that does that are: Autodesk Revit, AutoCAD, Sketchup, Catia, Creo, Autodesk Navisworks, Solidworks, Autodesk Inventor, Vectary, Cinema 4D, Maya
Vector:
All 3D models are made of vectors that get closed and extruded, similar to 2D vectors, where you go to Illustrator or Figma and draw lines to create your 2D object.
Rendering = display time
Rendering is the amount of time it will take to bring your model to the screen.
In technical terms: “3D models are created with complex materials and textures and then lit and animated in virtual simulations of real-world physical properties. Rendering is the process of calculating the results of these simulations and turning it into the final image on screen. The power of real-time rendering really shines in interactive experiences where the action on screen in not predetermined. Rendering complex scenes at this pace on today’s hardware requires some trade-offs in physical accuracy compared to an offline render where speed is less of a constraint.”
Select your platform
When you select the platform you will use, you will have to understand the platform limits, and therefore, the models resolution for the rendering.
A smartphone requires medium resolution:
1–3 objects in a scene, <500,000 triangles, 1–2 Materials per object
The HoloLens requires low resolution:
1–3 objects per scene, <1000,000 triangles, 1–2 materials per object
An Immersive Headset can have high resolution :
1–3 objects per scene, <15,000,000 triangles, 1–2 materials per objects
What are the building blocks of a 3D model?
Think of how a Layer is used in Photoshop, same as polycount in 3D. A polycount is the geometry of a 3D model, the skeleton.
If you want to go deeper, such as how pixels are for 2D, triangles are the pixels of 3D. This involves computing some of the math and generate a smaller version of polygons that are triangles, so like pixels when you need to compress an image. Triangles are the smallest version of polygons.
The more you have the smoother the shape looks, but it makes it more complex.
What is optimize?
You will have to optimize or reduce the amount of “triangles” aka pixels of the model. It is like in the old days “Saving for web” , you need a different resolution for this context. Similarly, it is as saving a jpg and reducing the size of the image while maintain most of the detail.
Optimization criteria
These affect the time it takes to display the 3D model:
The triangle count or pixels in 2D:
The more triangles the more resolution, but the slower the performance.
Number of individual objects in the model :
A table with 4 individual legs and a table top . This would make it 5 objects grouped as one. Low complexity scene is 1–3 objects to medium to high complex
Number of unique materials :
This add up the number of draw calls it takes for the GPU to represent to 3d object as it draws it on other screen from to back (closest to furthest)
Texture maps:
the resolution can eat up a lot of the memory and slow down the GPU as it passes information back to the CPU , computer processing unit, and frame rate can drop. Consolidating multiple textures into a single texture reduces draw calls from 22 to 1 in this example.
Convert
With Mixed Reality becoming more common, as designers, we need to understand the file formats and how to make it easier for users to create their own content.
Now that your 3D content is complete, you will need to export it in a supported 3D file format : glb.
Just like the difference in a jpg and png and like one of the standards is a png in the 2D web world. One of the standards in the 3D industry is a glb, which mean that is binary , so you can put a bunch of different data into that file format. Imagine: like a zip file that only shows you the finished png with all the layers on it.
How do you do that?
Using other 3D software tools such as there: blender, 3dsmax, vectary, maya , cinema 4d
Upload your model
So now, you have your 3D model optimized and converted. YAY! Those were the hardest steps. Now just uploading your 3D model wither by usb or cloud.
The tool I designed, Microsoft Dynamics 365 Import Tool converts your 3D model into a GLB and puts it on your 3D library in your PC. You can plug in your HoloLens and it will send that 3D library into the local folder in HoloLens. You can also use your cloud products. Place it in your cloud and open your cloud in HoloLens.
Our customers have been able to convert their models and transfer their models.
Now you can open the Mixed Reality app Layout and see your 3D model inside the HoloLens.
With the team we created documentation of how to convert your 3D models, this is still not the most optimal route, but it is the best one we could come up with for now. Here: https://docs.microsoft.com/en-us/dynamics365/mixed-reality/import-tool/optimize-models
3D models are more complex than they seem, but this should not scare you.
With this guide it can help you start 3D content creation.
And remember
Hopefully these analogies help you:
There are still a few limitations on every step. You will be finding challenges, or see why your 3D model takes too much to upload or why it is not rendered beautifully.
This is such a new topic. The industry is still trying to define processes and standards. We are all in this together.
