This workshop will show you how to

Final result

To complete this workshop you will need:

Language used:

Additional resources:

This workshop will demonstrate how to set up a spatial computing environment using the Meta Quest 3 headset; it will guide you through interacting with virtual objects in the scene; instruct you on navigating the environment using a teleportation system; and teach you to develop a custom script for interacting with an MQTT broker.

URP Scene and the XR Integration Toolkit

Create a new Universal Render Pipeline (URP) Unity project using Unity Hub.

Create a new Unity Project

Meta Quest is based on Android, so we need to change the target platform from the build settings. From File -> Build Settings select Android and press Switch Platform.

Install the Unity OpenXR Meta package from the Window -> Package Manager. The package is not yet part of the Unity Repository, so it needs to be manually added

Oculus Integration installation

Install the XR Interaction Toolkit from the Package Manager window. This is part of the Unity Repository, so it can be search from the top right search field. If an update is prompted, press yes and continue.
Still in the Package Manager window, within the XR Interaction Toolkit select Samples and import both Starter Assets and Hands interaction Demo

Oculus Integration installation

Once the plug-in is installed and the samples imported, from Project Settings select XR Plug-in Management enable the OpenXR Plug-in Provider and select the Meta Quest feature group. Select the Project Validation to see if there is any issues to address, most of them can be solved automatically.

Enable VR support

In Project Settings -> Player provide the following information:

Minor changes need to be done to the quality settings as well. When the URP is used, some quality settings are bypassed by the Universal Render Pipeline Asset

URP Pipeline and settings

A virtual place VR

Create a New Scene and using the Empty template and add in the Hierarchy panel, using right click:

Create a new GameObject XR -> XR Origin (VR). This GameObject is the digital representation of the Meta Headset, and its controllers, in the virtual scene.
Select the XR Origin GameObject just created in the Hierrachy window and in the Inspector window, change the Tracking Origin Mode to Floor. This parameter is used to set the right height of the headset based on the Meta calibration of the Guardian.

With the XR Origin in the scene it is already possible to build a VR experience and walk around the environment, however, the controllers are not yet in place and properly linked to the input control system.

On the XR Controller component of each hand (XR Origin -> Camera Offset -> LeftHand Controller and XR Origin -> Camera Offset -> RightHand Controller), there is a field Model -> Model Prefab that is used to control the mesh of the VR controllers.
The Meta Quest controllers, as the controllers of other VR HMD, are not provided with the XR Interaction package but they can be downloaded and added manually. To add the Meta Quest controllers:

The Meta Quest 2 needs a different approach. In the Meta Quest 2 Touch, the model to use is quest2_controllers_div2. Both models are in the same prefab.

Add 3D models of the controllers

A virtual place AR (Optional)

Create a New Scene and using the Empty template and add in the Hierarchy panel, using right click:

Create a new GameObject XR -> XR Origin (AR). This GameObject is the digital representation of the Meta Headset, and its controllers, in the virtual scene.
Change the Tracking Origin Mode to Floor. This parameter is used to set the right height of the headset based on the Meta calibration of the Guardian.
Select the Main Camera inside XR Origin -> Camera Offset and in the Inspector window control that the Environment is set to Solid Color and that the Alpha channel of the colour is set to 0.

Camera Environment Solid Colour Alpha

Add to the Hierarchy window a GameObject AR Session with Tracking Mode set to Position And Rotation.

With the XR Origin in the scene it is already possible to build a Passthrough AR experience and walk around the environment, however, the controllers are not yet in place and properly linked to the input control system.

On the XR Controller component of each hand (XR Origin -> Camera Offset -> LeftHand Controller and XR Origin -> Camera Offset -> RightHand Controller), there is a field Model -> Model Prefab that is used to control the mesh of the VR controllers.
The Meta Quest controllers, as the controllers of other VR HMD, are not provided with the XR Interaction package but they can be downloaded and added manually. To add the Meta Quest controllers:

The Meta Quest 2 needs a different approach. In the Meta Quest 2 Touch, the model to use is quest2_controllers_div2. Both models are in the same prefab.

Add 3D models of the controllers

Set the actions

The controllers are in place, but they need to be linked to the interaction system. We are going to use the Samples imported from the XR Interaction Toolkit:

Enable the Actions

Set the filters for the actions

Input Action Manager

Position and Rotation Actions

Test the scene

From File -> Build Settings and click on Add Open Scenes. With the Meta Quest connected to your computer, press Build and Run, select the local destination folder for the APK.

Test the VR scene

The APK is installed on the Meta Quest. You should be able to see the Quest Controllers tracked with a red line coming out from them. In the next part we will see how to use the controller to interact with the object in the scene.

Simple models can be imported directly in Unity. Original file formats such as .fbx, .dae and .obj for 3D models, .jpg and .png for images, .wav and .mp3 for sounds etc. are natively supported by Unity. Other formats can be imported using additional plugins from the Assets Store or from the Unity community.

A must-have package is glTFast to import, and export, GLTF file format. To install the package we need to use the Package Manager:

Sketchfab is a great service to share and download 3D models. Choose one model to add to the gallery and download the fbx format. For example, the Noise Gauge

Download model screen Sketchfab

Create a new folder in the Project window and copy the model in there.
From the Inspector window we can change the importing settings of the assets (e.g. scale factor, animation, uv mapping). Select the Materials tab and change the Location field to Use External Materials (Legacy), in this way the materials will be extracted from the model in the same folder of the asset as Unity Materials, easier to modify them.

Material Legacy

In this example the 3D model did not have embedded texture and they have been added from the Inpector window (the colour map on the small square next to the Albedo, and the Normal map in the small square next to the Normal Map)

Texture apply

We can now drag the 3d model in the scene, change is position using the Transform component in the inspector window or directly the gizmo axis in the Scene window

Texture apply

More complex models, using 3DTiles Format can be imported using service such as Cesium.

A Free Cesium account is needed to load assets directly from Cesium. The account is not needed to load local 3DTiles

Test the VR scene

In the Hierarchy window, create a new Empty object, name it CesiumGeoreference and attach a Cesium Georeference component.
From the Cesium menu, add a Blank 3D Tiles Tileset. It will create a new object under the Cesium Georeference just created.

In the Inspector window, we can opt to load the tileset from Cesium ION (a Cesium account is required) using the ID number of the asset. In this case, Unity will request a Key that can be created automatically from Unity.
It is possible to load a tileset from a URL. Unfortunately, at the moment, a local tileset would only work in the editor mode as the URL requires an absolute path. Alternatively, it is possible to load the tileset from a remote location, such as a bucket.

Test the VR scene

Tilesets require a geolocation that is set in the CesiumGeoreference GameObject. It is possible to force a Cartesian location and place the origin at the location of the camera by using the Place Origin Here button.

Test the VR scene

To interact with the virtual environment, we need to attach the interactable components of the XR Interaction Toolkit to the digital objects and to the controllers themselves. Both controllers of the XR Origin prefab have already a XR Ray Interactor component attached (the red line). We are going to transform the Right controller in a direct interaction control to grab the objects in the scene.

Grabbable object

Grabbable object

Create two new GameObjects cubes in the scene with the following parameters and name:

Create a different materials for each of them.

Table and Podium

Build the current scene to test the environment. It is now possible to interact with the red cube: with the right hand by touching it and pressing the grip button on the side of the controller; with the left controller by pointing at the cube and pressing and holding the grip button.

Create a new GameObject XR -> Locomotion System (Action-based) and add a new GameObject to the scene XR -> Teleportation Area in position (0,0,0). To avoiding the overlapping, we can now delete, or disable, the Plane create at the beginning of the workshop.

Create a new 3D Object -> Cylinder named customReticle with scale (0.5 , 0.01 , 0.5), position (0 , 0 , 0). Remove the component capsule collider and change the material with a green coloured one. Create a Prefab with this GameObject and delete it from the scene

The teleportation system is already working, however, the default straight line render and the key behaviour are not the most effective solutions. Moreover, we want to use both hands to teleport and interact with the objects.

Teleporting default

We are going to create two new parent GameObjects for both hands and, for each of them, we are going to add a grab controller and a teleport controller:

Teleport actions

The XR Origin should have this structure now:

Teleport actions

Create a new C# script TeleportationManager :

using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.Events;

public class TeleportationManager : MonoBehaviour
{
    public GameObject baseController; //the Right / Left Hand Controller GameObject
    public GameObject teleportationController; //the Right / Left Teleport Controller GameObject
    public InputActionReference teleportActivationReference; //The `XRI RightHand/Teleport Mode Activate` / `XRI LeftHand/Teleport Mode Activate`

    public UnityEvent onTeleportationActive;
    public UnityEvent onTeleportationCancel;

    private void Start()
    {
        teleportActivationReference.action.performed += TeleportModeActivate;
        teleportActivationReference.action.canceled += TeleportModeCancel;
    }

    private void TeleportModeCancel(InputAction.CallbackContext context) => Invoke("DeactivateTeleporter", 0.1f);

    void DeactivateTeleporter() => onTeleportationCancel.Invoke();

    private void TeleportModeActivate(InputAction.CallbackContext context) => onTeleportationActive.Invoke();

}

Attach the script to the RightHand Parent GameObject (it will be attached also to the LeftHand Parent GameObject but without the event for the XRDirectInteractor) with the following values for the public fields:

To switch between the main controller and the teleportation one, add four entries for each OnTeleportationActive and OnTeleportationCancel

On the OnTeleportationActive add:

On the OnTeleportationCancel the values will be inversed:

Teleport Manager

In the XR Ray Interactor of the RightHand Teleporter change the Line Type to Bezier Curve.

In the application, moving the controller stick forward activates the teleportation line and the customReticle. To move the camera to the location of the customReticle, simply release the stick to its rest position.

Screenshot Oculus Final teleport

Using the XR Interaction Toolkit it is possible to provide custom functions to interact with the scene. In this last part of the XR workshop we will link a virtual object with a MQTT broker to publishing and receiving messages when the red cube is place to the podium.

Download the M2MQTT library and copy the folders M2Mqtt and M2MqttUnity in the Assets folder of the Unity project.

Create a new C# script mqttManager:

/*
The MIT License (MIT)

Copyright (c) 2018 Giovanni Paolo Vigano'
****
Modified by Valerio Signorelli, UCL Connected Environments 2021
Subscribed to list of topics

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
****
Modified by Valerio Signorelli, UCL Connected Environments 2021
# Add Subscribed to list of topics

*/
using System.Collections.Generic;
using UnityEngine;
using M2MqttUnity;
using uPLibrary.Networking.M2Mqtt.Messages;
//the mqttObj is use to store the message received and topic subscribed, so we can select the right object from the controller
//C# Property GET/SET and event listener is used to reduce Update overhead in the controlled objects
public class mqttObj{
    private string m_msg;
    public string msg
    {
        get{return m_msg;}
        set{
            if (m_msg == value) return;
            m_msg = value;
        }
    }
    private string m_topic;
    public string topic
    {
        get
        {
            return m_topic;
        }
        set
        {
            if (m_topic == value) return;
            m_topic = value;
        }
    }
}

public class mqttManager : M2MqttUnityClient
{
    [Header("MQTT topics")]
    [Tooltip("Set the topic to subscribe. !!!ATTENTION!!! multi-level wildcard # subscribes to all topics")]
    //public string topicSubscribe = "#"; // topic to subscribe. !!! The multi-level wildcard # is used to subscribe to all the topics. Attention i if #, subscribe to all topics. Attention if MQTT is on data plan
    public List<string> topicSubscribe = new List<string>(); //list of topics to subscribe

    [Tooltip("Set the topic to publish (optional)")]
    public string topicPublish = ""; // topic to publish

    public string messagePublish = ""; // message to publish

    [Tooltip("Set this to true to perform a testing cycle automatically on startup")]
    public bool autoTest = false;

     mqttObj mqttObject = new mqttObj();

    public event OnMessageArrivedDelegate OnMessageArrived;
    public delegate void OnMessageArrivedDelegate(mqttObj mqttObject);

    //using C# Property GET/SET and event listener to expose the connection status
    private bool m_isConnected;


    public bool isConnected
    {
        get
        {
            return m_isConnected;
        }
        set
        {
            if (m_isConnected == value) return;
            m_isConnected = value;
            if (OnConnectionSucceeded != null)
            {
                OnConnectionSucceeded(isConnected);
            }
        }
    }
    public event OnConnectionSucceededDelegate OnConnectionSucceeded;
    public delegate void OnConnectionSucceededDelegate(bool isConnected);

     // a list to store the mqttObj received
    private List<mqttObj> eventMessages = new List<mqttObj>();

    public void Publish()
    {
        client.Publish(topicPublish, System.Text.Encoding.UTF8.GetBytes(messagePublish), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
        Debug.Log("Test message published");
    }
    public void SetEncrypted(bool isEncrypted)
    {
        this.isEncrypted = isEncrypted;
    }

    protected override void OnConnecting()
    {
        base.OnConnecting();
    }

    protected override void OnConnected()
    {
        base.OnConnected();
        isConnected = true;

        if (autoTest)
        {
            Publish();
        }
    }

    protected override void OnConnectionFailed(string errorMessage)
    {
        Debug.Log("CONNECTION FAILED! " + errorMessage);
    }

    protected override void OnDisconnected()
    {
        Debug.Log("Disconnected.");
        isConnected = false;
    }

    protected override void OnConnectionLost()
    {
        Debug.Log("CONNECTION LOST!");
    }

    protected override void SubscribeTopics()
    {
        foreach (string item in topicSubscribe) //subscribe to all the topics of the Public List topicSubscribe, not most efficient way (e.g. JSON object works better), but it might be useful in certain circumstances 
        {
         client.Subscribe(new string[] { item }, new byte[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });   
        }
        
    }

    protected override void UnsubscribeTopics()
    {
        foreach (string item in topicSubscribe)
        {
            client.Unsubscribe(new string[] { item });
        }
    }

    protected override void Start()
    {
        base.Start();
    }

    protected override void DecodeMessage(string topicReceived, byte[] message)
    {
        //The message is decoded and stored into the mqttObj (defined at the lines 40-63)
        
        mqttObject.msg = System.Text.Encoding.UTF8.GetString(message);
        mqttObject.topic=topicReceived;

        Debug.Log("Received: " + mqttObject.msg + "from topic: " + mqttObject.topic);

        StoreMessage(mqttObject);
        
        if(OnMessageArrived !=null){
        OnMessageArrived(mqttObject);
        }
    }
    private void StoreMessage(mqttObj eventMsg)
    {
        if (eventMessages.Count > 50)
        {
            eventMessages.Clear();
        }
        eventMessages.Add(eventMsg);
    }

    protected override void Update()
    {
        base.Update(); // call ProcessMqttEvents()

    }

    private void OnDestroy()
    {
        Disconnect();
    }

    private void OnValidate()
    {
        if (autoTest)
        {
            autoConnect = true;
        }
    }
}

Create a new C# script mqttPublisher

using System;
using System.Data;
using Unity.Mathematics;
using UnityEngine;
using UnityEngine.InputSystem;
public class mqttPublisher : MonoBehaviour
{
    public mqttManager _eventSender;
    private string messagePublish = "";

    private int row;
    private int column;
    private int[] colour;
    bool _connected;

    public InputActionReference bButtonActionReference;

    private void OnEnable()
    {
        bButtonActionReference.action.Enable();
        bButtonActionReference.action.performed += HandleBButtonPress;
    }

    private void OnDisable()
    {
        bButtonActionReference.action.Disable();
        bButtonActionReference.action.performed -= HandleBButtonPress;
    }

    private void HandleBButtonPress(InputAction.CallbackContext context)
    {
        triggerPublish();
        // Logic for when the B button is pressed
        Debug.Log("B button pressed!");
    }

    void Start()
    {
        _eventSender = this.gameObject.GetComponent<mqttManager>();

        _eventSender.OnConnectionSucceeded += OnConnectionSucceededHandler;
    }

    private void OnConnectionSucceededHandler(bool connected)
    {
        _connected = true;

        row = UnityEngine.Random.Range(0, 8);
        column = UnityEngine.Random.Range(0, 8);
        colour = new int[3] {
            UnityEngine.Random.Range(0, 256), // Range: 0-255
            UnityEngine.Random.Range(0, 256), // Range: 0-255
            UnityEngine.Random.Range(0, 256)  // Range: 0-255
        };

        //messagePublish = $"[{row},{column},'rgb({colour[0]},{colour[1]},{colour[2]})]";
        messagePublish = "{\"row\":" + row + ",\"column\":" + column + ",\"colour\":\"" + colour[0] + "," + colour[1] + "," + colour[2] + "\"}";


        if (!connected) //publish if connected
            return;
        //if the messagePublish is null, use the one of the MQTTReceiver
        if (messagePublish.Length > 0)
        { _eventSender.messagePublish = messagePublish; }

        _eventSender.Publish();
        Debug.Log("Publish" + messagePublish);
    }

    public void triggerPublish()
    {
        row = UnityEngine.Random.Range(0, 8);
        column = UnityEngine.Random.Range(0, 8);
        colour = new int[3] {
            UnityEngine.Random.Range(0, 256), // Range: 0-255
            UnityEngine.Random.Range(0, 256), // Range: 0-255
            UnityEngine.Random.Range(0, 256)  // Range: 0-255
        };

        //messagePublish = $"[{row},{column},'rgb({colour[0]},{colour[1]},{colour[2]})]";
        messagePublish = "{\"row\":" + row + ",\"column\":" + column + ",\"colour\":\"" + colour[0] + "," + colour[1] + "," + colour[2] + "\"}";


        if (!_connected) //publish if connected
            return;
        //if the messagePublish is null, use the one of the MQTTReceiver
        if (messagePublish.Length > 0)
        { _eventSender.messagePublish = messagePublish; }

        _eventSender.Publish();
        Debug.Log("Publish" + messagePublish);

    }
}

Create a new C# script mqttController:

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class mqttController : MonoBehaviour
{
    public string nameController = "Controller 1";
    public string tag_mqttManager = "";
    public mqttManager _eventSender;

    public GameObject spawnObj;

    void Start()
    {
        if (GameObject.FindGameObjectsWithTag(tag_mqttManager).Length > 0)
        {
            _eventSender = GameObject.FindGameObjectsWithTag(tag_mqttManager)[0].gameObject.GetComponent<mqttManager>();
        }
        else
        {
            Debug.LogError("At least one GameObject with mqttManager component and Tag == tag_mqttManager needs to be provided");
        }
        _eventSender.OnMessageArrived += OnMessageArrivedHandler;
    }

    private void OnMessageArrivedHandler(mqttObj mqttObject) //the mqttObj is defined in the mqttManager.cs
    {
        //We need to check the topic of the message to know where to use it 
        if (mqttObject.topic.Contains("live_counts"))
        {
            var response = JsonUtility.FromJson<Root>(mqttObject.msg);
            Vector3 position = new Vector3(0, 3, 0); // Replace with your desired position
            GameObject spawnedObject = Instantiate(spawnObj, position, spawnObj.transform.rotation) as GameObject;
            Debug.Log("Message, from Topic " + mqttObject.topic + " is = " + mqttObject.msg);
        }
    }
}

[Serializable]
public class Count
{
    public int count;
    public int id;
    public string name;
}
[Serializable]
public class Root
{
    public List<Count> counts;
    public int index;
    public DateTime time;
}

Attach the three scripts to the Red Cube GameObject:

As shown in the mqttPublisher, the publishing event is controlled by a button press that still needs to be configured:

Select the Podium GameObject from the Hierarchy and add a XRSocketInteractor component. This component will link the Red Cube GameObject and trigger the MQTT connection to the Broker:

Finally, set the Box Collider of the Podium GameObject to IsTrigger and add a RigidBody set to Kinematic and freeze the X, Y, and Z of both position and rotation.

Enable VR support

Build and Run the project. It will now be possible to grab the RedCube, teleport to the Podium, and trigger the MQTT connection by placing the RedCube near the Podium.
By pressing the B button on the controller, a message will be published to the MQTT broker.

Build and Run the project, it will be now possible to grab the RedCube, teleport to the Podium and trigger the MQTT connection by placing the RedCube near the Podium.
By pressing the button B on the controller, a message will be published on the MQTT broker.

Enable VR support

It is possible to use any other digital models instead of the two cubes, for example an actual Bell and an Low Poly Hammer.

Enable VR support

Select the Hammer GameObject from the Hierarchy Window and in the Inspector Window add the XR Grab Interactable component with Movement Type set to Kinematic. To avoid to grab the hammer in its gravity centre, add to it a child Empty GameObject named attachPosition and move it nearby the end of the handle and drag the attachPosition GameObject on the empty field _Attach Transform of the XR Grab Interactable component. On the RigidBody component, enable Is Kinematic and disable Use Gravity

Inside the bell GameObject select the Bell and add a Box Collider component, make it bigger as the bell (or slightly smaller, e.g. Center (0 , 0.005 , 0); Size (0.007 , 0.004 , 0.007)) and set Is Trigger enable. Add a RigidBodyKinematic without Gravity and freeze X, Y and Z for both position and rotation and change Collision Detection to Continuos Speculative.

Create a new C# Script named simpleCollision

using UnityEngine;
using System.Collections;

public class simpleCollision : MonoBehaviour
{
    public static simpleCollision current;

    private void Awake()
    {
        current = this;
    }


    private void OnTriggerEnter(Collider other)
    {
        if (other.name == "hammer")
        {
            this.GetComponent<mqttReceiver>().Connect();
        }
    }

    private void OnTriggerExit(Collider other)
    {
        if (other.name == "hammer")
        {
            this.GetComponent<mqttReceiver>().Disconnect();
        }
    }
    }

Control that the other.name as the exact name of the GameObject that will collide with the bell (in this case hammer), and add the script to the Bell GameObject.

Finally, add mqttReceiver, mqttController and mqttPublisher (the same used on the RedCube GameObject will work).

Enable VR support