This workshop will show you how to

Final result

To complete this workshop you will need:

Language used:

Additional resources:

This workshop will show you how to setup a virtual reality environment using the Unity XR interaction toolkit; how to interact with the virtual objects in the scene; how to move around the environment using a teleportation system; how to develop a custom script to interact with a MQTT Broker.

URP Scene and the XR Integration Toolkit

Create a new Universal Render Pipeline (URP) Unity project using UnityHub.

Create a new Unity Project

As the Oculus Quest 2 is based on Android, we need to change the target platform from the build settings. From File -> Build Settings select Android and press Switch Platform.

Install the XR Interaction Toolkit 1.0.0-pre.8 from the Package Manager window. Updated to the new input system if prompted, Unity will restart.
Import the Default Input Actions from the Sample menu. If prompted, enable the new input system.

Oculus Integration installation

Once the plug-in is installed and the samples imported, from Project Settings select XR Plug-in Management, install the plug-in if it is not, and enable the Oculus Plug-in Provider.

Enable VR support

In Project Settings -> Player provide the following information:

Minor changes need to be done to the quality settings as well. When the URP is used, some quality settings are bypassed by a Universal Render Pipeline Asset. It is possible to create a new Universal Render Pipeline Asset, or modify the UniversalRP-MediumQuality (the default used by Android).

UPR Settings

Select the asset and, in the inspector window, change the Anti Aliasing MSAA from Disable to 4x. From this window is possible to improve the quality of the shadows by changing the Shadows Max Distance to a lower value.

A virtual place

Create a New Scene using the Empty template and add:

Create a new GameObject XR -> XR Rig(Action-Based). This GameObject is the digital representation of the Oculus Headset, and its controllers, in the virtual scene.
Change the Tracking Origin Mode to Floor. This parameter is used to set the right height of the headset based on the Oculus calibration of the Guardian.

With the XR Rig in the scene it is already possible to build a VR experience and walk around the environment, however, the controllers are not yet in place and properly linked to the input control system.

On the XR Controller component of each hand (XR Origin -> Camera Offset -> LeftHand Controller and XR Origin -> Camera Offset -> RightHand Controller), there is a field Model -> Model Prefab that is used to control the mesh of the VR controllers.
The Oculus Quest2 controllers, as the controllers of other VR HMD, are not provided with the XR Interaction package but they can be downloaded and added manually. To add the Oculus Quest2 controllers:

Add 3D models of the controllers

Set the actions

The controllers are in place, but they need to be linked to the interaction system. We are going to use the Samples imported from the XR Interaction Toolkit:

Enable the Actions

Set the filters for the actions

Input Action Manager

Position and Rotation Actions

Test the scene

From File -> Build Settings and click on Add Open Scenes. With the Oculus Quest2 connected to your computer, press Build and Run, select the local destination folder for the APK.

Test the VR scene

The APK is installed on the Quest2. You should be able to see the Quest2 controllers tracked with a red line coming out from them. In the next part we will see how to use the controller to interact with the object in the scene.

To interact with the virtual environment, we need to attach the interactable components of the XR Interaction Toolkit to the digital objects and to the controllers themselves. Both controllers of the XR Rig prefab have already a XR Ray Interactor component attached (the red line). We are going to transform the Right controller in a direct interaction control to grab the digital objects in the scene.

Grabbable object

Create two new GameObjects cubes in the scene with the following parameters and name:

Create a different materials for each of them.

Table and Podium

Build the current scene to test the environment. It is now possible to interact with the red cube: with the right hand by touching it and pressing the grip button on the side of the controller; with the left controller by pointing at the cube and pressing and holding the grip button.

Create a new GameObject XR -> Locomotion System and add a new GameObject to the scene XR -> Teleportation Area in position (0,0,0). To avoiding the overlapping, we can now delete, or disable, the Plane create at the beginning of the workshop.

Create a new 3D Object -> Cylinder named customReticle with scale (0.5 , 0.01 , 0.5), position(0 , 0 , 0). Remove the component capsule collider and change the material with a green coloured one. Create a Prefab with this GameObject and delete it from the scene

The teleportation system is already working, however, the default straight line render and the key behaviour are not the most effective solutions. Moreover, we want to use both hands to teleport and interact with the objects.

Teleporting default

We are going to create two new parent GameObjects for both hands and, for each of them, we are going to add a grab controller and a teleport controller:

Teleport actions

The XR Origin should have this structure now:

Teleport actions

Create a new C# script TeleportationManager :

using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.Events;

public class TeleportationManager : MonoBehaviour
{
    public GameObject baseController; //the Right / Left Hand Controller GameObject
    public GameObject teleportationController; //the Right / Left Teleport Controller GameObject
    public InputActionReference teleportActivationReference; //The `XRI RightHand/Teleport Mode Activate` / `XRI LeftHand/Teleport Mode Activate`

    public UnityEvent onTeleportationActive;
    public UnityEvent onTeleportationCancel;

    private void Start()
    {
        teleportActivationReference.action.performed += TeleportModeActivate;
        teleportActivationReference.action.canceled += TeleportModeCancel;
    }

    private void TeleportModeCancel(InputAction.CallbackContext context) => Invoke("DeactivateTeleporter", 0.1f);

    void DeactivateTeleporter() => onTeleportationCancel.Invoke();

    private void TeleportModeActivate(InputAction.CallbackContext context) => onTeleportationActive.Invoke();

}

Attach the script to the RightHand Parent GameObject (it will be attached also to the LeftHand Parent GameObject but without the event for the XRDirectInteractor) with the following values for the public fields:

To switch between the main controller and the teleportation one, add four entries for each OnTeleportationActive and OnTeleportationCancel

On the OnTeleportationActive add:

On the OnTeleportationCancel the values will be inversed:

Teleport Manager

In the XR Ray Interactor of the RightHand Teleporter change the Line Type to Bezier Curve.

Screenshot Oculus Final teleport

Using the XR Interaction Toolkit it is possible to provide custom functions to interact with the scene. In this last part of the VR workshop we will link a virtual object with a MQTT broker to publish a message when the red cube is place to the podium.

Download the M2MQTT library and copy the folders M2Mqtt and M2MqttUnity in the Assets folder of the Unity project.

Create a new C# script mqttReceiver:

/*
The MIT License (MIT)

Copyright (c) 2018 Giovanni Paolo Vigano'

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/

using System.Collections.Generic;
using UnityEngine;
using M2MqttUnity;
using uPLibrary.Networking.M2Mqtt.Messages;

public class mqttReceiver : M2MqttUnityClient
{
    [Header("MQTT topics")]
    [Tooltip("Set the topic to subscribe. !!!ATTENTION!!! multi-level wildcard # subscribes to all topics")]
    public string topicSubscribe = "#"; // topic to subscribe. !!! The multi-level wildcard # is used to subscribe to all the topics. Attention i if #, subscribe to all topics. Attention if MQTT is on data plan
    [Tooltip("Set the topic to publish (optional)")]
    public string topicPublish = ""; // topic to publish
    public string messagePublish = ""; // topic to publish

    [Tooltip("Set this to true to perform a testing cycle automatically on startup")]
    public bool autoTest = false;

    //using C# Property GET/SET and event listener to reduce Update overhead in the controlled objects
    private string m_msg;
    public string msg
    {
        get
        {
            return m_msg;
        }
        set
        {
            if (m_msg == value) return;
            m_msg = value;
            if (OnMessageArrived != null)
            {
                OnMessageArrived(m_msg);
            }
        }
    }
    public event OnMessageArrivedDelegate OnMessageArrived;
    public delegate void OnMessageArrivedDelegate(string newMsg);

    //using C# Property GET/SET and event listener to expose the connection status
    private bool m_isConnected;
    public bool isConnected
    {
        get
        {
            return m_isConnected;
        }
        set
        {
            if (m_isConnected == value) return;
            m_isConnected = value;
            if (OnConnectionSucceeded != null)
            {
                OnConnectionSucceeded(isConnected);
            }
        }
    }
    public event OnConnectionSucceededDelegate OnConnectionSucceeded;
    public delegate void OnConnectionSucceededDelegate(bool isConnected);

    // a list to store the messages
    private List<string> eventMessages = new List<string>();

    public void Publish()
    {
        client.Publish(topicPublish, System.Text.Encoding.UTF8.GetBytes(messagePublish), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
        Debug.Log("Message published");
    }
    public void SetEncrypted(bool isEncrypted) => this.isEncrypted = isEncrypted;
    protected override void OnConnecting() => base.OnConnecting();

    protected override void OnConnected()
    {
        base.OnConnected();
        isConnected = true;
        if (autoTest) { Publish(); }
    }

    protected override void OnConnectionFailed(string errorMessage)
    {
        Debug.Log("CONNECTION FAILED! " + errorMessage);
    }

    protected override void OnDisconnected()
    {
        Debug.Log("Disconnected.");
        isConnected = false;
    }
    protected override void OnConnectionLost() => Debug.Log("CONNECTION LOST!");
    protected override void SubscribeTopics() => client.Subscribe(new string[] { topicSubscribe }, new byte[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
    protected override void UnsubscribeTopics() => client.Unsubscribe(new string[] { topicSubscribe });
    protected override void Start() => base.Start();
    protected override void DecodeMessage(string topic, byte[] message)
    {
        //The message is decoded
        msg = System.Text.Encoding.UTF8.GetString(message);

        Debug.Log("Received: " + msg);

        StoreMessage(msg);
        if (topic == topicSubscribe)
        {
            if (autoTest)
            {
                autoTest = false;
                Disconnect();
            }
        }
    }
    private void StoreMessage(string eventMsg)
    {
        if (eventMessages.Count > 50)
        {
            eventMessages.Clear();
        }
        eventMessages.Add(eventMsg);
    }

    protected override void Update() => base.Update(); // call ProcessMqttEvents()

    private void OnDestroy() => Disconnect();
    private void OnValidate()
    {
        if (autoTest)
        {
            autoConnect = true;
        }
    }
}

Create a new C# script mqttPublisher

using UnityEngine;
[RequireComponent(typeof(mqttReceiver))]
public class mqttPublisher : MonoBehaviour
{
    public mqttReceiver _eventSender;
    public string messagePublish = "";

    void Start()
    {
        _eventSender = this.gameObject.GetComponent<mqttReceiver>();

        _eventSender.OnConnectionSucceeded += OnConnectionSucceededHandler;
    }

    private void OnConnectionSucceededHandler(bool connected)
    {
        if (!connected) //publish if connected
            return;
//if the messagePublish is null, use the one of the MQTTReceiver
        if (messagePublish.Length > 0) 
        { _eventSender.messagePublish = messagePublish; } 

        _eventSender.Publish();
        Debug.Log("Publish" + messagePublish);
    }
}

Create a new C# script mqttController:

using UnityEngine;

[RequireComponent(typeof(mqttReceiver))]
public class mqttController : MonoBehaviour
{
    public GameObject objectToControl;
    public mqttReceiver _eventSender;
    void Start()
    {
        _eventSender = this.gameObject.GetComponent<mqttReceiver>();
        _eventSender.OnMessageArrived += OnMessageArrivedHandler;
    }

    private void OnMessageArrivedHandler(string newMsg)
    {
        Debug.Log("Event Fired. The message is = " + newMsg);
    }
}

Attach the three scripts to the Red Cube GameObject:

Teleport Manager

Select the Podium GameObject from the Hierarchy and add a XRSocketInteractor component. This component will link the Red Cube GameObject and trigger both connection and publishing to the MQTT Broker:

Finally, set the Box Collider of the Podium GameObject IsTrigger and add a RigidBodyKinematic and freeze X, Y and Z of both position and rotation

Enable VR support

Build and Run the project, it will be now possible to grab the RedCube, teleport to the Podium and trigger the MQTT connection by placing the RedCube near the Podium.

Enable VR support

Of course, It is possible to use any other digital models instead of the two cubes, for example an actual Bell and an Low Poly Hammer.

Enable VR support

Select the Hammer GameObject from the Hierarchy Window and in the Inspector Window add the XR Grab Interactable component with Movement Type set to Kinematic. To avoid to grab the hammer in its gravity centre, add to it a child Empty GameObject named attachPosition and move it nearby the end of the handle and drag the attachPosition GameObject on the empty field _Attach Transform of the XR Grab Interactable component. On the RigidBody component, enable Is Kinematic and disable Use Gravity

Inside the bell GameObject select the Bell and add a Box Collider component, make it bigger as the bell (or slightly smaller, e.g. Center (0 , 0.005 , 0); Size (0.007 , 0.004 , 0.007)) and set Is Trigger enable. Add a RigidBodyKinematic without Gravity and freeze X, Y and Z for both position and rotation and change Collision Detection to Continuos Speculative.

Create a new C# Script named simpleCollision

using UnityEngine;
using System.Collections;

public class simpleCollision : MonoBehaviour
{
    public static simpleCollision current;

    private void Awake()
    {
        current = this;
    }


    private void OnTriggerEnter(Collider other)
    {
        if (other.name == "hammer")
        {
            this.GetComponent<mqttReceiver>().Connect();
        }
    }

    private void OnTriggerExit(Collider other)
    {
        if (other.name == "hammer")
        {
            this.GetComponent<mqttReceiver>().Disconnect();
        }
    }
    }

Control that the other.name as the exact name of the GameObject that will collide with the bell (in this case hammer), and add the script to the Bell GameObject.

Finally, add mqttReceiver, mqttController and mqttPublisher (the same used on the RedCube GameObject will work).

Enable VR support