TCP client in a UWP Unity app on HoloLens

Published on July 21, 2017 under Blog

If you've ever had the pleasure of implementing a TCP client that would work in both the Unity editor (for development/debugging) and UWP on HoloLens (for production) you will know how painful the whole process is. Below you can find some info on how I tackled this problem. My code might not work for you as it's tailored for one specific purpose, nonetheless it might serve as a good starting point.

A bit of context on this (click here to jump straight to the code): I've been doing a lot of HoloLens development as a part of my internship with UCL Surgical Robot Vision lab. One of the pipelines I was developing involved processing IR tracking data from OptiTrack V120:Trio and sending rigid body positions/rotations to HoloLens, which is where the TCP client got involved. I'll post an article about the whole system some time in the future.

Disclaimer: This might not be the best solution, but it works. Before I came up with this I was googling for a while and could not find any useful info, so hopefully this article will at least point you in the right direction. If you want to suggest an improvement, use the comment section below.

Things to look out for

Unfortunately there isn't just a single thing that makes the whole process a pain, there is a whole lot of them. I summarise the obstacles I encountered and how I overcame them in the list below.

  1. Unity C# and UWP C# are different. There might be better ways to put it but this sentence summarises it pretty well. If you implement a TCP client the Unity way, UWP/HoloLens will complain that namespace System.Net.Sockets does not exist. If you implement a TCP client the UWP way, Unity editor with complain about Windows.Networking.Sockets or something similar. Additionally, Unity uses some "stone age" C# version that doesn't support async/await keywords which makes it even more of a hassle.
  1. UWP doesn't support threads (at the time of writing). If normally you'd just use the Thread class, in UWP you have to use the System.Threading.Tasks.Task class to achieve the same functionality. To clarify, when implementing a TCP client in Unity you absolutely most definitely have to use a separate thread/task or your code will hang the whole app. Task isn't that bad but I found it to be very poorly documented, and most importantly its interface is incompatible with Thread so you will have to rewrite your code.
  1. Most GameObject manipulations in Unity can only be performed by the main thread. I didn't have the time to find out why exactly that is but there is probably some resource-management reason, so we'll just have to deal with it. As your TCP client runs in a separate thread, whenever you try to make a change on some object Unity will call you out on not being the main thread. This happens in both the Unity editor and UWP, and I suspect everywhere else too.

The code

Here's the code I used in my project. At the moment I provide it as-is, so it will not work out of the box. I might trim it to a more general solution in the future.

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Threading;
using UnityEngine;

#if !UNITY_EDITOR
using System.Threading.Tasks;
#endif

public class USTrackingTcpClient : MonoBehaviour
{

    public USTrackingManager TrackingManager;
    public USStatusTextManager StatusTextManager;

#if !UNITY_EDITOR
    private bool _useUWP = true;
    private Windows.Networking.Sockets.StreamSocket socket;
    private Task exchangeTask;
#endif

#if UNITY_EDITOR
    private bool _useUWP = false;
    System.Net.Sockets.TcpClient client;
    System.Net.Sockets.NetworkStream stream;
    private Thread exchangeThread;
#endif

    private Byte[] bytes = new Byte[256];
    private StreamWriter writer;
    private StreamReader reader;

    public void Connect(string host, string port)
    {
        if (_useUWP)
        {
            ConnectUWP(host, port);
        }
        else
        {
            ConnectUnity(host, port);
        }
    }



#if UNITY_EDITOR
    private void ConnectUWP(string host, string port)
#else
    private async void ConnectUWP(string host, string port)
#endif
    {
#if UNITY_EDITOR
        errorStatus = "UWP TCP client used in Unity!";
#else
        try
        {
            if (exchangeTask != null) StopExchange();
        
            socket = new Windows.Networking.Sockets.StreamSocket();
            Windows.Networking.HostName serverHost = new Windows.Networking.HostName(host);
            await socket.ConnectAsync(serverHost, port);
        
            Stream streamOut = socket.OutputStream.AsStreamForWrite();
            writer = new StreamWriter(streamOut) { AutoFlush = true };
        
            Stream streamIn = socket.InputStream.AsStreamForRead();
            reader = new StreamReader(streamIn);

            RestartExchange();
            successStatus = "Connected!";
        }
        catch (Exception e)
        {
            errorStatus = e.ToString();
        }
#endif
    }

    private void ConnectUnity(string host, string port)
    {
#if !UNITY_EDITOR
        errorStatus = "Unity TCP client used in UWP!";
#else
        try
        {
            if (exchangeThread != null) StopExchange();

            client = new System.Net.Sockets.TcpClient(host, Int32.Parse(port));
            stream = client.GetStream();
            reader = new StreamReader(stream);
            writer = new StreamWriter(stream) { AutoFlush = true };

            RestartExchange();
            successStatus = "Connected!";
        }
        catch (Exception e)
        {
            errorStatus = e.ToString();
        }
#endif
    }

    private bool exchanging = false;
    private bool exchangeStopRequested = false;
    private string lastPacket = null;

    private string errorStatus = null;
    private string warningStatus = null;
    private string successStatus = null;
    private string unknownStatus = null;

    public void RestartExchange()
    {
#if UNITY_EDITOR
        if (exchangeThread != null) StopExchange();
        exchangeStopRequested = false;
        exchangeThread = new System.Threading.Thread(ExchangePackets);
        exchangeThread.Start();
#else
        if (exchangeTask != null) StopExchange();
        exchangeStopRequested = false;
        exchangeTask = Task.Run(() => ExchangePackets());
#endif
    }

    public void Update()
    {
        if(lastPacket != null)
        {
            ReportDataToTrackingManager(lastPacket);
        }

        if(errorStatus != null)
        {
            StatusTextManager.SetError(errorStatus);
            errorStatus = null;
        }
        if (warningStatus != null)
        {
            StatusTextManager.SetWarning(warningStatus);
            warningStatus = null;
        }
        if (successStatus != null)
        {
            StatusTextManager.SetSuccess(successStatus);
            successStatus = null;
        }
        if (unknownStatus != null)
        {
            StatusTextManager.SetUnknown(unknownStatus);
            unknownStatus = null;
        }
    }

    public void ExchangePackets()
    {
        while (!exchangeStopRequested)
        {
            if (writer == null || reader == null) continue;
            exchanging = true;

            writer.Write("X\n");
            Debug.Log("Sent data!");
            string received = null;

#if UNITY_EDITOR
            byte[] bytes = new byte[client.SendBufferSize];
            int recv = 0;
            while (true)
            {
                recv = stream.Read(bytes, 0, client.SendBufferSize);
                received += Encoding.UTF8.GetString(bytes, 0, recv);
                if (received.EndsWith("\n")) break;
            }
#else
            received = reader.ReadLine();
#endif

            lastPacket = received;
            Debug.Log("Read data: " + received);

            exchanging = false;
        }
    }

    private void ReportDataToTrackingManager(string data)
    {
        if (data == null)
        {
            Debug.Log("Received a frame but data was null");
            return;
        }

        var parts = data.Split(';');
        foreach(var part in parts)
        {
            ReportStringToTrackingManager(part);
        }
    }

    private void ReportStringToTrackingManager(string rigidBodyString)
    {
        var parts = rigidBodyString.Split(':');
        var positionData = parts[1].Split(',');
        var rotationData = parts[2].Split(',');

        int id = Int32.Parse(parts[0]);
        float x = float.Parse(positionData[0]);
        float y = float.Parse(positionData[1]);
        float z = float.Parse(positionData[2]);
        float qx = float.Parse(rotationData[0]);
        float qy = float.Parse(rotationData[1]);
        float qz = float.Parse(rotationData[2]);
        float qw = float.Parse(rotationData[3]);

        Vector3 position = new Vector3(x, y, z);
        Quaternion rotation = new Quaternion(qx, qy, qz, qw);

        TrackingManager.UpdateRigidBodyData(id, position, rotation);
    }

    public void StopExchange()
    {
        exchangeStopRequested = true;

#if UNITY_EDITOR
        if (exchangeThread != null)
        {
            exchangeThread.Abort();
            stream.Close();
            client.Close();
            writer.Close();
            reader.Close();

            stream = null;
            exchangeThread = null;
        }
#else
        if (exchangeTask != null) {
            exchangeTask.Wait();
            socket.Dispose();
            writer.Dispose();
            reader.Dispose();

            socket = null;
            exchangeTask = null;
        }
#endif
        writer = null;
        reader = null;
    }

    public void OnDestroy()
    {
        StopExchange();
    }

}

End of Article

Timur Kuzhagaliyev Author

I'm a computer science graduate from UCL & Caltech, working as a systems engineer at Jump Trading. Before Jump, I was doing computer vision for Video Quality Analysis team at Amazon.