Connecting an Android phone and the Kinect sensor

The Microsoft’s Kinect sensor revolutionized the touch-free gaming experience two years ago. The open source drivers and frameworks opened a new world of possibilities for researchers and developers. The official SDK made easier to develop new speech, posture and gesture applications.

The possibilities for smarthone applications are nearly endless in number and design. The Android SDK and all the resources available on Internet combined with the latest smartphone models make easy to imagine and develop new applications and new features.

nexusS

KinectForWindows

But, what if we go one step further to combine an Android-based smartphone and the Microsoft’s Kinect sensor? although the SDKs, the IDEs, the libraries, the programming language…everything is different, it is possible to integrate them. The first step consists on communicating them in order to share messages and data through the TCP protocol. In this post I will show how to program the TCP server running on the Kinect application and programmed in C# language. The client will run on the Android device and coded in Java language. 

KinectServer (C#)

KinectServerThe TCP server uses the official Microsoft Kinect SDK, the TCP protocol and C# language to run as a server. In this first implementation it accepts client connections and is able to send and receive data. The server accepts, in theory, an unlimited number of connections and it spawns a thread for each client connected.

The server class creates a new TcpListener to accept socket communications at port 3200. It creates a new Thread for client connections.

  class Server
  {
    private TcpListener tcpListener;
    private Thread listenThread;

    public Server()
    {
      this.tcpListener = new TcpListener(IPAddress.Any, 3200);
      this.listenThread = new Thread(new ThreadStart(ListenForClients));
      this.listenThread.Start();
    }
  }
}

The server blocks until a client has connected and when a client connects, it creates a new thread to handle communication with the client. 

private void ListenForClients()
{
  this.tcpListener.Start();

  while (true)
  {
     System.Console.WriteLine("Listening...");
    //blocks until a client has connected to the server
    TcpClient client = this.tcpListener.AcceptTcpClient();
    System.Console.WriteLine("Client connected");

    //create a thread to handle communications with connected client
    Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
    clientThread.Start(client);
  }
}

The next function allows to communicate the client and the server. The server waits until the client has sent a message and then it sends a new message to the client. After that, it closes the communication.

private void HandleClientComm(object client)
{
  TcpClient tcpClient = (TcpClient)client;
  NetworkStream clientStream = tcpClient.GetStream();

  byte[] message = new byte[4096];
  int bytesRead;

  while (true)
  {
    bytesRead = 0;

    try
    {
      //blocks until a client sends a message
      bytesRead = clientStream.Read(message, 0, 4096);
    }
    catch
    {
      break;
    }

    if (bytesRead == 0)
    {
      //the client has disconnected from the server
      break;
    }

    //message has successfully been received
    ASCIIEncoding encoder = new ASCIIEncoding();
    String mes = encoder.GetString(message, 0, bytesRead);
    System.Console.WriteLine(mes);

    //Server reply to the client
    byte[] buffer = encoder.GetBytes("Hello Android!");
    clientStream.Write(buffer, 0, buffer.Length);
    clientStream.flush()
   }
tcpClient.Close();
}

The server must run in background while the Kinect application is running and collecting data such as the video stream or the depth stream. To run it for testing I recommend to create a new instance from the MainWindow.

   public partial class MainWindow:Window {
          Server TCPServer = new Server();
}

Finally, to test if the server is listening for new connections, open a new command window and run the netstat. It will list the tcp connections on the port 3200.

netstat -anp tcp | find “:3200”

Telnet also allows to connect to the server and test the communication. The following command will show ‘Hello Android’ on your command window, which is the message that the server is sending to the client. 

telnet localhost 3200 //(telnet @ipserver #port)

AndroidClient (Java)

AndroidClientAny Android device running the TCP client should be able to connect and share information to the Kinect Server. The client is programmed in Java language and using the Android SDK. In this first implementation it sends a message to the server, although t is not difficult to implement the functionalities to share data from the phone such as a file or raw data collected from the sensors. 

The client will run on the background, which means we have to create a new AsyncTask to handle the communications with the server. The class has the socket and the input and output streams.

public class InternetTask extends AsyncTask<string, void,="" string=""> {
DataOutputStream dataOutputStream = null;
DataInputStream dataInputStream = null;
Socket socket;
String message;

The function doInBackground creates a new socket and exchange the messages. The socket connects to the server using its ip and port where is listening. Then it sends a message and wait for the reply which will be showed on the screen.

protected Void doInBackground(String... params) {
    try {
        socket = new Socket("10.0.0.4",3200); //connect to the server
        //Send message to the server
        dataOutputStream = new DataOutputStream(socket.getOutputStream());
        dataOutputStream.writeUTF("Hello Kinect!");

        //Receive message from the server
        dataInputStream = new DataInputStream(socket.getInputStream()); 
       message = inputStreamToString(dataInputStream);

        //Write the message on the screen
        TextView tv = (TextView) findViewById(R.id.status); //TextView element to show the received message
        tv.setText(message);

    } catch (UnknownHostException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        if (socket != null) {
            try {
                socket.close(); //close the connection
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        if (dataOutputStream != null) {
            try {
                dataOutputStream.close(); //close the output stream
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        if (dataInputStream != null) {
            try {
                dataInputStream.close(); //close the input stream
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
      } 
    

Depending on your application you will need to run the client when the app starts, when a button is clicked or when another application is running. In this example I created a button that starts the client to connect to the Kinect server.

public void btnConnectToServer(View view) {  
        InternetTask task = new InternetTask();
        task.execute();         
}

Tilt the Kinect from the Android phone

At this point the Kinect and the Android can exchange messages. However, it is not difficult to implement an application to send some data from the phone, such as a file, a song or raw data collected with the sensors. For example, you can tilt the Kinect to point to your phone using the raw values collected with the orientation sensor. This requires some work so I will explain how to do in the following post, although you can tilt the Kinect using Up and Down buttons.

Tilt Kinect

Any ideas for future applications integrating the Kinect and the Android phone?

Advertisements

5 thoughts on “Connecting an Android phone and the Kinect sensor

  1. I need some idea for my final year project in university using Kinect camera + android application. Please reply my email. Thank you… :)

    • I need some idea for my final year project in university using Kinect camera + android application. Please reply my email. Thank you… :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s