Using the Kinect to verify fall events

Microsoft’s Kinect is a powerful sensor for motion tracking and analysis. There are different applications that take advantage of its functionality of 3D motion capture. In the medical field, for example, the sensor offers great possibilities to the treatment and prevention of disease, illness or injury, as we discussed in this post.

The Kinect can be used in a fall detection system to detect when an individual is walking and suddenly falls. Its implementation is quite easy using the framework for skeleton tracking. However, we designed a system to detect fall events using a smartphone and we want to use the Kinect for verification after a fall. This verification consists of detecting if the individual is lying down in the floor. In this post we will discuss three different approaches for the verification of the fall event and its associated problems.

Skeleton tracking with the Microsoft SDK

The fall verification could consist of detecting some joints (head and hands, for example) using the skeleton tracking framework (included in the Microsoft SDK) and calculating the distance from the floor.The fall will be considered detected if the distance from the floor is almost zero from all joints.

We performed several experiments implemented with the official Microsoft SDK. The main challenge is to detect the joints when the Kinect turns on and the individual is already lying on the floor. The algorithm gives good results after small movements of the individual, but sometimes the person remains unconscious after a fall which makes this approach not useful.

skeletonKinect

Skeleton tracking with OpenNI

OpenNI is the open source SDK for the Kinect. As we discussed in this post, it has some advantages and disadvantages but is always an alternative to develop Kinecting applications. Since the first approach presented some problems to detect the individuals joints when the Kinect turns on and the individual is already lying on the floor, we decided to try with this SDK. Using this SDK we obtain better results in terms of accuracy of detection but not enough for a reliable verification of the fall.

skeletonOpenNI

User selection using depth data with OpenNI

We also performed some experiments using OpenNI and open source libraries. The fall verification consist of detecting the individual using the depth data to segment the user from the background. Once the individual is selected, we check if the individual’s bounding box is less tall than a threshold value and the position of the highest point is lower than a threshold, which means than the user is lying on the floor. This approach has the same limitation: pickup the person if is already lying on the floor when the Kinect turns on. This is the same problem we had with the previous approaches.

userselectionOpenNI

After all the experiments using both SDKs for the Kinect and different methodologies, we realized the Kinect has an important limitation to track joints when the individual is motionless. All SDKs present good accuracies after small movements but this is not useful in our system, where we want to detect if an individual is lying on the floor. Any ideas or suggestions about how to implement it?

Advertisements

Connecting an Android phone and the Kinect sensor

The Microsoft’s Kinect sensor revolutionized the touch-free gaming experience two years ago. The open source drivers and frameworks opened a new world of possibilities for researchers and developers. The official SDK made easier to develop new speech, posture and gesture applications.

The possibilities for smarthone applications are nearly endless in number and design. The Android SDK and all the resources available on Internet combined with the latest smartphone models make easy to imagine and develop new applications and new features.

nexusS

KinectForWindows

But, what if we go one step further to combine an Android-based smartphone and the Microsoft’s Kinect sensor? although the SDKs, the IDEs, the libraries, the programming language…everything is different, it is possible to integrate them. The first step consists on communicating them in order to share messages and data through the TCP protocol. In this post I will show how to program the TCP server running on the Kinect application and programmed in C# language. The client will run on the Android device and coded in Java language. 

KinectServer (C#)

KinectServerThe TCP server uses the official Microsoft Kinect SDK, the TCP protocol and C# language to run as a server. In this first implementation it accepts client connections and is able to send and receive data. The server accepts, in theory, an unlimited number of connections and it spawns a thread for each client connected.

The server class creates a new TcpListener to accept socket communications at port 3200. It creates a new Thread for client connections.

  class Server
  {
    private TcpListener tcpListener;
    private Thread listenThread;

    public Server()
    {
      this.tcpListener = new TcpListener(IPAddress.Any, 3200);
      this.listenThread = new Thread(new ThreadStart(ListenForClients));
      this.listenThread.Start();
    }
  }
}

The server blocks until a client has connected and when a client connects, it creates a new thread to handle communication with the client. 

private void ListenForClients()
{
  this.tcpListener.Start();

  while (true)
  {
     System.Console.WriteLine("Listening...");
    //blocks until a client has connected to the server
    TcpClient client = this.tcpListener.AcceptTcpClient();
    System.Console.WriteLine("Client connected");

    //create a thread to handle communications with connected client
    Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
    clientThread.Start(client);
  }
}

The next function allows to communicate the client and the server. The server waits until the client has sent a message and then it sends a new message to the client. After that, it closes the communication.

private void HandleClientComm(object client)
{
  TcpClient tcpClient = (TcpClient)client;
  NetworkStream clientStream = tcpClient.GetStream();

  byte[] message = new byte[4096];
  int bytesRead;

  while (true)
  {
    bytesRead = 0;

    try
    {
      //blocks until a client sends a message
      bytesRead = clientStream.Read(message, 0, 4096);
    }
    catch
    {
      break;
    }

    if (bytesRead == 0)
    {
      //the client has disconnected from the server
      break;
    }

    //message has successfully been received
    ASCIIEncoding encoder = new ASCIIEncoding();
    String mes = encoder.GetString(message, 0, bytesRead);
    System.Console.WriteLine(mes);

    //Server reply to the client
    byte[] buffer = encoder.GetBytes("Hello Android!");
    clientStream.Write(buffer, 0, buffer.Length);
    clientStream.flush()
   }
tcpClient.Close();
}

The server must run in background while the Kinect application is running and collecting data such as the video stream or the depth stream. To run it for testing I recommend to create a new instance from the MainWindow.

   public partial class MainWindow:Window {
          Server TCPServer = new Server();
}

Finally, to test if the server is listening for new connections, open a new command window and run the netstat. It will list the tcp connections on the port 3200.

netstat -anp tcp | find “:3200”

Telnet also allows to connect to the server and test the communication. The following command will show ‘Hello Android’ on your command window, which is the message that the server is sending to the client. 

telnet localhost 3200 //(telnet @ipserver #port)

AndroidClient (Java)

AndroidClientAny Android device running the TCP client should be able to connect and share information to the Kinect Server. The client is programmed in Java language and using the Android SDK. In this first implementation it sends a message to the server, although t is not difficult to implement the functionalities to share data from the phone such as a file or raw data collected from the sensors. 

The client will run on the background, which means we have to create a new AsyncTask to handle the communications with the server. The class has the socket and the input and output streams.

public class InternetTask extends AsyncTask<string, void,="" string=""> {
DataOutputStream dataOutputStream = null;
DataInputStream dataInputStream = null;
Socket socket;
String message;

The function doInBackground creates a new socket and exchange the messages. The socket connects to the server using its ip and port where is listening. Then it sends a message and wait for the reply which will be showed on the screen.

protected Void doInBackground(String... params) {
    try {
        socket = new Socket("10.0.0.4",3200); //connect to the server
        //Send message to the server
        dataOutputStream = new DataOutputStream(socket.getOutputStream());
        dataOutputStream.writeUTF("Hello Kinect!");

        //Receive message from the server
        dataInputStream = new DataInputStream(socket.getInputStream()); 
       message = inputStreamToString(dataInputStream);

        //Write the message on the screen
        TextView tv = (TextView) findViewById(R.id.status); //TextView element to show the received message
        tv.setText(message);

    } catch (UnknownHostException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        if (socket != null) {
            try {
                socket.close(); //close the connection
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        if (dataOutputStream != null) {
            try {
                dataOutputStream.close(); //close the output stream
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        if (dataInputStream != null) {
            try {
                dataInputStream.close(); //close the input stream
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
      } 
    

Depending on your application you will need to run the client when the app starts, when a button is clicked or when another application is running. In this example I created a button that starts the client to connect to the Kinect server.

public void btnConnectToServer(View view) {  
        InternetTask task = new InternetTask();
        task.execute();         
}

Tilt the Kinect from the Android phone

At this point the Kinect and the Android can exchange messages. However, it is not difficult to implement an application to send some data from the phone, such as a file, a song or raw data collected with the sensors. For example, you can tilt the Kinect to point to your phone using the raw values collected with the orientation sensor. This requires some work so I will explain how to do in the following post, although you can tilt the Kinect using Up and Down buttons.

Tilt Kinect

Any ideas for future applications integrating the Kinect and the Android phone?

Kinect overview before starting programming

The Microsoft Kinect is a set of sensors developed as a peripheral device to use with the Xbox 360 gaming console. Since it was released, hackers immediately saw potential in the device far beyond it and created open source libraries to use in other applications. Microsoft also released the official SDK and the Kinect for Windows, a more powerful device to use in research. Nowadays there is a big community of developers and researchers around the world and several new applications are emerging.

Developing for Kinect is really easy. There are lots of official and non-official tools, libraries, demos, tutorials….But Kinect sensor has some limitations that you should know before starting developing for this device.

Kinect for Windows vs Kinect for XBox 360

Kinect for Windows is specifically designed to be used with computers. It is licensed for commercial app distribution so it is the best option for development. Kinect for Xbox 360 was designed and tested for the console but can also be used on development with some limitations. 

The features unique to the Windows version of Kinect include:
kinect

  • A shortened USB cable to ensure reliability
  • A small dongle to increase compatibility with other USB devices
  • Expanded skeletal tracking abilities, including “seated” skeletal tracking that focuses on 10 upper body joints
  • A firmware update called “Near Mode” that allows the depth sensor to accurately track objects and users seated as close as 40 cm from the device
  • Expanded speech recognition including four new languages, French, Spanish, Italian, and Japanese
  • Language packs improving speech recognition for many other English dialects and accents
  • Improved synchronization between color and depth streams
  • Improved face tracking

Official Kinect SDK vs Open Source alternatives

The official SDK maintained for Microsoft is better than open source alternatives in some applications such as the skeleton tracking. However the force of the open source community with OpenNI (drivers, APIs, libraries, demos) and OpenKinect (drivers) to create the SDK, middleware libraries and apps.Both have cons and pros.


Official SDK

  • Programming languages supported: C++, C#, or Visual Basic by using Microsoft Visual Studio.
  • Operative System support: Windows 7 and 8.
  • Documentation and support: official website, development toolkit and support forum.
  • Calibration: not needed.SDKs

OpenNI and OpenKinect

  • Programming languages supported: Python, C, C++, C#, Java…not requiring Visual Studio.
  • Operative System support: Linux, Mac OS X and Windows.
  • Documentation and support: website, support forum, twitter...
  • Calibration: needed.

Features and limitations

The Kinect’s image, audio, and depth sensors allow to detect movements, identify faces and recognize speech of players. However they have some physical like the sensing range. On the other hand, Kinect for Windows SDK frameworks also have limitations such as the number of tracking skeletons.depth

  • RGB camera: angular field of view of 57° horizontally (plus 27° up or down with the motorized pivot). and 43° vertically.
  • Depth sensor: viewing distance range from 0.8 m to 4m. Practical limits are from 1.2m to 3.5m.
  • Depth near mode: viewing distance from 0.4m to 3m.
  • Audio beam forming: angular field to identify the current sound source of 100° in intervals of 10°.
  • Skeleton tracking: normal mode with 20-joints per player and seated mode with 10-joints. Both modes have simultaneously tracking up to six people including two active players (motion analysis and feature extraction of 20 joints per player).
  • Interactions: library with basic gestures (e.g. targeting and selecting with a cursor) which also supports the definition of gestures.
  • Face tracking: angle limits to track face movements are +-45 degrees (yaw), +-90° (roll) and +-25° (pitch).

skeletonTracking