Basic resources for iOS developers

The App Store is full of applications. Most of them are written in the Objective-C programming language, the native language to develop apps for iOS, basically iPhone and iPad. Other options to build iOS apps include JQuery, HTML5, C and other programming languages. There are also websites and platforms to build basic apps by dragging and dropping buttons, labels and images.  However, the native iOS is the best option in terms of performance, management of resources and user experience.

Apple provides lots of resources to help developers build their apps. Native apps are built using XCode, Apple’s IDE (Integrated Developer Environment) which includes tools to build the app’s interface, program the code, simulate the app on a simulator, and test the memory, CPU and resource usage. The apps are programmed using Objective-C, the Apple’s object oriented programming language.

To build an iOS app, developers must use a Mac OS X since other operating systems are not supported. The development tools needed are iOS SDK and Xcode. Both are free of charge and available on the Apple’s Developer Website. These tools allow any developer to build and test it using the iOS simulator, which is part of the iOS SDK. However, to run the app on a real device, developers must pay $99/year, which includes the license to publish unlimited apps in Apple’s App Store.

mac_xcode

The basic resources to learn iOS programming and build apps for iPhone and iPad among other iOS devices are:

  • Apple’s Developer Website: the developers website allows to register as an iOS Developer and download Xcode and iOS SDKs. It also includes lots of resources to learn how to build apps, API references, etc.
  • iOS Developer Library: detailed API information. References are classified by resource types, topics, and frameworks. All the information you need is here.
  • Stamford University Course: There are hundreds of courses and tutorials but the iPhone Application Development from Stamford is the most popular and well-regarded course. The video lectures, demos and sample code are available to watch and download on iTunes.
  • Ray Wenderlich’s Blog: this blog, owned by Ray Wenderlick, contains thousands of programming tutorials. The tutorials cover almost all topics and are very easy for everyone to understand and learn.
  • AppCoda’s Tutorials: Blog about iOS programming which includes lots of tutorials and a big community of programmers who share and discuss development tips.
  • Stackoverflow: Any developer knows stackoverflow. It is a question and answer site for programmers. Developers help other developers answering their questions and discussing programming topics and issues.

iphone_stamford

Besides of these websites, Internet is full of tutorials, blogs, forums and other resources. You only need time and enthusiasm to start building amazing apps for iOS devices. Hands on!

Using the Kinect to verify fall events

Microsoft’s Kinect is a powerful sensor for motion tracking and analysis. There are different applications that take advantage of its functionality of 3D motion capture. In the medical field, for example, the sensor offers great possibilities to the treatment and prevention of disease, illness or injury, as we discussed in this post.

The Kinect can be used in a fall detection system to detect when an individual is walking and suddenly falls. Its implementation is quite easy using the framework for skeleton tracking. However, we designed a system to detect fall events using a smartphone and we want to use the Kinect for verification after a fall. This verification consists of detecting if the individual is lying down in the floor. In this post we will discuss three different approaches for the verification of the fall event and its associated problems.

Skeleton tracking with the Microsoft SDK

The fall verification could consist of detecting some joints (head and hands, for example) using the skeleton tracking framework (included in the Microsoft SDK) and calculating the distance from the floor.The fall will be considered detected if the distance from the floor is almost zero from all joints.

We performed several experiments implemented with the official Microsoft SDK. The main challenge is to detect the joints when the Kinect turns on and the individual is already lying on the floor. The algorithm gives good results after small movements of the individual, but sometimes the person remains unconscious after a fall which makes this approach not useful.

skeletonKinect

Skeleton tracking with OpenNI

OpenNI is the open source SDK for the Kinect. As we discussed in this post, it has some advantages and disadvantages but is always an alternative to develop Kinecting applications. Since the first approach presented some problems to detect the individuals joints when the Kinect turns on and the individual is already lying on the floor, we decided to try with this SDK. Using this SDK we obtain better results in terms of accuracy of detection but not enough for a reliable verification of the fall.

skeletonOpenNI

User selection using depth data with OpenNI

We also performed some experiments using OpenNI and open source libraries. The fall verification consist of detecting the individual using the depth data to segment the user from the background. Once the individual is selected, we check if the individual’s bounding box is less tall than a threshold value and the position of the highest point is lower than a threshold, which means than the user is lying on the floor. This approach has the same limitation: pickup the person if is already lying on the floor when the Kinect turns on. This is the same problem we had with the previous approaches.

userselectionOpenNI

After all the experiments using both SDKs for the Kinect and different methodologies, we realized the Kinect has an important limitation to track joints when the individual is motionless. All SDKs present good accuracies after small movements but this is not useful in our system, where we want to detect if an individual is lying on the floor. Any ideas or suggestions about how to implement it?

Unlocking an Android phone

Android is an open source operative system released by Google. Carriers and manufacturers create their own version based on the Android code but they block some funtionalities and add commercial applications. Though it is an open source OS, they design a specifically Android OS version for the device and they don’t allow you to install other operative systems. Unlocking your phone allows to install a custom operating system (also known as custom ROMs) with innovatives features and root the phone to have complete access to the system.

Why root the phone ?

Rooting the device allows to modify the device’s software on the very deepest level. The greatest advantage that rooting provides is the ability to install powerful applications  that requires more than usual privileges on your device. When your phone is rooted you can
Android_unlock install apps to access and edit the system memory, speed up or slow the chip for more performance and battery life,  connect to WiFi networks that have proxy settings, block advertisements on websites and apps… Moreover, if your phone is rooted it is pretty easy to manage the operating system, backup and restore your data and manage custom ROMs.

Why install a custom ROM?

A custom ROM is simply a version of Android to replace the version of Android that the manufacturer provided on your device. Custom ROMs are created by developers and in most of cases they take away all the bloatware that is usually impossible to remove, increase performance and/or improve battery life.


There are hundreds of custom ROMs but CyanogenMod is probably the most popular and
cyanogenmod one of the most reputable. CyanogenMod is supported for a huge variety of phones and it includes some cool features such as gestures in locked-screen mode, music player and other apps, phone goggles, set up a VPN to tunnel all your IP and network data, install apps to the SD card (this safe a lot of space!) and much more. 

How to root the phone and install a custom ROM

The bootloader is the code that loads the system software on the devices and determines which applications must run in the startup (boot up) process. Manufactures block the bootloader for security reasons but also to prevent you to install custom ROMs. Unlock the bootloader is the first step to install a custom firmware on your Android phone.

HTC_BootloaderNot all the Android-phones can be unlocked although it is possible in the most advanced onces. Some manufactures such as HTC, Samsung or Google launched the official tools to unlock the bootloaders. In this post I will show the steps for HTC phones altough it is really easy to find the steps for other phones on the xdna developers forum.

 Be aware that unlock your bootloader may void your warranty.

1. Backup

Before start unlocking your phone it is highly recommended to backup all your data. You can use your Google account to backup your apps and automatically sync back when the process was complete. HTC Sync, available from the HTC website allows to backup your contacts, messages, notes, call logs….The application Go Backup Pro has the same purpose.

2. Unlocking the bootloader (HTC phones)

HTC_unlockBootloaderThe official webpage of HTCDev provides all the resources to unlock the bootloader of your HTC. If you follow all the steps of the HTCDev guide you will get your unlock code to root your phone. You need to download the Android SDK and once downloaded it basically consist on start the device into Bootloader mode and run the adb (which is on the SDK) to get the token, which is an unique code that identifies your phone. Then they will provide you with a unlock token to copy on the phone and unlock the bootloader from the Bootloader screen. All the steps are detailed in the website.

3. Rooting the phone

The bootloader is now unlocked but still not rooted. There are different ways to root your phone, for example installing the application ClockworkMod from Google Play and use the feature Reboot into Recovery. You can also create and run your temp_root script (be sure your phone is plugged in) with the following code:

adb shell mv /data/local/tmp /data/local/tmp.backup
adb shell ln -s /data /data/local/tmp
adb reboot
adb shell echo "ro.kernel.qemu=1 > /data/local.prop"
adb reboot

If you want to remove the temp root you can create and run the remove_temp_root script with the following code:

adb shell rm /data/local.prop
adb shell rm /data/local/tmp
adb shell mv /data/local/tmp.backup /data/local/tmp
adb reboot

4. Installing a custom ROM

There are lots of custom ROMs you can download and install on your phone. One of the most populars is CyanogenMod which offers features not found in the official Android based firmware. You could choose another one though, and the process will be basically identical.

clockwormodBe sure the custom ROM you chose is supported on you phone and download it. Then copy the downloaded zip file to the phone SD card. If you didn’t download Clockworkmod, you will need to do at that point. This applications allows to root your phone and install custom ROMs among other possibilities. Installing the downloaded custom ROM is as easy as tap into the Install ROM from SD Card and wait until process finish.  I highly recommend to do a backup of the current ROM before start the installation.

At this point your phone is rooted and running a customized operating system. You have super-user privileges, you can control everything from your phone,  you can do wathever you want!

Connecting an Android phone and the Kinect sensor

The Microsoft’s Kinect sensor revolutionized the touch-free gaming experience two years ago. The open source drivers and frameworks opened a new world of possibilities for researchers and developers. The official SDK made easier to develop new speech, posture and gesture applications.

The possibilities for smarthone applications are nearly endless in number and design. The Android SDK and all the resources available on Internet combined with the latest smartphone models make easy to imagine and develop new applications and new features.

nexusS

KinectForWindows

But, what if we go one step further to combine an Android-based smartphone and the Microsoft’s Kinect sensor? although the SDKs, the IDEs, the libraries, the programming language…everything is different, it is possible to integrate them. The first step consists on communicating them in order to share messages and data through the TCP protocol. In this post I will show how to program the TCP server running on the Kinect application and programmed in C# language. The client will run on the Android device and coded in Java language. 

KinectServer (C#)

KinectServerThe TCP server uses the official Microsoft Kinect SDK, the TCP protocol and C# language to run as a server. In this first implementation it accepts client connections and is able to send and receive data. The server accepts, in theory, an unlimited number of connections and it spawns a thread for each client connected.

The server class creates a new TcpListener to accept socket communications at port 3200. It creates a new Thread for client connections.

  class Server
  {
    private TcpListener tcpListener;
    private Thread listenThread;

    public Server()
    {
      this.tcpListener = new TcpListener(IPAddress.Any, 3200);
      this.listenThread = new Thread(new ThreadStart(ListenForClients));
      this.listenThread.Start();
    }
  }
}

The server blocks until a client has connected and when a client connects, it creates a new thread to handle communication with the client. 

private void ListenForClients()
{
  this.tcpListener.Start();

  while (true)
  {
     System.Console.WriteLine("Listening...");
    //blocks until a client has connected to the server
    TcpClient client = this.tcpListener.AcceptTcpClient();
    System.Console.WriteLine("Client connected");

    //create a thread to handle communications with connected client
    Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
    clientThread.Start(client);
  }
}

The next function allows to communicate the client and the server. The server waits until the client has sent a message and then it sends a new message to the client. After that, it closes the communication.

private void HandleClientComm(object client)
{
  TcpClient tcpClient = (TcpClient)client;
  NetworkStream clientStream = tcpClient.GetStream();

  byte[] message = new byte[4096];
  int bytesRead;

  while (true)
  {
    bytesRead = 0;

    try
    {
      //blocks until a client sends a message
      bytesRead = clientStream.Read(message, 0, 4096);
    }
    catch
    {
      break;
    }

    if (bytesRead == 0)
    {
      //the client has disconnected from the server
      break;
    }

    //message has successfully been received
    ASCIIEncoding encoder = new ASCIIEncoding();
    String mes = encoder.GetString(message, 0, bytesRead);
    System.Console.WriteLine(mes);

    //Server reply to the client
    byte[] buffer = encoder.GetBytes("Hello Android!");
    clientStream.Write(buffer, 0, buffer.Length);
    clientStream.flush()
   }
tcpClient.Close();
}

The server must run in background while the Kinect application is running and collecting data such as the video stream or the depth stream. To run it for testing I recommend to create a new instance from the MainWindow.

   public partial class MainWindow:Window {
          Server TCPServer = new Server();
}

Finally, to test if the server is listening for new connections, open a new command window and run the netstat. It will list the tcp connections on the port 3200.

netstat -anp tcp | find “:3200”

Telnet also allows to connect to the server and test the communication. The following command will show ‘Hello Android’ on your command window, which is the message that the server is sending to the client. 

telnet localhost 3200 //(telnet @ipserver #port)

AndroidClient (Java)

AndroidClientAny Android device running the TCP client should be able to connect and share information to the Kinect Server. The client is programmed in Java language and using the Android SDK. In this first implementation it sends a message to the server, although t is not difficult to implement the functionalities to share data from the phone such as a file or raw data collected from the sensors. 

The client will run on the background, which means we have to create a new AsyncTask to handle the communications with the server. The class has the socket and the input and output streams.

public class InternetTask extends AsyncTask<string, void,="" string=""> {
DataOutputStream dataOutputStream = null;
DataInputStream dataInputStream = null;
Socket socket;
String message;

The function doInBackground creates a new socket and exchange the messages. The socket connects to the server using its ip and port where is listening. Then it sends a message and wait for the reply which will be showed on the screen.

protected Void doInBackground(String... params) {
    try {
        socket = new Socket("10.0.0.4",3200); //connect to the server
        //Send message to the server
        dataOutputStream = new DataOutputStream(socket.getOutputStream());
        dataOutputStream.writeUTF("Hello Kinect!");

        //Receive message from the server
        dataInputStream = new DataInputStream(socket.getInputStream()); 
       message = inputStreamToString(dataInputStream);

        //Write the message on the screen
        TextView tv = (TextView) findViewById(R.id.status); //TextView element to show the received message
        tv.setText(message);

    } catch (UnknownHostException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        if (socket != null) {
            try {
                socket.close(); //close the connection
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        if (dataOutputStream != null) {
            try {
                dataOutputStream.close(); //close the output stream
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        if (dataInputStream != null) {
            try {
                dataInputStream.close(); //close the input stream
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
      } 
    

Depending on your application you will need to run the client when the app starts, when a button is clicked or when another application is running. In this example I created a button that starts the client to connect to the Kinect server.

public void btnConnectToServer(View view) {  
        InternetTask task = new InternetTask();
        task.execute();         
}

Tilt the Kinect from the Android phone

At this point the Kinect and the Android can exchange messages. However, it is not difficult to implement an application to send some data from the phone, such as a file, a song or raw data collected with the sensors. For example, you can tilt the Kinect to point to your phone using the raw values collected with the orientation sensor. This requires some work so I will explain how to do in the following post, although you can tilt the Kinect using Up and Down buttons.

Tilt Kinect

Any ideas for future applications integrating the Kinect and the Android phone?

Google I/O at the Brightcove

Google I/O is an annual developer-focused conference held by Google. The event takes place in San Francisco (CA) but they have extended events in many cities around theIMGP6679.jpg world. The Brightcove, in Boston, showed off the Zencoder cloud transcoding service at the conference. I had the opportunity to join developers, engineers and passionate students to watch and discuss some of the conferences.

Google Play

Google Play is the Google’s distribution platform which allows to download music, books, movies, videos, games and applications. Google redesigned the platform both mobile and web versions. They also introduced new services such as Google Play Games and new features such as the personalize suggestions based on the user preferences and search history.

Google Play Games is the new game service with real-time multiplayer action and multi-platform synchronization. The gaming experience becomes more social, allowing for example to invite and play with your friends and share your scores. It can be synchronized with other platforms such as tablets, computers, smartphones running both Android and iOS, which is an interesting feature for the developers.

Google Play StoreA monthly music subscription service has been introduced for the Google Play Music. It allows to play all songs available in the market and also to upload your songs.  Google Play Books now allows to upload your files to the cloud and it includes a Read Now section that features books you have recently uploaded, purchased or read.

Google also announced a new education program that will help teachers manage and push out apps, books and other educational content to student tablets and computers: Google Play for Educational Program.

Google +

The social network of Google has been redesigned  with responsive design in the feeds, a tags feature to dig into more content and a new photo manager and editor.

google-plus-featuresWhen you upload hundreds of pictures, Google+ Photos will choose the best ones based on the bright and contrast, number of people in the picture, famous landmarks and attractions and it will use face-recognition algorithms to select the one where you and your friends appear in the picture.

Google Hangouts is a new application that substitutes Google Talk. The new multi-platform app allows to chat, share videos and pictures with your friends. It is available on Android, iOS and Chrome.

Google Maps and localization

Google released new APIs to improve localization and battery life on Android devices. An interactive map will recommend location, compare travel modes and integrate Street View and Google Earth on Google Maps. The new APIs includes better outdoor localization, indoor localization and navigation and human activity recognition.

Android2312719-a6

The most common IDE to develop Android apps is Eclipse which runs on Windows, Linux and Mac. But Google developed their own IDE, Android Studio that runs on all operative systems.

The new features that Android Studio includes are the navigation drawer, a better integration with Google Analytics and the possibility to launch beta and alpha versions of the applications.

Samsung Galaxy S4

The new Samsung Galaxy S4 from Google is a completely unlocked device. It comes with an unlocked bootloader that gives to the developers a root access to the operating system. The phone will be release on June and it costs $649.

Samsung-Galaxy-S4-Google-Edition-2

Other features

Google Voice Search is already available for Android devices but Google wants to expand its voice search capabilities and other features on Google Search for desktop devices, a fundamentally change the way the people look things up on its search engine.

Google Wallet, the online payment method from Google, enables now to send money as a Gmail attachment over other interesting functionalities. It will be integrated on the Chrome browser wich gets faster, more secure and includes faster video-streaming.

Using Android’s sensors

Most of the smartphones have a wide array of sensors to measure motion, position and envirommental conditions. The Android platform supports several sensors, both hardware-based and virtual, such as accelerometer, gyroscope, proximity, temperature, illuminance or humidity.

Managing sensors in Android is not difficult and there are good references and documentation on the developers website.  The basic steps to get the sensor values, such as the accelerometer’s data or the temperature, are summarized in this post.

1.List the sensors

Before using the senors it’s important to know which are available in the device. To check it on the app:

List sensors

2. Enable the sensors

To enable the sensors and get the data we have to obtain instances of SensorManager and sensors we want to access. Then enable them and register the listeners to start collecting the data. In the example we enable accelerometer, orientation and temperature sensors.
Enable sensors

It is important not to forget to unregister the listeners when the app is not working or when sensor values are not needed. At least it is needed to register them on the onCreate() and onResume() methods and unregister them on the onPause() method. Unregister sensors

3. Get the sensor values

The OnSensorChanged() method is used to get the sensor values, the timestamp, the type of sensor or the accuracity. The contents of the values array depend on which sensor type is being monitored. For the accelerometer values, values] correspond to i=x,y,z but for other sensors check Android documentationGet values

4. Use the sensors data

Usually sensor values are used in the app but it also can be stored in a file to analyze, for example, on the computer. Following there are two examples, the first one shows the data on the device screen and the second one correspond to a plot of the collected accelerometer data while walking.

StepsAcc

SensorValuesScreen

Kinect overview before starting programming

The Microsoft Kinect is a set of sensors developed as a peripheral device to use with the Xbox 360 gaming console. Since it was released, hackers immediately saw potential in the device far beyond it and created open source libraries to use in other applications. Microsoft also released the official SDK and the Kinect for Windows, a more powerful device to use in research. Nowadays there is a big community of developers and researchers around the world and several new applications are emerging.

Developing for Kinect is really easy. There are lots of official and non-official tools, libraries, demos, tutorials….But Kinect sensor has some limitations that you should know before starting developing for this device.

Kinect for Windows vs Kinect for XBox 360

Kinect for Windows is specifically designed to be used with computers. It is licensed for commercial app distribution so it is the best option for development. Kinect for Xbox 360 was designed and tested for the console but can also be used on development with some limitations. 

The features unique to the Windows version of Kinect include:
kinect

  • A shortened USB cable to ensure reliability
  • A small dongle to increase compatibility with other USB devices
  • Expanded skeletal tracking abilities, including “seated” skeletal tracking that focuses on 10 upper body joints
  • A firmware update called “Near Mode” that allows the depth sensor to accurately track objects and users seated as close as 40 cm from the device
  • Expanded speech recognition including four new languages, French, Spanish, Italian, and Japanese
  • Language packs improving speech recognition for many other English dialects and accents
  • Improved synchronization between color and depth streams
  • Improved face tracking

Official Kinect SDK vs Open Source alternatives

The official SDK maintained for Microsoft is better than open source alternatives in some applications such as the skeleton tracking. However the force of the open source community with OpenNI (drivers, APIs, libraries, demos) and OpenKinect (drivers) to create the SDK, middleware libraries and apps.Both have cons and pros.


Official SDK

  • Programming languages supported: C++, C#, or Visual Basic by using Microsoft Visual Studio.
  • Operative System support: Windows 7 and 8.
  • Documentation and support: official website, development toolkit and support forum.
  • Calibration: not needed.SDKs

OpenNI and OpenKinect

  • Programming languages supported: Python, C, C++, C#, Java…not requiring Visual Studio.
  • Operative System support: Linux, Mac OS X and Windows.
  • Documentation and support: website, support forum, twitter...
  • Calibration: needed.

Features and limitations

The Kinect’s image, audio, and depth sensors allow to detect movements, identify faces and recognize speech of players. However they have some physical like the sensing range. On the other hand, Kinect for Windows SDK frameworks also have limitations such as the number of tracking skeletons.depth

  • RGB camera: angular field of view of 57° horizontally (plus 27° up or down with the motorized pivot). and 43° vertically.
  • Depth sensor: viewing distance range from 0.8 m to 4m. Practical limits are from 1.2m to 3.5m.
  • Depth near mode: viewing distance from 0.4m to 3m.
  • Audio beam forming: angular field to identify the current sound source of 100° in intervals of 10°.
  • Skeleton tracking: normal mode with 20-joints per player and seated mode with 10-joints. Both modes have simultaneously tracking up to six people including two active players (motion analysis and feature extraction of 20 joints per player).
  • Interactions: library with basic gestures (e.g. targeting and selecting with a cursor) which also supports the definition of gestures.
  • Face tracking: angle limits to track face movements are +-45 degrees (yaw), +-90° (roll) and +-25° (pitch).

skeletonTracking