Core ML (Machine Learning) in iOS

Machine Learning has undoubtedly been one of the hottest topics over the past 3 years, It is one of the technology that is taking mobile app development to a new level. Machine Learning making our smart devices really smart. Machine Learning can now recognise speech, images, gestures and translate voices with extraordinary success rates.

What is Machine Learning? 
    Machine Learning is a set of methods that can automatically detect patterns in data, and then use the uncovered patterns to predict future date, or to perform other kinds of decision making under uncertainty.
      Machine Learning is all about using data to answer questions. First data(e.g., images, text, voice) is provided along with answers to the data, then the computer (model) is trained on this data so it can learn and later make predictions on the mobile device.
      At WWDC 2017, Apple released a lot of exciting frameworks and API's for developer to use, among all the new frameworks, one of the most popular is definitely Core ML. It makes it easy for iOS Developers to add deep machine learning to their apps.

Core ML
   Core ML is a framework that can be harnessed to integrate machine learning models into your app. One should not need an extensive knowledge about neural networks or machine learning, Core ML is that you can use pre-trained data models as long as you convert it into Core ML model.

   

   Core ML is very easy to use.  Let's integrate Core ML into our demo application. Core ML model that is available on Apple's Developer Website.  I'll walk you through the development of an app that uses a pre-trained Core ML model to detect an object from an input image. So, here we go.

Prerequisite:

• Mac OS(10.12 or above)
• Xcode 9 or above
• iOS 11 or above

Demo App Overview

    The app we are trying to make is fairly simple. Our app lets user choose a photo from their library, then the machine learning algorithm will try to predict what the object is in the picture.

Getting Started

    Select the single-view application template for this project, and make sure the language is set to Swift.

Create User Interface in Storyboard
    Let's begin...! The first thing we want to do is to head on over to Main.storyboard and add a couple of UI elements to the view. Choose the view controller in storyboard, and add the below UI elements
UIButton: Photos - It will help to input an image from photo library
UIImage - This will display your input image
UILabel - This will display the predicated label

Create outlets for UIButton, UIImage and UILabel in storyboard

           @IBOutlet weak var displayImageView: UIImageView!

           @IBOutlet weak var objectLabel: UILabel!

Integrating the Core ML Data Model
  As we mentioned earlier, we need a pre-trained model to work with Core ML. You can build your own model, but for this demo, we will use the pre-trained model available on this link Inception V3, download and add the model to our project.

Now, let's add the model in our code. Go back to ViewController.swift. First, import the CoreML framework at the very beginning

                           import CoreML

Next, declare a model variable in the class for Inceptionv3 model, and initialize it in the ViewDidLoad() method

                var inception_model: Inceptionv3!

    

                override func viewDidLoad() {

                    super.viewDidLoad()

        

                 inception_model = Inceptionv3()

             }

Add the method call in the didFinishPickingMediaWithInfo delegate method added earlier. The delegate method should look like this:

   func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey : Any]) {

        

        picker.dismiss(animated: true, completion: nil)

        

        guard let image = info[UIImagePickerController.InfoKey.originalImage] else {

            return

        }

        self.predictImage(predict_image: image as! UIImage)

    }

Add the below lines of code in a function, in that you don't understand the most that code, no worries, this is really some advance Core Image code, which is out of the scope , convert the image to pixel buffer

    func predictImage(predict_image: UIImage) {

        UIGraphicsBeginImageContextWithOptions(CGSize(width: 299, height: 299), true, 2.0)

        predict_image.draw(in: CGRect(x: 0, y: 0, width: 299, height: 299))

        let newImage = UIGraphicsGetImageFromCurrentImageContext()!

        UIGraphicsEndImageContext()

        

        let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary

        var pixelBuffer : CVPixelBuffer?

        let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(newImage.size.width), Int(newImage.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)

        guard (status == kCVReturnSuccess) else {

            return

        }

        

        CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

        let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)

        

        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()

        let context = CGContext(data: pixelData, width: Int(newImage.size.width), height: Int(newImage.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue) //3

        

        context?.translateBy(x: 0, y: newImage.size.height)

        context?.scaleBy(x: 1.0, y: -1.0)

        

        UIGraphicsPushContext(context!)

        newImage.draw(in: CGRect(x: 0, y: 0, width: newImage.size.width, height: newImage.size.height))

        UIGraphicsPopContext()

        CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

        displayImageView.image = newImage;

        

        // CoreMl Predicates the object type

        guard let prediction = try? inception_model.prediction(image: pixelBuffer!) else {

            return

        }

        

        objectLabel.text = "This is \(prediction.classLabel)"

    }

The Inceptionv3 class has to be generated method called predictImage(image:) that is used to predict the object in the given image. Here we pass the method with the pixelBuffer variable, which is the resized image. Once the prediction, which is of the type String, is returned, we update the classifier label to set its text to what it has been recognized.

And that's it! You're good to go. Run the project and get ready to predict the images.