Snip Insights – Cross-platform open source AI tool for intelligent screen capture

Devices and technologies are moving forward at a rapid pace, though the everyday tools we use remain relatively unchanged. What if we could infuse AI into everyday tools to delight and inspire developers to do more using Microsoft AI platform? With just a little bit of creativity and using Microsoft's current AI offerings, we can bring AI capabilities closer to customers and create applications that will inspire every organization, every developer, and every person on this planet.

Introducing Snip Insights

An open source cross-platform AI tool for intelligent screen capture. Snip Insights revolutionizes the way users can generate insights from screen captures. The initial prototype of Snip Insights, built for Windows OS and released at Microsoft Build 2018 in May, was created by Microsoft Garage interns based out of Vancouver, BC. Our team at Microsoft AI Lab in collaboration with the Microsoft AI CTO team took Snip Insights to the next level by giving the tool a new intuitive UX, cross-platform availability (MacOS, Linux, and Windows), and free download and usage under MSA license. Snip Insights leverages Microsoft Azure's Cognitive Services APIs to increase users' productivity by reducing the number of steps needed to gain intelligent insights.

The Solution

Snip Insights is an open source desktop utility that enables cross-platform users to retrieve intelligent insights over a snip or screenshot. Screenshots are essentially snapshots of moments. Snip Insights leverages cloud AI services to convert images to translated text, automatically detect and tag image content, along with many smart image suggestions that improve workflows while showcasing Azure Cognitive Services’ potential. By combining a familiar tool with Cognitive Services, we have created a one-stop shop for all your image insights. Imagine that you have a scan of a textbook or work report. Rather than having to manually type out the information, snipping it will return editable text in just one click using OCR. Or maybe you’re scrolling through your social media feed and come across someone wearing a cool pair of shoes, you can simply snip to find out where to purchase them. Snip Insights can show you relevant information based on what you’ve just snipped, including identifying famous people and landmarks. It streamlines the process of screenshotting, saving the picture, uploading it to reverse image search engines, and then drawing results from there.

Snip Insights highlights

Celebrity Search – Snip an image of a celebrity and the tool will provide you with all the news and information about the celebrity.

Object Detection and Bing Visual Search – You like a t-shirt or pair of shoes your friend is wearing in a pic, and want to know where to buy it from. Just use snip insights and it will give you recommendations, similar product images and where to buy from in a matter of a second.

OCR, Language Translation and cross-platform compatibility – You like a phrase in a specific language, let’s say English, and wish to convert that to French or another language to personalize it and vice versa. Just use Snip Insights to get those insights. The tool is free and works on Windows, Linux,  and MacOS.

Supported Platforms: Windows, Mac OS, and Linux

The Snip Insights app is available for three platforms:

• Universal Windows Platform (UWP)
• Mac OS
• Linux

Xamarin.Forms GTK# App Mac OS and Linux for Snip Insights

Xamarin.Forms enables you to build native UIs for iOS, Android, macOS, Linux, and Windows from a single shared codebase. You can dive into app development with Xamarin.Forms by following our free self-guided learning from Xamarin University.

Xamarin.Forms has preview support for GTK# apps. GTK# is a graphical user interface toolkit that links the GTK+ toolkit and a variety of GNOME libraries, allowing the development of fully native GNONE graphics apps using Mono and .NET. Xamarin.Forms GTK#.

Installation

Windows

• Download the zip.
• Install the certificate (".cer" file) according to the instructions detailed on the tInstall Certificate section.
• Install Snip Insights by double click over the .appx package file.

Linux

• Install Mono by following the official steps depending on your Linux distribution.
• Install the .deb package.
• Launch the app from the applications section.

Mac OS

• Download and install Mono (Stable channel).
  • Such includes GTK#, the UI toolkit on which Xamarin.Forms relies on this project.
• Install the .pckg as a normal macOS application.
• Snip Insights app is available on the Applications section on macOS.

Requirements

Visual Studio 2017 version 15.8 or Visual Studio for Mac version 7.6.3

Using your own subscription

To add the keys to Snip Insights, a Microsoft Garage Project, start the application. Once running, click/tap the settings icon in the toolbar. Scroll down until you find the Cognitive Services, enable AI assistance toggle, and toggle it to the on position. You should now see the Insight Service Keys section.

• Entity Search – Create new Entity Search Cognitive Service. Once created, you can display the keys. Select one and paste into Settings.
• Image Analysis – In Azure, create a Computer Vision API Cognitive Service and use its key.
• Image Search – In Azure, create a Bing Search v7 API Cognitive Service and use its key.
• Text Recognition – You can use the same key as used in Image Analysis. Both Image Analysis and Text Recognition use Computer Vision API.
• Translator – Use the Translator Text API Cognitive Service.
• Content Moderator – Use the Content Moderator API Cognitive Service.

For the LUIS App ID and Key, you will need to create a Language Understanding application in the Language Understanding Portal. Use the following steps to create your LUIS App and retrieve an App ID

• Click on create new app button.
• Provide an app name. Leave culture (English) and description as defaults.
• Click done.
• In the left navigation pane, click entities.
• Click manage prebuild entities.
• Select datetimeV2 and email.
• Click done.
• Click the train button at the top of the page.
• Click the publish tab.
• Click the publish to production slot button.
• At the bottom of the screen, you will see a list with a Key String field. Click the copy button and paste that key value into the LUIS Key field in settings for Snip Insights.
• Click the settings tab at the top.
• Copy the application ID shown and paste into the LUIS App Id field in Settings for Snip Insights.

You can now paste each key in the settings panel of the application. Remember to click the save button after entering all the keys.

For each key entered there is a corresponding service endpoint. There are some default endpoints included you can use as an example, but when you copy each key also check and replace the service endpoint for each service you are using. You will find the service endpoint for each Cognitive Service on the overview page. Remember to click the save button after updating all the service endpoints.

Congratulations! You should now have a fully working application to get started. Have fun testing the project and thank you for your contribution!

You can find the code, solution development process, and all other details on GitHub.

Source: Azure Blog Feed