“Snip Insights” – An open source cross-platform AI tool for intelligent screen capture

This post was written by Tara Shankar Jana, Senior Technical Product Marketing Manager at Microsoft.

What if we could incorporate AI into everyday tools to make everyday users happy? With just a little creativity – and the power of the Microsoft AI platform behind us – it's now easier than ever to build AI-powered apps that can delight users.

Introducing Snip Insights!

An open source cross-platform AI tool for intelligent screen capture, Snip Insights is a step-change in how users can generate insights from their screen captures. An early prototype of Snip Insights was built for Windows OS and released at Microsoft Build 2018 in May. Microsoft Garage Interns Based out of Vancouver, Canada.

Our team at Microsoft AI LabSupported by Microsoft AICTO The team is taking Snip Insights further by giving the tool an intuitive new user interface, adding cross-platform support (for macOS, Linux, and Windows), and offering free download and use under an MSA license. Took to the level.

Snip Insights taps into Microsoft Azure Cognitive Services APIs and helps increase user productivity by automatically providing intelligent insights on their screen captures.

Solution overview

Snip Insights taps into cloud AI services and – depending on the screen-captured image – can turn it into translated text, automatically detect and tag images, and provide smart image suggestions. can improve the user's workflow. This simple act of connecting a familiar everyday desktop tool with Azure Cognitive Services has helped us create a one-stop shop for image insights.

For example, imagine you've scanned a textbook or a work report. Instead of manually typing the information it contains, cutting it will now give you editable text, thanks to the power of OCR. Or maybe you're scrolling through your social media feed and come across someone wearing a cool pair of shoes – now you can snag them to find out where to buy them. Snip Insights can also help you identify famous people and famous places.

In the past, you would have to take a screenshot, save the image, upload it to an image search engine, and then derive your results and insights from there.

That's pretty clever, isn't it?

key skills.

1. Celebrity Search: Snap a picture of a celebrity and the tool will provide you with relevant information about them.

   

2. Object detection and Bing visual search: You dig that t-shirt your friend is wearing in their latest social media post and want to know where you can buy it. No problem! Just use Snip Insights and you can see matching product images and where to buy them – all in seconds!

   

3. OCR, language translation and cross-platform support: You find a quote or phrase in English and want to convert it to French or another language. Just use Snip Insights and you can do it easily. What's more, this tool is free and works on Windows, Linux and MacOS, so everyone is covered!

Supported platforms

Snip Insights is available on three platforms:

• Universal Windows Platform (UWP)
• Mac OS
• Linux

Xamarin

Xamarin.Forms enables you to build native UIs for iOS, Android, macOS, Linux, and Windows from a common code base.

You can dive into app development with Xamarin.Forms by following us. Free self-guided learning from Xamarin University. Xamarin.Forms has preview support for GTK# apps. GTK# is a graphical user interface toolkit that combines the GTK+ toolkit and a variety of GNOME libraries, allowing the development of fully native GNONE graphics apps using Mono and .NET. Learn more here: Xamarin.Forms GTK#.

Installation

Windows

1. Download the zip from Here.
2. Install the certificate (“cer” file) according to the instructions in Install the certificate. Section
3. Install Snip Insights by double-clicking the .appx package file.

Linux

1. Install Mono by following the appropriate steps based on your Linux distro, https://www.mono-project.com/download/stable/#download-lin.
2. Install the .deb package from Here.
3. Launch the app from the Applications section.

Mac OS

1. Download and install Mono (Stable Channel): https://www.mono-project.com/download/stable/. It includes GTK#, the UI toolkit that Xamarin.Forms relies on for this project.
2. Install .pckg as a normal macOS application. Here.
3. The Snip Insights app is available in the Applications section on macOS.

needs

Using your own subscription

To add keys to Snip Insights, a Microsoft Garage project, start the application. Once running, click/tap the Settings icon in the toolbar. Scroll down until you find the “Cognitive Services, Enable AI assistance” toggle, and toggle it to the On position. You should now see the Insight Service Keys section.

• Entity Search: Create a new entity search knowledge service. Once created, you can display the keys. Select one and paste in “Settings”.
• Image Analysis: In Azure, create a **Computer Vision API** cognitive service and use its key.
• Image Search: In Azure, create a Bing Search v7 API Cognitive Service and use its key.
• Text recognition: You can use the same key used in image analysis above. Image analysis and text recognition both use computer vision APIs.
• Translator: Use the Translator Text API knowledge service.
• Content Moderator: Use the Content Moderator API cognitive service.

For a LUIS app ID and key, you will need to create a Language Understanding application in the Language Understanding portal. https://www.luis.ai. Use the following steps to create your LUIS app and retrieve the app ID:

• Click the Create New App button.
• Provide the app name. Leave the Culture (English) and Description as defaults.
• Click Done.
• In the left navigation pane, click Entities.
• Click Manage pre-built entities.
• Select DateTime v2 and Email.
• Click Done.
• Click the Train button at the top of the page.
• Click the Publish tab.
• Click the Publish button on the production slot.
• At the bottom of the screen, you'll see a list with a key string field. Click the Copy button and paste this key value into the LUIS Key field in the Snip Insights settings.
• Click the Settings tab (at the top).
• Copy the displayed application ID and paste it into the LUIS App Id field in the Snip Insights settings.

Now you can paste each key in the settings panel of the application. Remember to click the Save button after entering all the keys.

Note: Each key entered has a corresponding service endpoint. There are some default endpoints included (you can use these as examples) but as you copy each key, also check and change the service endpoint for each service you are using. You will see the service endpoint for each knowledge service on the overview page. Remember to click the Save button after updating all service endpoints.

Abstract

If you've made it this far, and followed the steps above, you'll have a fully functional application to start with. Congratulations! We hope you enjoy testing the project and thanks in advance for your support! You can search on code, solution development process and other details. GitHub.

We hope this post inspires you to get started with AI today, and inspires you to become an AI developer.

Tara