A new kind of team

Feb 24, 2017

Using Object Recognition and AI to improve UX Design

Through my role at Madeo, I get to explore the intersection of content, design, and technology. More specifically, I focus on how they intersect to improve user experiences on websites, within products, and through other digital experiences that we create.In this post, I wanted to share a number of accessible Object Recognition tools that rely on Artificial Intelligence (AI), and how they can help improve relevant user experiences.

For context, imagine the internet without any images for a day. Surfing the web would be more like looking through a graveyard of skeletal wireframes. Scrolling through social networks would would be an oddly Dadaist barrage of words with missing context. Buying a jacket online would be an exercise in trust that the product name and reviews fully capture the garment and its style.

This is all to say: Visual content is interwoven into almost every user experience. This might all seem obvious, but let’s not forget that things didn’t always used to be this way. Yes, there was a time when we didn’t have images at all. For example, watch this short video that shows every single New York Time front cover since the 1800’s. You can see how images work their way at the turn of the century — and they just keep adding more.



Of course, these trends aren’t developing in a technological vacuum. The technologies that made photography possible, both to take photos and to print them, go hand in hand with content creators using them more frequently. The same applies today; just as content creators are contributing to the rise of visual-first digital communication, AI researchers and engineers are rapidly building tools that allow people to interact with visuals in new and expansive ways.



How does this help UX designers?

The old adage “a picture is worth a thousand words” is more relevant than ever. Images are rich in subtle information, cultural context, and personal experience that people take in just by looking. As users expect more from visuals, we should learn more about the different tools that we can add to our UX toolkit.

For example, automated object recognition is an area with different tools that could be helpful; it is among the new capabilities brought about by the meteoric rise of artificial intelligence in the past few years. Computers can now effectively see and, increasingly, understand what they see. Computers can finally answer age-old questions like “is this dog wearing a hat?”

AI recognizing a dog in a hat

Harnessing the power of an artificial intelligence network might seem like something limited to massive tech companies that have equally massive data sets and expert resources to develop those networks.

However, anyone with a basic knowledge of API integration can start leveraging the capabilities of object recognition using services like Amazon Rekognition or Google Cloud Vision, and see if an AI solution can be a good fit for a particular UX challenge. Open-source machine intelligence libraries, like TensorFlow from Google, also provide easy access to start experimenting with AI.


New methods can help solve age-old UX problems

Consider a perennial UX challenge: Organizing and structuring a huge database of items for people to interact with and search. Even if a user knows exactly what they are searching for, they may not know how to manipulate a search query to find their desired outcome.

With the implementation of AI object recognition, users can actually show an interface exactly what they’re looking for – more like showing a photo to a friend than trying to crack a code. ClarifAI is one such tool that allows users to communicate to digital interfaces using images and receive visually similar search recommendations.

Gilt, of online fashion flash sale fame, is using a similar approach to incorporate deep learning into search options for clothing, and help people find different styles in much more sophisticated and natural ways. For example, to search for dresses that are appropriate for a certain occasion, and then be suggested similar dresses that are also appropriate.

Consider another UX challenge, often overlooked: making visually rich sites accessible to users with visual impairments. Most internet users who are blind or visually impaired use screen readers and similar tools to determine the contents of a webpage. If sites don’t include alt text for photos and images, the screen reader will just tell the user that there is a photo – not quite enough information for someone to understand the content or design.



A new development by Facebook, automatic alt text, uses object recognition artificial intelligence to automate the creation of photo descriptions so that screen readers can provide a more holistic user experience for blind users.


There are a lot of interesting challenges ahead

Object recognition is not entirely perfect, but what it can already do is promising to upend many of the conventional approaches and challenges that we face in UX design. Although these new tools largely function behind the scenes, their ability to smooth out and provide more natural-feeling interactions can add up. Image and object recognition algorithms allow a user’s experience to be more organic and, a little paradoxically, more human.

Based on the growth and rising adoption of AI in new tools, services, and business solutions, it’s a safe bet that AI-enhanced resources for UX design will be on the rise. The sooner we all start testing these solutions and talking about the opportunities they provide, the faster we can adapt and grow to provide our users with better experiences.


Written by Kevin Ackermann

Madeo Newsletter

We don't send mass emails very often, but when we do we make sure it's something interesting.