Buyers often choose goods and products based on appearance, especially in the ecommerce industry. AI-based visual search and ecommerce blended relatively recently. Yet, companies have already started to exploit visual search in e-commerce.
The Intelliarts AI team of engineers participated in the visual product recognition challenge. We acquired useful insights while building a high-performing ML model for suggesting similar goods and products based on their images. And we are willing to share these insights with you throughout the article, so follow us here.
In this post, you’ll deepen into some technical aspects of building visual search for ecommerce. Besides, as a C-level executive, you may also be interested in the concrete business advantages of investing in such technology as well as in reviewing examples of strong market players and their experience with visual search.
Visual search in e-commerce: Brief overview
Let’s start exploring how to use visual search in ecommerce. The first step is getting an insight into the technology and its modern-day market.
From a technical perspective, visual search is a functionality enabled by a combination of AI-driven algorithms. The resulting solution can be composed of the two following technologies:
- Object detection. It enables to locate objects of interest within a digital image and identify individual products based on visual attributes, i.e., color, texture, pattern, shape, etc.
- Visual search engine. This technology utilizes ML algorithms to match identified products with ones in a database and suggest related products based on detected visual similarities.
From an e-commerce business perspective, visual search technology is all about the user experience. It helps simplify the process of searching for a particular product, which increases the chances of a purchase. At the same time, visual search offers a form of entertainment as an extra benefit.
Visual search e-commerce market size
Now, let’s attempt to predict how visual search engines will impact e-commerce in the future. Estimated to surpass $14 billion by the end of 2023, the global visual search market will rise up to as much as $33 billion by 2028, according to historical data and forecasts. For the forecast period of 2021-2028, the Compound Annual Growth Rate (CAGR) of the market is predicted to be 17.5% respectively.
It brings to the conclusion that the interest to this relatively new technology is at a high level, and e-commerce businesses worldwide are likely to increasingly invest in visual AI in fashion ecommerce. Potentially, within the next decade, most e-commerce businesses will adopt visual search as part of their platforms’ functionality.
If the question “how visual search is transforming ecommerce” came across your mind, you are likely to be equally interested in exploring our machine learning guide for executives.
The benefits of visual search in e-commerce
There is a range of main business benefits related to adopting visual search technology. Examples include the following:
Improved search accuracy and efficiency
Search engines, including semantic ones, do a good job of helping customers find the products they are looking for. Yet, visual search technology raises the efficiency of the processes to a whole new level. With visual search, a customer can take a picture of the item they desire, and the ML-driven engine will identify matching or similar products without the need for an exact name or description of the item. Easier search for customers means a higher probability of a customer going for a purchase.
Reduce bounce rates
Higher bounce rates indicate that customers are likely uninterested in browsing the site for any reason or face difficulties making them abandon the site and go to another one. Since visual search as an additional e-commerce platform’s functionality can provide customers with relevant, accurate, and personalized search results, users are more likely to find products they are interested in. This way, customers are more engaged in browsing the platform.
Enhanced online shopping experience
Online shopping experience is all about finding the necessary products via the online platform. Visual search engines enable customers to browse a wide range of products based on the picture of the item they desire. It saves time and effort, once again making a customer more likely to make a purchase from you.
Check on the below video to find out how video search functionality may work:
Enhanced offline shopping experience
Fashion AI search for ecommerce is tightly connected even from the perspective of running an offline business, such as a clothing store. Customers can take pictures of items they see in an exhibition hall and then find them and similar products via the store’s online platform. It’s especially useful when the desired item is absent on-site in needed sizes. Alternatively, visual search engines may help customers find a range of products they want to ask consultants about or try on if available on-site.
The creators of the challenge strongly believe that image-based search can be particularly handy when a customer observes the desired product in real life, movies, or media.
Product discovery
Since visual search technology not only finds exact matches but also suggests similar items, it plays a crucial role in product discovery. It means that it can engage customers in considering additional items from the same or similar categories compared to the initial interest. Helping customers discover items that may not have been found overwise leads to additional sales for businesses.
Increased sales and revenue
It can be stated that visual search in e-commerce increases revenue and sales. Customers are more likely to find and purchase things they are interested in when they find desired products with ease and simultaneously receive customized search suggestions. Furthermore, visual search and the accurate results it provides help make more informed purchasing decisions leading to reduced returns rates.
You may also find it useful to discover more about computer vision technology applications across industries.
Challenges and considerations of visual search in e-сommerce
Before deepening any further, let’s get to know the challenges and considerations of visual search technology in e-commerce. The list includes the following aspects:
- Accuracy and relevance. Ensuring the search returns correct and pertinent product matches.
- Large datasets. Managing and updating vast and varied image databases efficiently.
- Speed and performance. Providing instantaneous search results without delays.
- User privacy. Addressing concerns about storing and analyzing user-submitted images.
- Adoption and usability. Ensuring users understand and adopt the new search feature effectively.
- Diverse product representation. Handling varying angles, lighting, and quality of product images for consistent results.
- Integration complexity. Seamlessly integrating visual search with existing e-commerce platforms and systems.
Handling these matters will significantly enhance the performance of your visual search solutions and increase the positive outcomes of such an intervention.
Metrics for tracking visual search success in e-commerce
The effect of incorporating visual search for retail industry players can be measured and assessed.
Business metrics
Now, let’s proceed with reviewing common business metrics typically utilized to evaluate the impact of integrating new technology.
- Conversion rate
This metric measures the percentage of visitors who made a desired action, such as making a purchase or filling out a form. The conversion rate is calculated as the number of conversions divided by the number of website visitors and multiplied by 100%.
- Click-through rate (CTR)
It’s a percentage of people who interacted with an ad and visited a website or landing page. CTR is calculated as the number of clicks divided by the number of ad impressions and multiplied by 100%.
- Basket size
This refers to the average number of items in a shopping cart of a customer. This measure indicates how well the functionality of a platform performs in offering complementary products.
- Bounce and exit rate
It’s the percentage of website visitors who left the website without taking any action. Bound and exit rate is calculated as the number of visitors who left the website divided by the total number of visitors and multiplied by 100%.
Visual search technology, when integrated into the website as an additional functionality, can have both direct and indirect impacts on the performance of the platform. This influence can be tracked by comparing the historical data for performance evaluated with the use of metrics to the performance after the integration of the visual search.
Technical metrics
Aside from business metrics of the success of visual search engines in an e-commerce project, there are technical ones. They help assess the performance of individually trained models for solving visual search tasks. Examples include:
- Accuracy
It’s the percentage of correct predictions. Accuracy is calculated as the number of true positive and true negative predictions divided by the total number of predictions and multiplied by 100%.
- Precision
It’s the estimation of how reliable a prediction for a particular class, i.e., prediction output, is. Precision for class 1 is calculated as the number of true positive predictions divided by the number of positive predictions and multiplied by 100%. Precision for class 2 is calculated in the same way but with negative values.
- Recall
It’s the estimation of how well the model can predict a class. Recall for class 1 is calculated as the number of true positive predictions divided by the number of true positive and false negative predictions and multiplied by 100%. Recall for class 2 is calculated in the same way but with true negative and false positive values.
- Mean average precision (mAP)
This metric is the evaluation of a model’s general precision. MAP is calculated as the sum of average precision values for each class divided by the number of classes.
In the visual search competition, the Intelliarts team of engineers used exactly this metric. It was so because our model was intended to find the 1000 most similar products based on a digital reference. Unlike the other three metrics, mAP suited the evaluation of such search results perfectly.
The representation of what’s understood under predicted and actual, as well as positive and negative values in the above formulas, is known as Confusion Matrix. It’s shown in the image below.
Implementing visual search in your e-commerce store
Merely being aware of the technology is not enough to have your innovative e-commerce store up and running. Here are some best practices for integrating the visual search functionality:
- Dataset collection: Gather large, diverse sets of product images you need the model to be able to search among, and categorize them with accurate tags and metadata and label them.
- Select or build a visual search engine: Choose from existing visual search platforms (like Google Cloud Vision or Clarifai) or build custom models using ML frameworks.
User interface integration. Incorporate a camera or upload feature on your website/app allowing customers to input their visual queries. - Optimize for mobile: Ensure visual search is seamless on mobile devices, considering the higher camera usage by mobile shoppers.
- Educate customers: Promote and demonstrate the visual search feature to customers through tutorials, demos, and marketing campaigns.
By exploiting these tips, you can integrate a visual search functionality into your e-commerce store. Should you require technical assistance, don’t hesitate to reach out to a provider of ML solutions.
Real examples of visual search for e-commerce
Well-established businesses that operate in e-commerce have already adopted visual search functionality. Moreover, they have been using it successfully for years. Among such are the following companies:
- ASOS
This British online fashion and cosmetic retailer has been operating in the e-commerce industry since 2000. It offers a “Style Match” feature, which is exactly an implementation of the visual search technology on the ASOS platform.
Style Match functionality offers to take a picture of an object of interest using the phone’s camera or upload a picture from the photo library to run the search process. Then, the ML algorithms of the visual search engine and the IT infrastructure behind it will do the rest. ASOS claims that they don’t store the image, they just analyze the information from the visual, i.e., color, patterns and type of clothing, etc., to find a match and make personalized recommendations to customers.
- Alibaba
Alibaba is a Chinese multinational company specializing in e-commerce. It’s regarded to be one of the fastest-growing retail platforms in the world. Alibaba connected manufacturers and wholesalers with retailers and individual customers, and the range of products traded via its website is vast.
Alibaba offers the “Image Search” functionality, which provides the same conventional capabilities to find product items based on uploaded digital imagery. As a complementary feature, the Image Search engine allows uploading a link to an image to run the search process effortlessly.
- Amazon
Amazon is an American multinational technology company. Its mission is to be Earth’s most customer-centric company. Amazon exploits innovative technologies in its IT environment extensively. As such, 1-click shopping, personalized recommendations, and other industry-latest technology advancements.
In 2019, Amazon introduced the visual search functionality packed into the feature called “StyleSnap,” available through the Alexa app. As usual, the technology enables users to get the most relevant products to their search by simply uploading a photo and letting artificial intelligence perform the search.
- BooHoo
Boohoo is a UK-based online fashion retailer founded in 2006. It became a popular online store offering affordable and trendy clothing and having a presence in over 100 countries. BooHoo is considered to be one of the largest fast-fashion retailers in the world.
The visual search functionality that the BooHoo platform offers to its users is “Camera Search.” It was introduced back in 2017 with the goal of enhancing customer experience and, potentially, increasing the revenue from the business operation. The technology supports searching based on screenshots or pictures taken in real-time with the help of a phone’s camera.
Google is a global technology company founded in 1998 and headquartered in Mountain View, California. It has become the world’s leading search engine, offering a wide range of internet services and products, including online advertising, cloud computing, software, and hardware. With its presence in over 190 countries, Google is considered one of the most influential and innovative companies in the world.
Circle to Search, introduced by Google in 2024, is a visual search feature designed to enhance user experience on Android smartphones. It allows users to search for information directly from their current screen by using simple gestures like circling, highlighting, or scribbling on content. This innovative tool aims to provide seamless access to information without the need to switch between apps, making the search process more intuitive and integrated.
You can discover more about Circle to Search functionality by Google from the video below:
The listed companies are well-established businesses that made a fortune by pioneering innovations. It’s safe to say that integration of the visual search technology in their platform played an essential role in their success.
Technical visual search requirements and principles
A visual search functionality is a complex solution. Discover what components required for its creation are and what they serve for from the infographics below.
In the visual search competition, the creation of the ML solution followed exactly the detailed pattern. We had the database of product images separated into user and seller photos. The latter was our ground truth. At the same time, user photos, which were typically snapshots of products taken with a phone camera in cluttered scenes, were used for testing the ML model.
The role of AI and machine learning in e-commerce visual search
Now, let’s review some of the best practices for improving the publication of images via an e-commerce platform and optimizing a visual search solution:
- Alt text
It’s important to include brief information about a product shown in an image. It helps search engines understand the context of an image which results in a higher likelihood of finding the necessary product.
- Numerous images
Include multiple images of the same product. It doesn’t only allow customers to have a better overview of a product but also increases the chances of a search engine finding the item.
- Keyword usage
Descriptions should contain several long-tail keywords to help the image appear in search results.
- Product detection first
It’s recommended to have product detection as the first step of image processing with an artificial intelligence visual search solution. It ensures that no background elements will be treated as objects of interest, which might interfere with the accurate matching.
- Visual transformers (ViTs)
It’s advised to ground a visual search solution on the ViT models family and utilize a Contrastive Language-Image Pretraining (CLIP) network. The latter can predict an accurate text snippet based on an image. Usage of the technology can give a huge boost to the accuracy and speed of search for a resulting solution.
The team of ML engineers of Intelliarts could see the high usefulness of these practices throughout the course of our projects.
Our experience with visual search
While working on a visual search for retail, we tried multiple approaches as a part of the AI competition. We started with Convolutional Neural Network (CNN) architectures as classical neural networks to build an ML solution on. Yet, we came to two important conclusions in the process:
- Insight 1. ViT models family shows huge outperformance over classical CNN architectures.
- Insight 2. It’s crucial to use re-ranking techniques when building visual search solutions. Evaluating search outcomes and moving more similar objects higher in the list of results adds to the usefulness of the solution overall.
You may give a try to the online demo that presents an interactive playground for the trained AI model. The demo shows the capabilities of a visual search engine to suggest goods and products available on the web based on an input image or video frame.
Final take
Visual search is a trending technology in the e-commerce niche. Such large market players as ASOS, Amazon, Alibaba, and BooHoo already offer visual search capabilities to the users of their platforms. The enhanced shopping experience, lower bounce rates, and higher revenue from business operations are only a few of the advantages the technology may offer.
Implementing an e-commerce visual search solution requires high expertise and experience. As a C-level executive, you may consider contacting a trusted machine learning development firm. This way, you will get professional help with building and integrating the visual search functionality into your platform, which may result in numerous business benefits for your company.
FAQ
With visual search technology, buyers can upload a picture of an item of their interest into an engine and find it via a particular e-commerce site easily. Combined with smart recommendations, visual search allows for a smart, high-level shopping experience, which likely improves the chances of making a purchase.
Visual search technology has gained popularity among buyers and impressive usage among e-commerce businesses. Besides, engines will naturally become more accurate and efficient. So, it can be concluded that the future of visual search looks promising.
The process of building a fully-fledged vidual search engine consists of the following stages: collecting and labeling data, choosing an algorithm and training a model, testing and optimizing the model, developing a user-friendly interface, and deploying the solution.
