Cases of fraud in digital merchandising are frequent and, alas, inevitable. Field personnel – probably due to the nature of without regular supervision of a manager – are very good at creating new and original tricks to work less and earn more. Therefore, for companies that have merchandisers on staff, the issue of auditing and checking their employees is always relevant.

In this article we will talk about the types of fraud in merchandising and our experience in how to technologically overcome the pitfalls of careless employees. In the 1^st and 2^nd, practical parts, we will dwell in more detail on how we work with certain types of fraud. And in the 3^d part, the theoretical one, to complete the picture we will talk about other popular methods of deception on the part of field personnel.

Fraud in digital merchandising: goals

Fraud is an attempt to cheat the merchandising system in order to influence the result. Since field specialists engaged in taking photos of shelves and creating reports are often paid by the piece, manipulating photographs allows them to create the appearance of a larger number of points of sale visited and shelves captured. Hence, fraud occurs when a field specialist cheats the system to make more money.

Another case is when a specialist’s fee depends not on the number of photographs taken, but on the quality of the work – that is, the KPIs achieved. In this case, the more high-quality photographs a person provides to the customer in terms of ideal shelf layout, the greater reward the person will receive.

In this post we will tell how ShelfMatch^TM works with fake photos, namely:

1. Identifies and searches for duplicate photos
2. Distinguishes and rejects poor quality photographs from the entire stream
3. Identifies photographs taken from smartphone/tablet screen or print media

1) Detecting duplicate photos

Photo duplication is the most common type of fraud. The merchandiser uploads the same photo into the system with an interval of several days or weeks. This photo can be a real photo from a POS (point of sale), or it can be downloaded from a special database: merchandisers all over the world have secret chat groups for exchanging high-quality photographs of shelves, which contain comprehensive gallery of images which could be used in almost any situation. The point of this trick is not to pay a visit to the POS, but to send the supervisor an old, out-of-date photo of shelves with goods – as if the employee had visited the outlet. Consequently, all the merchandisers had amazing results at the end of the day.

How ShelfMatch^TM works with this type of fraud:

On the one hand, finding duplicate images is a fairly well-known, understandable and solved problem for Machine Learning. An image processing service creates a digital fingerprint of the photo, adds it to the database, creates such a virtual fingerprint for each incoming image and compares each new fingerprint with the previous ones, thereby finding duplicate images.

On the other hand, in our context this task is not as straightforward as it might seem at first glance. There are several difficulties that may make work difficult.

First, what is considered a duplicate? It would seem that the answer is simple: exactly the same image of a shelf. But with this understanding of the duplicate, the system will be quite easy to deceive. For example, you can slightly crop the original image, even partially sacrifice some SKUs in the photo. And then, for computer vision, it will no longer be an absolute duplicate – although will be a duplicate, in terms of content. And this is the first difficulty that has to be taken into account when creating a system for finding duplicate images.

Second, for what period should we search through the database of previous visits? This complexity lies in the plane of server capacity. It’s one thing when there is a certain image database of a fixed size. But it’s another matter when this database grows daily by several tens of thousands of pictures or more. As a result, you need to solve the issues of: a) physical space for storing fingerprints and b) speed of comparing each fingerprint with the entire database. You can periodically clear the database of old entries, but sooner or later merchandisers will get a chance to upload older images.

Despite these nuances, ShelfMatch^TM can protect you from persistent and obvious attempts to forge a photo.

2) Detecting bad quality images

It also happens that employees send poor quality photographs: overexposed, blurry, unclear images, photos taken at a random angle, or in which the entire rack is not visible, and other unusable files. All such photos make the automated image recognition impossible. The reason why employees take such photos is because of amount of data. It’s easier to take a lot of low-quality photos and send them to the supervisor than to try to take high-quality photos. And, of course, human factors always play a vital role: everyone can make mistakes and accidentally take a bad photo.

How do we deal with such fraud?

In ShelfMatch^TM a special separate module is responsible for assessing the quality of photographs. Image processing occurs in 2 stages:

1. pre-processing (before SKU recognition) and
2. post-processing (after recognizing the SKU and price tags in the photo).

1. Pre-processing

In the preprocessing stage, the incoming shelf image passes through several neural networks that evaluate following parameters:

• Shooting angle. In particular, the neural network looks for the angle of inclination of the shelves in the photograph. This is a fairly important parameter, since the quality of SKU recognition depends on the shooting angle. The more frontally the photo is taken, the more accurate the recognition results will be.
• Presence of goods in the photo. Another neural network determines whether the photo contains images of goods or has other content. “Other” can mean a lot of options: in practice, we come across photographs from a car, photos of the entrance area of a store, images of the floor/ceiling sent by mistake, and similar irrelevant images.
• Image quality, overall. A separate neural network determines the parameters in combination (“image clarity degree” + “exposure”) and counts a specific value: whether it is an image of poor or good quality.
• Screen photo or real photo. We will talk about this version of fraud in more detail below.

The results of preprocessing allow us to weed out poor quality photos and not to take them into recognition phase, and the merchandiser, in turn, can receive a quick request for a new, high-quality photo (if the work takes place online).

2. Post-processing

It often happens that not the whole photo is bad, but some part of it. ShelfMatch^TM can evaluate such cases using post-processing.

During the post-processing process, the quality of not the entire photo is measured, but separately each small piece of the image containing the SKU or price tag.

During post-processing you can:

• track which merchandisers take perfect or close to perfect photos, and which ones don’t try very hard
• receive additional information about the accuracy/doubtfulness of the SKU and/or price tag recognition results in the specific location of the image.

One way or another, in the field of automated product recognition there is a constant struggle for recognition accuracy. But the recognition accuracy often suffers for reasons beyond the control of neural networks: a large glare crept into the photo, or the hand trembled when taking a photo and part of the image was blurred, etc. In such cases, assessing the quality of the photo, in general, will not give anything. But post-processing and evaluation of individual areas will show that the results from such a photograph are not very accurate and they need to be at least partially omitted. Thus, we have another tool for identifying low-quality photographs, allowing us to increase the accuracy of the analysis.

Besides the question “How to grab the attention of customers and make them want to buy from me rather than from my rivals”, every day, managers, marketers, merchandisers and sales representatives are faced with a large number of other questions – regarding the display of goods on store shelves:

• Are my products placed on shelves uniformly and coherently across multiple stores in accordance with the planogram?
• Are products placed in the order demonstrated in the planogram?
• Are all product items (SKUs) present on the shelf?
• Are product labels facing the consumer?
• Are products surrounded by complimentary items to drive cross selling?
• What is my competitive environment?
• Are all my goods affixes with price tags?
• Has the retailer leveraged promotional shelf labels?
• Are signs, banners, and shelf dividers adequately visible?

And so on.

Timely and accurate answers on the questions above are extremely important to improve the inventory turnover, increase sales volume, skillfully manage items with irregular demand (e.g. seasonal products), and effectively manage contract obligations compliance.

Will you please count the number of SKUs in the photo above in a couple of seconds by eye? How about determining the share of shelf for each product category and comparing the data with a planogram – yep, with your own eyes, but accurately and quickly?

Here’s where Artificial Intelligence will help you out!

In 2,5 seconds AI will recognize and count the number of SKUs, share of shelf, conduct planogram compliance and detect inconsistencies of product layouts. An automated inventory management system can also help businesses keep track of their stock levels automatically with minimal human intervention. How long would it take your sales rep to manually calculate these metrics?

Computer vision in AI is the development of automated systems to interpret images in the same manner as people do. Computer vision technologies are extremely sophisticated and developing quickly. They are capable of coping with merchandising audit (retail execution) tasks not only much more quickly, than a human eye, but also more accurately than the human brain.

Artificial Intelligence and Retail image recognition are coming to store shelves, delivering almost instant access to data captured by field representatives and enabling stakeholders gain smart vital insights into what’s on their shelves and make better evidence-based decisions almost in the twinkling of an eye. 

To be continued.