• Fraud in Digital Merchandising. Technological Counteraction. Part 1

    05.04.2024 ShelfMatch talks

    Cases of fraud in digital merchandising are frequent and, alas, inevitable. Field personnel – probably due to the nature of without regular supervision of a manager – are very good at creating new and original tricks to work less and earn more. Therefore, for companies that have merchandisers on staff, the issue of auditing and checking their employees is always relevant.

    In this article we will talk about the types of fraud in merchandising and our experience in how to technologically overcome the pitfalls of careless employees. In the 1st and 2nd, practical parts, we will dwell in more detail on how we work with certain types of fraud. And in the 3d part, the theoretical one, to complete the picture we will talk about other popular methods of deception on the part of field personnel.

    Fraud in digital merchandising: goals

    Fraud is an attempt to cheat the merchandising system in order to influence the result. Since field specialists engaged in taking photos of shelves and creating reports are often paid by the piece, manipulating photographs allows them to create the appearance of a larger number of points of sale visited and shelves captured. Hence, fraud occurs when a field specialist cheats the system to make more money.

    Another case is when a specialist’s fee depends not on the number of photographs taken, but on the quality of the work – that is, the KPIs achieved. In this case, the more high-quality photographs a person provides to the customer in terms of ideal shelf layout, the greater reward the person will receive.

    In this post we will tell how ShelfMatchTM works with fake photos, namely:

    1. Identifies and searches for duplicate photos
    2. Distinguishes and rejects poor quality photographs from the entire stream
    3. Identifies photographs taken from smartphone/tablet screen or print media

    1) Detecting duplicate photos

    Photo duplication is the most common type of fraud. The merchandiser uploads the same photo into the system with an interval of several days or weeks. This photo can be a real photo from a POS (point of sale), or it can be downloaded from a special database: merchandisers all over the world have secret chat groups for exchanging high-quality photographs of shelves, which contain comprehensive gallery of images which could be used in almost any situation. The point of this trick is not to pay a visit to the POS, but to send the supervisor an old, out-of-date photo of shelves with goods – as if the employee had visited the outlet. Consequently, all the merchandisers had amazing results at the end of the day.

    How ShelfMatchTM works with this type of fraud:

    On the one hand, finding duplicate images is a fairly well-known, understandable and solved problem for Machine Learning. An image processing service creates a digital fingerprint of the photo, adds it to the database, creates such a virtual fingerprint for each incoming image and compares each new fingerprint with the previous ones, thereby finding duplicate images.

    On the other hand, in our context this task is not as straightforward as it might seem at first glance. There are several difficulties that may make work difficult.

    First, what is considered a duplicate? It would seem that the answer is simple: exactly the same image of a shelf. But with this understanding of the duplicate, the system will be quite easy to deceive. For example, you can slightly crop the original image, even partially sacrifice some SKUs in the photo. And then, for computer vision, it will no longer be an absolute duplicate – although will be a duplicate, in terms of content. And this is the first difficulty that has to be taken into account when creating a system for finding duplicate images.

    Second, for what period should we search through the database of previous visits? This complexity lies in the plane of server capacity. It’s one thing when there is a certain image database of a fixed size. But it’s another matter when this database grows daily by several tens of thousands of pictures or more. As a result, you need to solve the issues of: a) physical space for storing fingerprints and b) speed of comparing each fingerprint with the entire database. You can periodically clear the database of old entries, but sooner or later merchandisers will get a chance to upload older images.

    Despite these nuances, ShelfMatchTM can protect you from persistent and obvious attempts to forge a photo.

    2) Detecting bad quality images

    It also happens that employees send poor quality photographs: overexposed, blurry, unclear images, photos taken at a random angle, or in which the entire rack is not visible, and other unusable files. All such photos make the automated image recognition impossible. The reason why employees take such photos is because of amount of data. It’s easier to take a lot of low-quality photos and send them to the supervisor than to try to take high-quality photos. And, of course, human factors always play a vital role: everyone can make mistakes and accidentally take a bad photo.

    How do we deal with such fraud?

    In ShelfMatchTM a special separate module is responsible for assessing the quality of photographs. Image processing occurs in 2 stages:

    1. pre-processing (before SKU recognition) and
    2. post-processing (after recognizing the SKU and price tags in the photo).

    1. Pre-processing

    In the preprocessing stage, the incoming shelf image passes through several neural networks that evaluate following parameters:

    • Shooting angle. In particular, the neural network looks for the angle of inclination of the shelves in the photograph. This is a fairly important parameter, since the quality of SKU recognition depends on the shooting angle. The more frontally the photo is taken, the more accurate the recognition results will be.
    • Presence of goods in the photo. Another neural network determines whether the photo contains images of goods or has other content. “Other” can mean a lot of options: in practice, we come across photographs from a car, photos of the entrance area of a store, images of the floor/ceiling sent by mistake, and similar irrelevant images.
    • Image quality, overall. A separate neural network determines the parameters in combination (“image clarity degree” + “exposure”) and counts a specific value: whether it is an image of poor or good quality.
    • Screen photo or real photo. We will talk about this version of fraud in more detail below.

    The results of preprocessing allow us to weed out poor quality photos and not to take them into recognition phase, and the merchandiser, in turn, can receive a quick request for a new, high-quality photo (if the work takes place online).

    2. Post-processing

    It often happens that not the whole photo is bad, but some part of it. ShelfMatchTM can evaluate such cases using post-processing.

    During the post-processing process, the quality of not the entire photo is measured, but separately each small piece of the image containing the SKU or price tag.

    During post-processing you can:
    • track which merchandisers take perfect or close to perfect photos, and which ones don’t try very hard
    • receive additional information about the accuracy/doubtfulness of the SKU and/or price tag recognition results in the specific location of the image.

    One way or another, in the field of automated product recognition there is a constant struggle for recognition accuracy. But the recognition accuracy often suffers for reasons beyond the control of neural networks: a large glare crept into the photo, or the hand trembled when taking a photo and part of the image was blurred, etc. In such cases, assessing the quality of the photo, in general, will not give anything. But post-processing and evaluation of individual areas will show that the results from such a photograph are not very accurate and they need to be at least partially omitted. Thus, we have another tool for identifying low-quality photographs, allowing us to increase the accuracy of the analysis.