Other Fraud Methods
In this part of the article, we will present theoretical facts: we will describe the main scenarios encountered by us, our clients and partners. Interestingly, merchandisers from different countries use typical fraud methods, so these scenarios can be grouped into several categories.

“It’s Not Me, It’s Him!”
Since field specialists’ work depends on gadgets, software, internet coverage and other force majeure circumstances, there is a strong temptation to shift responsibility for inability to work to external causes. The most common explanations sound like this:

“The app wouldn’t launch/work, so I couldn’t do my job”
“The software drained my phone’s battery and it died”
“Your app froze my phone”
“There was no internet in the store, I couldn’t send anything.”
What can be done:
Track the employee’s equipment status using special software (for example, what battery level they started their shift with). Monitor their actions in the app and the app’s behavior (whether it actually froze).

Location Spoofing
This method is quite old and was used by merchandisers even before retailers implemented image recognition technology. The idea was that specialists would report taking photos in one store while actually being in another. It’s worth noting that such deception was possible until vendors introduced employee geolocation tracking.

In the modern world, this scenario has undergone technological improvement: now field specialists spoof or reset (making real GPS coordinates disappear) their location using apps like “fake GPS” that deceive tracking systems. The motivation for using fake geolocation may be higher scores for visiting certain stores (for example, newly opened ones) or more convenient store locations for the employee.

What can be done:
According to our statistics, about 5% of merchandisers attempt to deceive employers using such programs. The optimal solution currently is continuous monitoring of employee geolocation. Supervisors can see the merchandiser’s movement route between stores with time stamps, thus tracking the movement map and time spent in each store. There is also an alert system where an employee leaving a store early receives a device notification. If they don’t return, a countdown timer starts, and the agent gets another notification that their visit may be marked as potentially fraudulent.

Fake Camera and Object Substitution
Third-party “fake camera” apps are widely available in app stores. These apps have interfaces very similar to real cameras but can’t actually take photos: instead they use fake images (in our case – others’ photos of displays) preloaded to the device storage. Thus, dishonest agents use “fake camera” apps to upload photos of perfect displays from shared databases. Without making any real effort, they get excellent “work” results, high scores and good compensation by the end of the day. Meanwhile, trade marketers remain completely unaware of the actual shelf situation.

What can be done:
Replace the system camera with proprietary software. Develop special algorithms to check image uniqueness. Ban duplicate images in the system.

Staged Photos
This fraud type could be called the most creative. Merchandisers, having material interest in placing certain SKUs (e.g., new products) in premium locations (“golden shelf”, checkout zone), temporarily place SKUs in required spots, photograph them, submit reports, then remove the products from premium placements.

What can be done:
This fraud type is hardest to track, as it’s difficult to programmatically detect photographing one product in multiple store zones, and impossible to prevent merchandisers from moving products.
However, practice shows reduced fraud of this type when changing incentive programs: removing bonuses for premium placements eliminates the motivation.

Low-Tech Fraud
Until now we’ve discussed technology-related fraud. But there’s also “low-tech” fraud, which is quite common. We know cases where an agency hires one employee and gives them two smartphones to do two people’s work, while charging clients for two specialists though only paying one.
Merchandisers themselves sometimes work multiple contracts simultaneously, devoting just a couple hours to each rather than a full workday. This can critically impact merchandising quality in stores.

What can be done:
When possible, prohibit multiple specialists from visiting one store. If this restriction isn’t feasible and clients need multiple reps in one store simultaneously, set other limits: for example, allow hypermarkets but prohibit smaller stores.

#fraud #shelfmatch #recognition #AI #tech

The retail landscape is at a crossroads. As consumer expectations soar and labor challenges persist, a critical question emerges: Can robots solve what humans simply can’t scale? The answer may lie in Nvidia’s groundbreaking robotics initiatives, reigniting a revolution that began years ago—but was derailed by an unexpected disruptor: the pandemic.

The Problem: Precision Meets Pressure

Retail’s Achilles’ heel has long been *inventory accuracy, operational efficiency, and 24/7 consistency. Humans excel at customer interaction, but repetitive tasks like restocking, shelf auditing, and real-time data collection often lead to errors, burnout, and rising costs. Enter *autonomous robots—designed to handle these tasks with surgical precision, freeing staff to focus on what truly matters: the customer.

The Pioneers: A Dream Deferred

Before COVID, companies like Walmart and Amazon led the charge. Walmart famously partnered with Bossa Nova Robotics, deploying shelf-scanning robots to track inventory in real time. These machines roamed aisles, identifying gaps and pricing errors, promising a leap toward the “perfect store.” But the dream hit a wall: Walmart abruptly ended the partnership in 2020, and Bossa Nova, reliant on the retail giant, shuttered operations shortly after.

Lessons learned?
•⁠ ⁠Scalability ≠ sustainability: Early robots struggled with ROI and adaptability.
•⁠ ⁠Over-reliance on single clients can be fatal.
•⁠ ⁠The pandemic accelerated retail’s pivot to survival mode, freezing ambitious tech experiments.

Amazon, meanwhile, doubled down on warehouse automation, proving that some robotics could thrive—but the path was far from smooth.

2025: The Perfect Storm for a Robotic Renaissance

Fast-forward to today. Nvidia’s AI-driven robotics platforms (think Omniverse simulations and Isaac robotics tools) are enabling smarter, faster, and more adaptable machines. These solutions learn from past failures: they’re cheaper, more flexible, and designed to integrate with—not replace—human workflows.

Paired with post-COVID realities—chronic labor gaps, demand for omnichannel agility, and tighter margins—the conditions for a retail robotics reboot have never been better.

Will This Finally Unlock the “Perfect Store”?

Imagine stores where robots:
– Auto-restock shelves before items run out.
– Generate real-time analytics to predict trends.
– Seamlessly blend with human teams to elevate service.

But the ghosts of Bossa Nova linger: Can this generation of robots avoid the pitfalls of their predecessors?

What do YOU think?

1.⁠ ⁠Is the “Walmart-Bossa Nova saga” a cautionary tale or a stepping stone for future tech?
2.⁠ ⁠Will AI-driven robots finally crack the code of the “perfect store,” or are we chasing a sci-fi fantasy?

P.S. If Philip K. Dick were alive today, he might write: “Do algorithms dream of optimized supply chains?”

RetailTech #Robotics #FutureOfWork #AI #Innovation

3) Detecting Photos Re-shot from Screens or Other Photos

In this type of fraud, a merchandiser photographs not an actual product display in a store but an image of the desired display (a photo of a photo). The merchandiser uses the same databases of “correct” product placement images as described in Section 1. Typically, the agent uses two devices—a personal one and a corporate one. They open the target display image on their personal device and then re-photograph it with the corporate device, uploading it into the system “for verification.”

While photos taken from screens were once easily recognizable, advancements in display clarity and smartphone camera quality have made detection much more challenging.

Another source of high-quality images can be printed merchandising guidelines (merch books) that showcase ideal product placements. In our experience, such cases were rare, but this method does exist—even if it seems somewhat absurd at first glance. Here, an employee photographs a printed image and uploads it instead of a real shelf photo. We’ve even encountered cases where merchandisers took screenshots from digital merch books and submitted those.

How We Tackle This Fraud

Our first attempts to train our system to recognize this type of fraud date back several years. At the time, screen-captured photos differed significantly from standard ones. Additionally, we worked with smaller clients, had lower photo processing volumes, and merchandisers were less inventive—making fraud detection relatively straightforward.

However, as technology improved, merchandisers grew more creative, and ShelfMatch™ clients became larger and more demanding. The old “brute-force” training approach no longer delivered the required high accuracy.

The main issue we faced with one of our largest clients was a high false-positive rate. While our neural network excelled at detecting manipulated images, it also frequently flagged legitimate photos as fraudulent. We found that our fake detection accuracy hovered around just 60%.

Since our client wanted not just to detect fraud but also to penalize negligent employees, achieving near-perfect accuracy was critical. While 100% precision is unattainable in real-world data, we needed to get as close as possible.

Our Solution

We researched academic literature and explored various approaches to improve detection.

After studying the problem, we supplemented our dataset with external datasets used in similar fields:

• Rewind Recapture Dataset
• MRGAN for Recaptured Image Detection

This expanded our data pool but did not significantly improve recognition quality—a major hurdle.

Research confirms that neural networks often struggle with domain shift (poor generalization to unseen data). Simply adding external datasets doesn’t help; the training data must closely resemble real-world production images.

Manual labeling of client-submitted photos was an option, but it was too slow to achieve the desired accuracy quickly.

The Breakthrough

We thought: When we lack training images of specific SKUs, we visit stores and take photos ourselves. Why not apply the same logic here?

So, we asked our team to play the role of “inventive merchandisers” and submit fake store display photos.

It worked!

• We added these self-generated images to our training dataset.
• Testing was conducted on real client photo streams.

Results:

• 90% fraud detection accuracy
• Only ~3% false positives

While we continue refining our system, our solution already helps clients detect fraud at scale.

“For every clever merchandiser, there’s an equally clever software developer!” 😊

Need Help Fighting Merchandising Fraud?

If digital fraud detection is a priority for your business, our experts are ready to assist. Contact us—we’ll respond promptly and tailor a solution to your needs.

In the next (final) article, we’ll cover other fraud methods in retail merchandising.

Cases of fraud in digital merchandising are frequent and, alas, inevitable. Field personnel – probably due to the nature of without regular supervision of a manager – are very good at creating new and original tricks to work less and earn more. Therefore, for companies that have merchandisers on staff, the issue of auditing and checking their employees is always relevant.

In this article we will talk about the types of fraud in merchandising and our experience in how to technologically overcome the pitfalls of careless employees. In the 1^st and 2^nd, practical parts, we will dwell in more detail on how we work with certain types of fraud. And in the 3^d part, the theoretical one, to complete the picture we will talk about other popular methods of deception on the part of field personnel.

Fraud in digital merchandising: goals

Fraud is an attempt to cheat the merchandising system in order to influence the result. Since field specialists engaged in taking photos of shelves and creating reports are often paid by the piece, manipulating photographs allows them to create the appearance of a larger number of points of sale visited and shelves captured. Hence, fraud occurs when a field specialist cheats the system to make more money.

Another case is when a specialist’s fee depends not on the number of photographs taken, but on the quality of the work – that is, the KPIs achieved. In this case, the more high-quality photographs a person provides to the customer in terms of ideal shelf layout, the greater reward the person will receive.

In this post we will tell how ShelfMatch^TM works with fake photos, namely:

1. Identifies and searches for duplicate photos
2. Distinguishes and rejects poor quality photographs from the entire stream
3. Identifies photographs taken from smartphone/tablet screen or print media

1) Detecting duplicate photos

Photo duplication is the most common type of fraud. The merchandiser uploads the same photo into the system with an interval of several days or weeks. This photo can be a real photo from a POS (point of sale), or it can be downloaded from a special database: merchandisers all over the world have secret chat groups for exchanging high-quality photographs of shelves, which contain comprehensive gallery of images which could be used in almost any situation. The point of this trick is not to pay a visit to the POS, but to send the supervisor an old, out-of-date photo of shelves with goods – as if the employee had visited the outlet. Consequently, all the merchandisers had amazing results at the end of the day.

How ShelfMatch^TM works with this type of fraud:

On the one hand, finding duplicate images is a fairly well-known, understandable and solved problem for Machine Learning. An image processing service creates a digital fingerprint of the photo, adds it to the database, creates such a virtual fingerprint for each incoming image and compares each new fingerprint with the previous ones, thereby finding duplicate images.

On the other hand, in our context this task is not as straightforward as it might seem at first glance. There are several difficulties that may make work difficult.

First, what is considered a duplicate? It would seem that the answer is simple: exactly the same image of a shelf. But with this understanding of the duplicate, the system will be quite easy to deceive. For example, you can slightly crop the original image, even partially sacrifice some SKUs in the photo. And then, for computer vision, it will no longer be an absolute duplicate – although will be a duplicate, in terms of content. And this is the first difficulty that has to be taken into account when creating a system for finding duplicate images.

Second, for what period should we search through the database of previous visits? This complexity lies in the plane of server capacity. It’s one thing when there is a certain image database of a fixed size. But it’s another matter when this database grows daily by several tens of thousands of pictures or more. As a result, you need to solve the issues of: a) physical space for storing fingerprints and b) speed of comparing each fingerprint with the entire database. You can periodically clear the database of old entries, but sooner or later merchandisers will get a chance to upload older images.

Despite these nuances, ShelfMatch^TM can protect you from persistent and obvious attempts to forge a photo.

2) Detecting bad quality images

It also happens that employees send poor quality photographs: overexposed, blurry, unclear images, photos taken at a random angle, or in which the entire rack is not visible, and other unusable files. All such photos make the automated image recognition impossible. The reason why employees take such photos is because of amount of data. It’s easier to take a lot of low-quality photos and send them to the supervisor than to try to take high-quality photos. And, of course, human factors always play a vital role: everyone can make mistakes and accidentally take a bad photo.

How do we deal with such fraud?

In ShelfMatch^TM a special separate module is responsible for assessing the quality of photographs. Image processing occurs in 2 stages:

1. pre-processing (before SKU recognition) and
2. post-processing (after recognizing the SKU and price tags in the photo).

1. Pre-processing

In the preprocessing stage, the incoming shelf image passes through several neural networks that evaluate following parameters:

• Shooting angle. In particular, the neural network looks for the angle of inclination of the shelves in the photograph. This is a fairly important parameter, since the quality of SKU recognition depends on the shooting angle. The more frontally the photo is taken, the more accurate the recognition results will be.
• Presence of goods in the photo. Another neural network determines whether the photo contains images of goods or has other content. “Other” can mean a lot of options: in practice, we come across photographs from a car, photos of the entrance area of a store, images of the floor/ceiling sent by mistake, and similar irrelevant images.
• Image quality, overall. A separate neural network determines the parameters in combination (“image clarity degree” + “exposure”) and counts a specific value: whether it is an image of poor or good quality.
• Screen photo or real photo. We will talk about this version of fraud in more detail below.

The results of preprocessing allow us to weed out poor quality photos and not to take them into recognition phase, and the merchandiser, in turn, can receive a quick request for a new, high-quality photo (if the work takes place online).

2. Post-processing

It often happens that not the whole photo is bad, but some part of it. ShelfMatch^TM can evaluate such cases using post-processing.

During the post-processing process, the quality of not the entire photo is measured, but separately each small piece of the image containing the SKU or price tag.

During post-processing you can:

• track which merchandisers take perfect or close to perfect photos, and which ones don’t try very hard
• receive additional information about the accuracy/doubtfulness of the SKU and/or price tag recognition results in the specific location of the image.

One way or another, in the field of automated product recognition there is a constant struggle for recognition accuracy. But the recognition accuracy often suffers for reasons beyond the control of neural networks: a large glare crept into the photo, or the hand trembled when taking a photo and part of the image was blurred, etc. In such cases, assessing the quality of the photo, in general, will not give anything. But post-processing and evaluation of individual areas will show that the results from such a photograph are not very accurate and they need to be at least partially omitted. Thus, we have another tool for identifying low-quality photographs, allowing us to increase the accuracy of the analysis.

Besides the question “How to grab the attention of customers and make them want to buy from me rather than from my rivals”, every day, managers, marketers, merchandisers and sales representatives are faced with a large number of other questions – regarding the display of goods on store shelves:

• Are my products placed on shelves uniformly and coherently across multiple stores in accordance with the planogram?
• Are products placed in the order demonstrated in the planogram?
• Are all product items (SKUs) present on the shelf?
• Are product labels facing the consumer?
• Are products surrounded by complimentary items to drive cross selling?
• What is my competitive environment?
• Are all my goods affixes with price tags?
• Has the retailer leveraged promotional shelf labels?
• Are signs, banners, and shelf dividers adequately visible?

And so on.

Timely and accurate answers on the questions above are extremely important to improve the inventory turnover, increase sales volume, skillfully manage items with irregular demand (e.g. seasonal products), and effectively manage contract obligations compliance.

Will you please count the number of SKUs in the photo above in a couple of seconds by eye? How about determining the share of shelf for each product category and comparing the data with a planogram – yep, with your own eyes, but accurately and quickly?

Here’s where Artificial Intelligence will help you out!

In 2,5 seconds AI will recognize and count the number of SKUs, share of shelf, conduct planogram compliance and detect inconsistencies of product layouts. An automated inventory management system can also help businesses keep track of their stock levels automatically with minimal human intervention. How long would it take your sales rep to manually calculate these metrics?

Computer vision in AI is the development of automated systems to interpret images in the same manner as people do. Computer vision technologies are extremely sophisticated and developing quickly. They are capable of coping with merchandising audit (retail execution) tasks not only much more quickly, than a human eye, but also more accurately than the human brain.

Artificial Intelligence and Retail image recognition are coming to store shelves, delivering almost instant access to data captured by field representatives and enabling stakeholders gain smart vital insights into what’s on their shelves and make better evidence-based decisions almost in the twinkling of an eye. 

To be continued.

In the last decade, the field of off-line retail has become increasingly dynamic. In conditions of fierce competition and pressure from on-line retail, there was a need to receive data in real time, new tools for working with data from retail outlets were required.
FMCG retail strives to implement the perfect store concept as efficiently as possible, solving the tasks of high-quality logistics construction, compliance with the standards of layout, creating a positive customer experience.

Solving these tasks requires tools for effective placement and control of goods on the shelves.

Shelfmath has developed a highly efficient solution that responds to retail requests. The Shelfmatch service perfectly recognizes products on the shelf. And the use of machine learning and AI allows you to get ready-made reports, plan logistics, modernize the salary fund, track the dynamics of sales growth, as well as set your own framework and get the necessary analytics.

The Shelfmatch service is a new window of opportunity for retail companies. The service provides valuable and high-quality data in real time, helps to improve the efficiency of product management and saves resources.