3) Detecting Photos Re-shot from Screens or Other Photos
In this type of fraud, a merchandiser photographs not an actual product display in a store but an image of the desired display (a photo of a photo). The merchandiser uses the same databases of “correct” product placement images as described in Section 1. Typically, the agent uses two devices—a personal one and a corporate one. They open the target display image on their personal device and then re-photograph it with the corporate device, uploading it into the system “for verification.”
While photos taken from screens were once easily recognizable, advancements in display clarity and smartphone camera quality have made detection much more challenging.
Another source of high-quality images can be printed merchandising guidelines (merch books) that showcase ideal product placements. In our experience, such cases were rare, but this method does exist—even if it seems somewhat absurd at first glance. Here, an employee photographs a printed image and uploads it instead of a real shelf photo. We’ve even encountered cases where merchandisers took screenshots from digital merch books and submitted those.
How We Tackle This Fraud
Our first attempts to train our system to recognize this type of fraud date back several years. At the time, screen-captured photos differed significantly from standard ones. Additionally, we worked with smaller clients, had lower photo processing volumes, and merchandisers were less inventive—making fraud detection relatively straightforward.
However, as technology improved, merchandisers grew more creative, and ShelfMatch™ clients became larger and more demanding. The old “brute-force” training approach no longer delivered the required high accuracy.
The main issue we faced with one of our largest clients was a high false-positive rate. While our neural network excelled at detecting manipulated images, it also frequently flagged legitimate photos as fraudulent. We found that our fake detection accuracy hovered around just 60%.
Since our client wanted not just to detect fraud but also to penalize negligent employees, achieving near-perfect accuracy was critical. While 100% precision is unattainable in real-world data, we needed to get as close as possible.
Our Solution
We researched academic literature and explored various approaches to improve detection.
After studying the problem, we supplemented our dataset with external datasets used in similar fields:
This expanded our data pool but did not significantly improve recognition quality—a major hurdle.
Research confirms that neural networks often struggle with domain shift (poor generalization to unseen data). Simply adding external datasets doesn’t help; the training data must closely resemble real-world production images.
Manual labeling of client-submitted photos was an option, but it was too slow to achieve the desired accuracy quickly.
The Breakthrough
We thought: When we lack training images of specific SKUs, we visit stores and take photos ourselves. Why not apply the same logic here?
So, we asked our team to play the role of “inventive merchandisers” and submit fake store display photos.
It worked!
- We added these self-generated images to our training dataset.
- Testing was conducted on real client photo streams.
Results:
- 90% fraud detection accuracy
- Only ~3% false positives
While we continue refining our system, our solution already helps clients detect fraud at scale.
“For every clever merchandiser, there’s an equally clever software developer!” 😊
Need Help Fighting Merchandising Fraud?
If digital fraud detection is a priority for your business, our experts are ready to assist. Contact us—we’ll respond promptly and tailor a solution to your needs.
In the next (final) article, we’ll cover other fraud methods in retail merchandising.