Amazon’s New Grocery Stores Providing a Radically New Checkout-less Experience
Last December, Amazon announced their foray into physical grocery stores, Amazon Go, with the disruptive twist of a ‘checkout-less shopping’ experience.1 Shoppers check into the store by tapping their smart phone on the turnstile as they enter. After that, they simply shop, putting whatever they want into their bag or basket and walk out of the store. While they are shopping, cameras and sensors2 throughout the store keep track of exactly which items the shopper picks up and keeps, and which ones they put back. Once the shopper leaves the store, the shopping app automatically totals up their bill, makes the payment, and sends the shopper a receipt, similar to the way Lyft or Uber automatically pays and sends a receipt once the rider is dropped off at the destination, without the rider having to do anything else.
As of this writing, Amazon has just one Amazon Go store and it is only open to Amazon employees. No doubt, this is sort of a beta test bed with a friendly audience to make sure performance is up to snuff before opening it up to a more demanding general public, who will be much less tolerant of mistakes (except perhaps being undercharged). Their website says it will be open to the public in ‘early 2017’ (which I interpret as ‘maybe by August … if we’re lucky’).
Deep Learning Analytics Has Come a Long Way
This really is a testament to how far deep-learning video analytics has come. We (and a lot of others I’ve talked to) were surprised that we were this close to such an advanced application of video analytics. For this kind of application, 80% or 90% accuracy is not good enough. I don’t know the threshold of accuracy (how many ‘nines’3 required) that Amazon has set for themselves, but it has to be high for ensuring they almost never charge people for something they didn’t buy and that all items taken are accurately identified and counted. We suspect that many elements of the store have been tuned to make this high level of recognition accuracy possible, including the lighting, selection of specific products and packaging that make them easier to recognize, layout of the shelves to provide good visibility, and so forth. The use of sensors (speculatively, perhaps scales on the shelves, for example) no doubt help improve the accuracy of the algorithms.
The reaction of competitors has been immediate and intense. This is not just about the radical new customer experience, but also about Amazon’s rumored intent to open 2,000 grocery stores, potentially challenging everyone from Walmart, Kroger’s, and Costco on down. So, grocers are scrambling to respond. And more broadly, retailers who previously assumed checkout-less shopping was a goal for a decade from now, are now considering how to get there quicker. At the NRF ‘Big Show’ this year, there were a couple of startups (Everseen and Focal) that claim to have technology that can give retailers an Amazon-Go-like capability now — or at least this year. See below for a bit more on them.
The Broader Picture — The Rise of the Intelligent Integrated Store
The technology behind Amazon Go is part of a broader and much older trend of instrumenting4 retail stores — i.e. putting various sensors, measuring devices, and feedback/control mechanisms in the store. One could argue that EAS devices from the ‘60s, and barcode readers and video surveillance in the ‘70s, were early examples of store sensor systems. More recently we’ve seen the adoption of tracking counters, RFID readers, Bluetooth LE, and more. Increasingly, intelligence/analytics are being added to this mix, and retailers are experimenting with integrating multiple sensor streams together to provide a higher level of intelligence about what is happening in the store. The abstract for The Intelligent Integrated Store, described it well:
There has been much discussion about marrying continuous overhead RFID/RTLS locating capabilities, video analytics, BLE, traffic counters, EAS, POS data, receiving data, and other store-based sensors to create an integrated, intelligent, self-aware store. In this vision, retailers have highly granular insights into the movements, actions, and intentions of shoppers in their stores, analogous to how they can analyze visitors’ clickstreams on their websites today. Store associates could be alerted the moment a shopper needs attention or an item needs to be restocked. Merchants and store planners would know exactly how shoppers are buying, which items are being tried on but put back on the shelf, the physical paths taken through the store, and so forth. Shoppers can be presented with a personalized context-relevant digital-physical experience.
This broader vision includes the checkout-less experience of Amazon Go and more. No doubt, the Amazon Go announcement has given more impetus for retailers to pursue this vision. Here we describe some of the solutions we saw at NRF in this arena.
To jump to a particular company, click the link below:
Everseen — From Detecting Non-Scans at the Point-of-Sale to Providing Amazon-Go-Like Capabilities for Other Retailers
Everseen is an Ireland-based startup, using video analytics for loss-prevention by detecting ‘irregular activity’ at the point of sale(POS). The irregular activity they detect includes non-scans (item taken out of the cart and possibly waved near the scanner, but not actually scanned), ‘basket-based loss’ (items simply left in the shopping cart, thereby not being scanned), overcharges, and irregular voiding and refunds (cashier taking advantage of abandoned products already paid for by the customer). Their deep learning5 algorithms have been trained over the past five years by analyzing tens of thousands of hours of video of checkouts, involving millions of employees and shoppers.
They told me that their system will alert the cashier within three seconds any time a product wasn’t scanned. As they put it to me, by analyzing so much data, Everseen has “learned the DNA of a transaction and of a non-scan event.” They are working with five of the top ten global retailers.
But Everseen is not stopping with loss prevention. They are working with a large retailer on providing a checkout-less experience, similar to Amazon Go, but with origins pre-dating the Amazon Go announcement. Everseen filed a patent for a virtual manager system in 2012 that contains the blueprint for their attendant-less and checkout-free system. To bring it into fruition, they are starting with a simple use case and format — a very small convenience store. They are working on optimizing the layout of the store, using a simplified selection of products, and selecting the right packaging, to ensure the most reliable tracking of purchases. This experience will allow Everseen to work out the kinks and eventually offer other retailers a way to provide an Amazon-Go-like experience, at least for certain store formats with suitable product mixes.
It took Amazon Go to shake up the industry. But larger retailers are unlikely to partner with Amazon to introduce this type of technology into their stores (assuming Amazon even makes that option available to them). The key for retailers will be finding retailer-agnostic partners who have a track record of working with high quantities of sensitive POS and customer data. As a result, more retailers are talking to Everseen, which will be good for them (Everseen), even if ultimately, many of those retailers decide to start with the loss prevention solution as an evolutionary step, before taking the more radical leap to a checkout-less store.
Focal Systems — Shopping Carts Tracking Shelf Inventory — and What Goes in and Out of the Cart
Francois Chaubard and Adriano Quiroga met in a graduate AI class at Stanford, where they teamed up to build a sparse SLAM (Simultaneous Localization and Mapping)6 system using deep reinforcement learning. In contemplating what business problem this technology could solve, they thought “people have created systems that can drive a car, but no one has built a system that can figure out which aisle and location the Clorox is in a grocery store.” So, they duct-taped an Android phone to the handlebar of a grocery cart, put a mirror to point the cameras at 90 degrees, so it could tell where you were at all times, and trained their SLAM system to build up a map of the store and, at the same time, locate items as the cart travelled up and down the aisles. A VP at Kellogg saw it, was impressed, and said if it could also tell him what was out of stock, it would be extremely valuable. So, Francois and Adriano started Focal Systems with an initial investment from Y Combinator, raising further funds in B and C rounds.
Their system attaches to the handlebar of shopping carts, not unlike other cart-mounted personal shopping assistants, except this one has two cameras on it: one pointing to the side, and one straight ahead viewing the top edge of the shopping cart (i.e. the plane through which all items must pass going into or out of the cart). The side camera can identify where items are located on the shelves and which items are out of stock (OOS). It alerts store associates about items that need restocking, thereby reducing loss of sale. It could also be used to validate planogram compliance, locate misplaced items (such as when a customer changes their mind), and provide the shopper with basket- and location-relevant suggestions. Francois described the system as “impressively accurate.” In many cases where they thought the system had made an error, it turned out their system was correct, and that it was actually human error or mistakes in other systems. He believes their accuracy is significantly greater than other image-based retail inventory management systems on the market.
Focal Systems has twelve different retailers trying out their solution. These trials have achieved meaningful reduction in out of stocks. The system also gives recommendations to the shopper, based on what they have purchased and where they are in the store (the shopper’s location being a potential indicator of intent to buy something, even if it’s not in the basket yet). They can potentially influence the shopper to buy more and/or different products. They have found that around 61% of shoppers interact with the tablet in some way; 41% end up clicking on offered deals (by comparison, the average click through rate is 0.35% for a web display ad and 1.9% for a search ad). Around 6%-8% of shoppers end up actually buying from offers made on the Focal Systems device.
Among several patents they filed, there is one that keeps track of what items are put into and out of the shopping cart, adds or subtracts those items from a shopping cart tally, and then enables instant checkout once the shopper is ready to leave. So, while their initial focus had been on reducing out-of-stock rates and providing effective upsell/cross-sell advertising, once the Amazon Go announcement was made, retailers started asking about that checkout-less capability. As with Everseen, the checkout-less capabilities may be what draws prospects to look at Focal, but they still might implement the out-of-stock and advertising/recommendations solution first. Either way, the added attention generated by Amazon Go should be a plus for Focal.
Zebra — Fusing Different Store Sensor Data to Provide Higher Level of Actionable Intelligence
Startups aren’t the only ones developing sophisticated store sensor systems and analytics. At NRF, Zebra announced SmartSense for Retail and SNAP (Sensing Network Appliance) SmartSensor, a system that combines video analytics, accurate locating of store associates and shoppers using ultra-sonic technologies, RFID-based inventory and asset locating capabilities,7 and a WiFi access point. The video analytics can identify and locate individuals within its field of view. The RFID reader uses a phased array of antennas to provide an approximate location of items it ‘sees’ within its read range. If the shopper has a smart phone and has opted in to the retailer’s app, then the system can add ultrasonic locating of the shopper,8 which is more accurate (± ½ meter)9 than either RFID or video. Retailers that equip store associates with mobile devices can use ultrasonic locationing of the store associate. Zebra’s local edge processing analytics engine (built into the SNAP appliance) combines those different signals to deliver a higher level of intelligence and understanding about events happening in the store, such as shopper #123 picked up item ABC and XYZ, and put item ABC in their basket and XYZ back on the shelf. It can also provide prescriptive analytics, alerts and recommended actions for store associates and managers. Zebra has five initial applications for their SmartSense for Retail platform:
- SmartSense Inventory Management – Dashboard providing total item counts and predictions/alerts of which items need replenishment.
- SmartSense Omnichannel Enablement – program to help store associates pick omnichannel orders from store inventory, providing them with the location of inventory to be picked, and automatically recommending an efficient path/pick sequence. Geiger-counter mode can be used to home in on the exact location of items. This solution is for RFID-tagged items.
- SmartSense Loss Prevention — Geofencing and motion alerts when high-value items are moved, for example when items are taken into the dressing room or out exits without going through the register. Time stamped events can be married with video to view incidents as needed.
- SmartSense FindMy – ccurate locating of store associates (or their misplaced devices), using the ultra-sonic micro-locating capabilities.
- SmartSense Video Analytics – lerts when checkout lines are too long, enables hot-spot analytics of dwell times, counts of shoppers, and other functions.
Zebra is currently piloting SmartSense with several major retailers. In the future, Zebra will provide workflow optimization; i.e. the dynamic routing and assignment of store associates to various tasks. Their platform opens up many other possibilities as well, such as when a shopper comes in to pick up a BOPIS order (buy online, pickup in store), they could start shopping for other items while a store associate is alerted and brings the BOPIS items right to the customer’s cart. It’s not clear when and how this platform could provide a checkout-less experience like Amazon Go (for non-RFID tagged items), but there are many other high-value scenarios and use cases it already supports, that should be attractive to many retailers.
Tyco Retail — Showing Innovation Mojo
Since their acquisition of Vue Technology almost 10 years ago, Tyco Retail Solutions (TRS) has had a forward-looking, evolving vision of the intelligent integrated store. Last year, Tyco (which includes TRS and their fire, security, and connectivity services) merged with Johnson Controls, the world’s largest building automation systems company. This has led to an expansion of that vision to include more smart building concepts and a broader integrated platform. Becoming an even bigger company has not stopped Tyco from innovating and trying out some more agile approaches. Case in point, at NRF they gave an advanced look at some ‘minimum viable products,’ something they’ve never done before. One was a new interactive display mounted on their EAS pedestals. The display includes a video camera and machine learning to create a personalized interactive experience as the shopper enters the store.
They also showed a prototype inventory counting robot,10 developed in partnership with Knightscope. The current incarnation of that robot is already being used as a security robot patrolling stores, checking if doors are locked, and looking for unauthorized personnel in the store. Tyco demonstrated a self-service detaching kiosk, where the customer can use their own phone to scan and purchase items and then use the kiosk to detach the EAS tag. They showed several new tags, including one that had an E Ink11 display, that would allow instant changing of prices across the whole store — or across a whole chain. This is a fascinating concept, as it could allow time-of-day pricing, instant response to competitors’ pricing moves, much more sophisticated and gradual incremental markdown strategies, and more. Tyco told me they have received a resoundingly positive response from retailers for this concept.
Tyco acquired ShopperTrak a little over a year ago and acquired Footfall almost two years ago. The combined data set from these two has allowed Tyco to create a global index of about 40 billion shopper visits per year. They told me they will combine that data with RFID data, to help retailers offset the decline in store foot traffic, by increasing conversion rates and basket size. They accomplish this by helping retailers better understand and improve the customer experience, provide better tools for store associates to serve shoppers, ensure higher on-floor availability, and optimize product prices, potentially adjusted in real time. Lord knows that brick and mortar retailers need all the help they can get to lure shoppers back into the stores and keep them there (see Stores’ Identity Crisis: The Reimagining of Physical Stores). If Tyco can help stores succeed in that endeavor, they will be embraced by retailers, many of whom are struggling for survival.
Impinj — xSpan Reader Providing Directional Monitoring of Doors and Narrow Spaces
In addition to being the largest supplier of UHF RAIN RFID chips, Impinj sells RAIN RFID reader chips, RAIN RFID readers, and software, which combined, provide a complete platform. Impinj’s platform is used in many different environments and industries, such as healthcare, manufacturing, and retail. Three years ago, they introduced their Impinj xArray gateway, an overhead UHF RAIN RFID reader providing continuous coverage with locating capabilities. As we discussed in Road to Value for RFID in Retail, most retailers have opted to use handheld readers, with their primary use case being inventory cycle counting. Only a few have experimented with overhead locating readers like xArray,12 which can be used for real-time inventory management and more. Part of the dominance of handheld readers in retail is because they are much less expensive, and easier to repurpose than overhead locating readers. We may see some adoption of overhead locating readers as retailers figure out which use cases best leverage the value of real-time locating capabilities.
Retailers have been somewhat more receptive to doorway gateways and readers — these are typically mounted to the top of the doorway and they detect when tagged items pass through the door opening. Without having to cover the whole store, highly useful information can be gleaned, even from a single reader. For example, a single reader over the door separating the back of the store from the sales floor could sense all inventory movements between back and front. Last year, Impinj announced their xSpan reader. Like the xArray, it has an array of antennas, however it cannot sense location (as the xArray does), but it can detect the direction of travel of tagged items through the doorway. Direction is a valuable piece of additional information beyond the mere presence of a tag. It can tell, for example, whether specific items were just moved from the front to the back of the store or vice versa. xSpan can also be used over dock doors to read pallets being unloaded or loaded onto a truck. For dock doors, this setup is much less prone to damage compared to the traditional approach of putting antennas on either side of the dock door, where they are much more exposed to getting smashed by forklifts, pallets, and other moving objects. The xSpan’s beam pattern is rectangular, so it could even be used to cover a small, narrow store. Impinj told me that the xSpan has been well received.
Shelfbucks – Practical Approach to Tracking Promotion Execution
Our research has shown that one of the biggest factors determining the success or failure of promotions is execution — were the displays and product delivered to the retailer on time? Did the retailer set up and stock the displays on time for the promotion? Shelfbucks is focused on providing visibility to both the manufacturer/brand owner and retailer to ensure proper execution of promotions. Their solution is designed to minimize the effort and friction to get set up and start. They are working with promotional display manufacturers to get their tags built into the displays. Their readers are completely self-contained, with a 5-year battery and built-in cellular connectivity, so there’s no requirement for power or network connectivity from the retailer on-site. It is magnetically mounted, so just find a place to stick it and you’re good to go. Neither the retailer nor the manufacturer pays for readers or infrastructure, so there are zero incremental startup costs. Instead, they pay by the campaign, so are only paying as they are receiving value, getting ongoing data about the disposition of the displays for the campaign.
Their solution uses proprietary long-range IoT radio sensor technology, which Shelfbucks told me has a read range of over a kilometer. That means you don’t need many readers — just three readers per store, even for very large stores, it provides the required locating capabilities. With that setup, they can tell when a display has arrived at the store and whether it is still sitting in the back of the store or has been put out onto the sales floor. Those of you in the industry with longer memories may recall Goliath did promotional display tracking for several years. There was a lot of interest and Goliath proved out the potential and value. However, the technology wasn’t quite ready; readers were quite expensive, the cost per store was high, and the business model not quite right. Costs have come down a lot and a lot has been learned about the right business and technology model. Shelfbucks acquired the IP assets of Goliath last year, as well as some of their key employees. It looks like Shelfbucks is well positioned to finally make promotional display execution tracking work at scale, with a clear ROI.
SMARTRAC – ll Sorts of Tags + Sophisticated Software
SMARTRAC is one of the world’s largest makers of RFID and NFC tags. They are technology and frequency agnostic, working with RAIN RFID (UHF), HF/NFC, LF in a range of formats including inlays, labels, hard tags, pre-laminates, and cards. Based on their enablement platform Smart Cosmos, they combine these tags with sophisticated software and solutions they have developed, such as Metrics, which tracks tagged items from source through the supply chain and through the store. The tracking can be to whatever level of granularity the brand and retailer’s reader infrastructure provides. If stores have continuous overhead readers with locating capabilities, this could include automatic inventory management (continual inventory counting and replenishment alerts), location of misplaced items, support for omnichannel fulfillment (providing data for distributed order management/order promising, and for in-store order picking), merchandising, customer engagement, and store layout (understanding products considered but not bought, physical path to purchase, and so forth).
At NRF 2017, they demoed their IoT solution Experiences, showcasing innovative consumer experience for SPYDER, utilizing a SPYDER jacket with an embedded NFC tag. By tapping a phone near the tag, it brings up a consumer experience including social media engagement with US Ski Team members. If they’ve opted-in, it can also provide useful information for a skier, based on their current location, such as snow conditions and trail maps. It will be fascinating to watch where SPYDER and SMARTRAC take this concept and how it will be received.
Alien — Pushing the Envelope on Reader Performance and UHF RFID Chip Reliability
Alien is one of the ‘big three’ UHF chip makers, who also makes their own tags and readers. At NRF, they were showing off their new ALR-F800 4-port, enterprise class reader, with PoE13 (providing full power to antennas) and the ability to dynamically switch between DC and PoE if the LAN goes down, without resetting the readers. It has a dual core processer, running Linux,which can run a webserver in the reader, with software to make configuration of the readers visual and easy. They also showed their ALR-H450 Android-based ruggedized reader with a full physical QWERTY keyboard, high res display, quad core 1.3 GHz CPU, and 4 dBic antenna that can read from 20-30 feet away (depending on conditions, of course), priced aggressively at $1,600. Recently they added Higgs-EC to their line of RFID chips. It includes error correcting codes that will identify and correct any single-bit errors, and detect any dual-bit errors. This will reduce the number of defective chips or marginal chips that go undetected. The same technology allows them to increase the number of guaranteed write-cycles to 200,000, double the norm. Recently, Alien announced two new UHF RFID antennas: The ALR-A1001, a very thin 10” X 10” X ½” antenna with 8.5 dBic gain (same as larger form factor antennas), and the ALR-A0501 5” X 5” antenna with 6 dBic gain. Alien continues to push the performance frontiers in this ever-competitive space.
Plexure — Shopper specific personalization, without asking for personal information
Plexure (formerly VMob) integrates CRM and POS data with device-agnostic IoT data from consumer devices and sensors, to drive personalized customer experiences and analytics for retailers to improve promotions and merchandising. They told me they can integrate with any POS system and that they can use a consumer’s app installation ID without having to ask the shopper for their email address, name, or other personal identifying information. They said that not asking customers to register and provide information increases download and usage rates by about 30%. By getting the customer to scan this ID at checkout, they can track who is responding to which promotions, how different discount levels are working, and let retailers use that information to optimize promotions. It can also be used to segment customers and personalize offers. They have also done other kinds of projects, like helping a college food service track calorie compliance (which is required by regulation in some regions). Plexure is a Microsoft partner of the year and a GE Predix partner.
Amazon Go, Compelling Other Retailers to Get Going
Amazon Go may be the big splashy story for in-store IoT technology and user experience. But as we can see, there are many other ways retailers can compete with Amazon. We expect the checkout-less experience will initially be only practical in certain constrained types of settings. It will take many years for it to spread from small format, limited selection grocery to more general settings. Meanwhile, there are many other applications of in-store sensor systems, including RFID, video analytics, BLE, that are being implemented today, for everything from better inventory management and reduced out-of-stocks, to better promotions, improved customer experiences, more insightful merchandising, and more. Amazon Go may be taking the limelight, but it is also spurring on other retailers to step up their in-store experience.
1 Some other retailers have experimented with what they call ‘checkout-less’ shopping, which involves customers using their own smart phone to scan items themselves and put them into their bag, then being able to just walk out of the store. Having to scan each item with your phone is much more cumbersome for the shopper than what Amazon is doing. Actually, it is more cumbersome than even using existing self-checkout machines (albeit minus waiting in line). In addition, it is probably harder to prevent both innocent mistakes and intentional theft. As a result, this form of checkout-less shopping has yet to take off. — Return to article text above
2 Amazon has not been specific about what sensors they are using in their Amazon Go store. Their patent filing mentions scales and ‘other sensors.’ The filing also mentions RFID, leading some people to speculate that Amazon is tagging items with RFID tags, but we have always been highly skeptical that was the case. RFID tags on grocery items do not make economic sense. Recently I read that Amazon confirmed they are not using RFID tags. Whatever sensors they are using are most likely there to augment and improve the accuracy of their machine vision algorithms. — Return to article text above
3 Accuracy, reliability/uptime, purity of a substance, and other measures of quality or performance can be expressed as a number of nines, for example ‘three nines’ (99.9%), ‘four nines’ (99.99%), ‘five nines’ (99.999%), and so forth. Three nines (99.9%) accuracy will be correct 999 out of 1,000 times; five nines will be correct 99,999 out of 100,000 times. — Return to article text above
4 We use instrumentation here in a similar sense to instrumentation on manufacturing plant floors; outfitting them with sensors and control systems/mechanisms to allow more precision, automation, and control. — Return to article text above
5 Deep learning is a form of machine learning that incorporates multiple layers of linear and non-linear transformation processing. Some deep learning algorithms can do unsupervised or semi-supervised learning and have proven to be particularly suited to video analytics, natural language processing, and other areas. — Return to article text above
6 Most locating systems require that a map is created (or given to the system) as a pre-requite step before they can do locating on the map. Then the locating mechanism figures out where you (or some object) are on the map. Simultaneous localization and mapping (SLAM) refers to a system that builds out a map of its surroundings and figures out where it is on the map, all at the same time. Typically these system will have many sensors that give it a rich reading of its environment. Self-driving cars and UAVs (unmanned aerial vehicles) employ SLAM systems (combined with pre-existing maps) to navigate. When the system has fewer sensors, it receives a sparse set of data points about its environment and needs to fill in the gaps in what its sensors ‘see’ — sometimes in real-time. That is referred to as ‘sparse SLAM.’ SLAM is challenging enough. Sparse SLAM is even harder. — Return to article text above
7 For more information about the evolution of RFID from handheld to overhead readers, see The Rapid Road to Value for RFID in Retail. For now, Zebra’s solution relies on having RFID tags on the items to track product movement — i.e. they don’t yet track non-RFID items. — Return to article text above
8 The SNAP SmartSensor emits an ultrasonic signal (too high for humans to hear). The microphone in the smartphone, tablet, or other mobile device equipped with Zebra’s app picks up those ultrasonic signals which are then used to provide the precise location in the store. — Return to article text above
9 Zebra calls this ‘micro-locating.’ — Return to article text above
10 The inventory counting capabilities are still in development and have not yet been piloted at a customer. — Return to article text above
11 E Ink makes electronic tags and displays that use extremely low power (compared with traditional LED displays) and, since they are reflective, can be viewed in bright light. The very lower power alleviates battery life concerns. — Return to article text above
12 Currently the Impinj xArray gateway has seen more traction in healthcare. — Return to article text above
13 PoE = Power over Ethernet — Return to article text above
To view other articles from this issue of the brief, click here.