Monitoring the Health and Wealth of 45,000 Connected Kiosks (with Eric Hoersten)

We're talking with Eric Hoersten who started his career as the internet was taking its shape and how connectivity has always been a core aspect of the technology solutions he's been involved with. Eric learned a lot about the complexities of managing large connected fleets as vice president of information technology throughout the early years of Redbox. Today, Eric is the chief product officer at Banyan Hills where they are helping companies leverage software tools and technical expertise to build effective automation around fleets of connected devices and the operations teams that manage them. 

Kick back and let the nostalgia wash over you as we revisit the pre-streaming years when a trip to your local McDonald's was just as likely to rent or return a DVD movie than it was to buy a cheeseburger and fries. The Redbox automated kiosk experience leveraged early cellular data technology to deploy and manage over 45,000 locations across the US.  Eric shares what it was like building out the technology stack from the ground up to remotely manage their own systems and data from a central system.

What I think you'll enjoy about this conversation is the opportunity we had to look in the rearview mirror and examine how the Redbox experience provided unique insights. Insights have been baked into the software platforms that Banyan Hills provides to their customers to manage and monitor their own fleets of connected hardware devices. 

Transcript

Ryan: 0:00

Welcome to conversations and connectivity. I'm Ryan Carlson, your host. This is a podcast for IOT professionals on the IOT, curious that find themselves responsible for growing executing, or educating others about what to expect when you're connecting products and services to the internet. Today, we're talking with Eric Hoersten who started his career as the internet was taking its shape. And how connectivity has always been a core aspect of the technology solutions that he's been involved with. Eric learned a lot about the complexities of managing large connected fleets as vice-president of information technology throughout the early years of Redbox. And, and is now the chief product officer at Banyan Hills. Eric is now helping companies leverage software tools and technical expertise to build effective automation around their own fleets of connected devices and the operations teams that manage them. Kick back and let the nostalgia wash over you. As we revisit the pre streaming years, when a trip to your local McDonald's was just as likely to rent or return a DVD movie at an automated internet connected to large red kiosk than it was to buy a cheeseburger and fries. The Redbox, automated kiosk experience leveraged early cellular data technology to deploy and manage our 45,000 locations across the US and Eric shares what it was like building out the technology stack from the ground up to centrally manage their own systems and data. What I think you'll enjoy about this conversation is the opportunity we had to look a bit in the rear view mirror and examine how has experience has provided unique insights that have been baked into the software platforms that he and his team at Banyan Hills have created for internal teams that support large fleets of connected services on their own. To find out what exactly that path looks like. You'll just have to tune in and find out more. This episode is brought to you by Soracom, a cellular network provider built specifically for supporting IOT applications with a reliable connectivity across urban and rural areas around the world. Stick around at the end of the episode, to learn a little bit more and some additional interview outtakes. Thank you for taking some time to talk about connectivity, technology and where we are in the world today. I'd love to understand what your relationship with connectivity has been in your

Eric: 2:17

so connectivity, in terms of just internet connectivity or,

Ryan: 2:23

making things talk to the internet.

Eric: 2:26

I started my career as the internet was really, taking shape and, the.com bubble, was taking place. most of my careers involved started with web applications and, the, move from traditional client server applications to what we now call the cloud, where you have hosted applications, in a data center accessed via browser. my initial days I was a programmer, did web developing, built a lot of web apps, more on the commercial side, not necessarily consumer facing. and then a lot of folks migrated that into, more of a business focus where it became less of a browser based application, but leveraged into devices and other technologies, still kind of client server from a, there was a local, some type of PC or some type of controller connecting to a data center, using the internet as, the way to. or make that connection. and, as the internet continued to evolve and as connectivity, became more and more ubiquitous, went from, the early phases of dial up or cable modem or DSL to really leveraging cellular. part of my career was with the, uh, Redbox DVD rental, business where we had about 45,000 locations across the United States. almost all of those leveraged cellular, which back in 2005, 2006 was, still a bit of a pioneering effort because the technology and the ubiquity of routers for that type of, solution wasn't quite, where it is, certainly not where it is today, where that's, it's now, very simple to, to stand that up and manage those plans. for me, the connectivity has always been, a core piece of the technology solutions that I've been involved with. And, in fact, at Banyan here now and with my previous lives, that's been a core design principle that we've had to build our applications around. understanding that, being able to connect from the client to the server enables you to do certain things. And if you lose that connection, architecting the solution such that it can still operate in, quote, an offline mode, or, be able to navigate, while it loses that connectivity because, maybe less so now, but 10, 15 years ago, the reliability of that connection, isn't where it is today. building solutions that could withstand, the, a situation where you didn't have what you typically have in a data center where you've got pipes, going to different gateways, allowing you to, not have any interruptions at service, but rather. Building it in such a way that you expect interruptions in that service and, try to insulate the, the customer from that.

Ryan: 5:19

I think that's fascinating that you went from the data center world to, that consumer side of the world with Redbox. you think about connectivity, having been in consumer facing services myself it was from 2000, 2001 to about 2012 was in the carwash and laundromat industry, bringing connected monitoring solutions, remote payment systems. And these were all places that didn't have any internet to begin with anyway. So you're bringing in dial up because that's all you had. And bringing in a hard connection. That was for colleges, right? Your t1, t2, t3. then at some point, DSL and broadband gets to a price point about 50 bucks a month, where it was reasonable. If we wanted to do cellular, you had to buy a hotspot. And data plans still came with a phone number. So in 2004, 2005, having explored connectivity, the, these telecommunications infrastructure companies we're not quite ready to handle what that future place would be. Where we are now, where my light bulb has a wifi chip right? or it's got a ble chip for low energy Bluetooth. And it's fascinating to think about that connectivity in your world has always been a connective tissue to something else. It enables something else. No one buys connectivity because that's what they need. It's to accomplish something else. And so I wanted to talk to you about, at Redbox. You've got a unique experience. The fact that you were getting the interface with a piece of equipment that nearly every family, regardless of their socioeconomic status, was able to interact with and get access to the same movie titles as everybody else If you got a McDonald's or a Walmart in town, you probably had a red box. You probably still have a Redbox

Eric: 7:13

Yeah.

Ryan: 7:14

And what connectivity did was not steal someone's job, connectivity commoditized. The worst parts of a task that no one ever wanted, right? delivery route. I think about, the things that I do today from my phone, and even back in 2005 when me and my family wanted to rent a movie, we'd go, what's in the local red box? look and we could know in real time, oh, there's one of'em at the Cub Foods down on such and such street. Reserve it, right? Just like I can reserve my haircut. you go into a great clips, I don't know if they have those where you're at, but they're like, did you check in online? It's the first words out of their mouth, and if you didn't, you feel kind of dumb because, they text you 15 minutes out before your appointment. And these are things that are all enabled because a store is now able to communicate with somebody else through the power of the ether, right? And the e-commerce shopping experience, the, now we've seen the rise in, Uber Eats and all the delivery places. Like I know exactly where my food's at. Yeah. crazy right now everything is starting to have its own connected experience. And as consumers it gets to be overwhelming, right? Like smart home. So there's a whole new category of product that is smart home management, right? So you've got Google Home, you've got Apple Home, and it's meant to pull in all these disparate things and you're setting a standard. And if you don't communicate in Google Standard with the Alexa standard or the Apple standard, you're not gonna be able to have your voice magically have your lights go on and off,

Eric: 9:05

Yeah, I totally agree. And I think the, the internet was such a pioneering push for this connectivity. whereas, cell phones obviously existed and I could make a call and using some of the same technology to a tower and route that to someone else's phone. I certainly remember the first time, this is probably 2000. Two where someone showed me that, yeah, I can order a book from Amazon using, a cell phone. And this is before a smartphone. It was just hit one, hit two to navigate through the menu system. and at the time I'm like, wow, that blows my mind away. And now, it's, it's an afterthought that you can do that type of stuff. and certainly you touched on it from a red box perspective. We did hit the, we hit the market or the technology curve at the right point. a, as I mentioned, we were early adopters. We were using cellular technology when it was running on like networks like one X R T or eventually it was C D M A, which, by today's standards, seems like a a very, very slow connection. but we went that way because we couldn't manage for the deployments that we were trying to do as quickly as we could. pulling in, a telephone line or a DSL connection and managing the, all the local telcos to be able to establish, that connectivity, as we're trying to deploy these units as fast as possible. for us, and I think for the internet in general, The cellular data was such a turning point for just technology as a whole, because it enables that ubiquitous, gateway or connection that you have to the internet to be able to connect to anything. And certainly that worked for us from devices, but as you mentioned now, it's just, just generally accepted that's part of your experience of having a smartphone is being able to do those of things. I can order an Uber and see how far away that Uber is and watch it as it's dropping off the previous passenger and then on its way to pick me up. all because this connectivity, of systems, using this common internet, gateway and even. To your point, an Apple system may not always talk to a Google system, and there's different ecosystems that exist within these silos. The underlying, development of just TCPIP technology so that, even if they aren't sharing the same APIs, they're on the same network and they could talk to each other if, the companies work together to, to enable that. I think that's, just foundationally is, one of the pieces that makes the internet what it is today.

Ryan: 11:36

Well, the fact that it is its own standard, whether you're part of. Smart home standard, you still are, on a network with an ip, you need dns, you're gonna be talking to a router. so there's always ways around that. And I've found that, there's been a lot of product journeys that I've been part of, working with engineering teams where we in essence, had to reverse engineer how something else worked, just to be able to intercept the data feed, parse the packets to set triggers and alerts. my family is owned gas and operated gas stations and the complexities that most consumers don't know is that there's actually on-premise systems. So my point of sale system and the gas pumps, there's the near premise systems, which are the coolers that have the Coke, the Pepsi, all these things. We don't own those coolers. The Coke, Pepsi, the distributor does, and it's their job to keep them up and running, pay for repairs, make sure the light bulbs are working, that the stickers are all in there, pricing is all updated. And then you've got the off-premise, which sometimes could just be right on the other side of the curb. You know that the air hose, the air filling station or the vacuum station. And one thing that is crazy is that extrapolates out to factory settings, the on-prem, near prem, and off-premise. And the connectivity profiles aren't just because of distance. and I'd love to hear your thoughts on this, how did cellular play into just not having to go to gas station or McDonald's and go, can we bum some connectivity? Yeah, no, it's, it's an excellent question and, I think it stemmed from a few different things. One was just when you're dealing with major department stores and the, it. Teams and systems that they have in place behind those organizations. they don't, they're not as nimble as we wanted them to be able to, have a line pulled and dropped in the location that we were gonna be, or it required a lot of pre-planning on our part so that we had to know exactly where that red box location was gonna be within the store.

Eric: 13:50

And if we got there, and it turned out that they had rearranged the store and we actually wanted to be, on the exit vestibule as opposed to the entrance vestibule. that required a whole bunch of work to have them move that line that they had, they had already run. So with cellular, as long as you had that signal, you throw an antenna on top of the device, and you're off and running. So the flexibility of just placement of the kiosk, played a big factor, but probably more importantly was just, as things like, as these store networks started to take shape before they got to the point where they are now, where, you know, a lot of vendors are now using the connectivity at the store and, companies that figured out how to create like a DMZ or a separate, vendor network that they can segregate from their internal networks. Before that existed, there was a lot of, just, begging to be able to be put on that vendor network. And if you did get put on that vendor network, you had to prove that you had gone through all types of compliance. In order to do that. PCI compliance was a big deal as more and more people were starting to, enable credit card transactions. And because all these stores had, point of sale solutions that were on that network, just adding our system to that network was something that the IT teams were really resonant to do because now we were in scope of making sure that when they had their audit performed, that we lived up to that security standard, just like they did. And even though we were, it was just extra work for them to have to incorporate us into those annual audits to prove that, that success. And as you can imagine, a lot. It security, teams are just skeptical of something they don't control. So even though we may have had the best in class security, they don't, they, don't trust us because they don't have that full oversight, of that. So using cellular as a way to just control our own destiny and be able manage that ourselves, just made it, so much simpler for us to navigate that. And, and took a lot of the friction out of the rollout process and just, removed hurdles that we had to deal with, before that was an option

Ryan: 15:58

There is an operational aspect, an operational tax that a company that maintains connectivity. So you have deployed fleets of all of these Redbox devices out in the world. And as a consumer, I only ever saw the app that gave me a nice front end interface, my account settings, my payment information, and all of the menus. We only had one interface, a website and an app. Talk to me about the number of backend systems that an operation, would've been interacting with and what were some of those jobs that they were serving?

Eric: 16:35

Yeah, you kind hit on it earlier where, when we launched the mobile app, we gave the consumer the ability to find any kiosk based on, where they are. Or they could put in any zip code and see, where those kiosks were, and then even identify what movies were in stock, at that, location. So this was, I don't know, 2000. Seven, 2006. which at the time, I would always remind, the non-technical folks of the executive team, we basically built a inventory management system across the fleet of, 30,000 locations. way before you could do that, now you take that for granted, I'm going to Home Depot. I can log in, see what's in stock and what aisle it's in, and how many they have. that, that is a relatively, technologically speaking, recent, innovation. and I think, we pioneered that, that, that customer behavior where before I make that trip, I want to know that it's gonna be there. In fact, I'm gonna reserve it and ensure that when I get there, it can be there. From a backend perspective, we had an enterprise data center that would process every one of these transactions. So that's where things like payment processing, and we would maintain, the business rules. So if you had, I forget what the exact numbers were, but if you were a new customer, you could only rent three movies at one time. And so if you rented one at, the kiosk down the road, and then another one from a different kiosk and another one from a different kiosk, that central, enterprise server was keeping track of every user and how many disks that they currently had outstanding. And just holding all of that, data. So it was the brains of the operation. Obviously the analytics and the operational components, were all, built off of the data that we were accumulating at the enterprise layer. but then we had to design the system in such a way, or the kiosk in such a way. going back to some of the stuff we talked about earlier, where if we did lose that connectivity, it was smart enough to continue to operate, on its own until that connectivity was restored. So being able to kinda store and forward those transactions so that, when it came back online, it could process the fact that these returns happened and this was the time that returned happened, and kinda replay history for the server so that it would sync up with any activity that happened while it wasn't able to talk to that enterprise system.

Ryan: 18:57

I personally know that any systems that were built before 2015, 16 ish was pre aws iot, o t, and any platforms that existed to both host data, to do identity management, to have tie-ins and hooks for device status, you had to make it all on your own and or cobbling together a whole bunch of different vendors, right, build by partner You're partnering with a professional services company in order to buy someone else's pre cammed solution for the payment processing piece with the gateway processor, you've got your inventory management people, you've probably got, all sorts of different tie-ins to some of the internal components for counting, like a maybe a M E I R'S bill validator inside. is it full, does someone need to go and pull all of the$20 bills out of there because we can't take any more cash, which happened, right? I remember so many times people like, card only I, I come here because I don't have a card. and it's cuz the bill validator was right? Knowing that there's a whole bunch of backend systems that you had to pull together. There was no magic view. Now there was no single pane of glass that would make everybody happy. You probably had a bunch of different stakeholders from a business perspective that they themselves had their own job to perform, but they had different pieces of information that probably came from not just the machine. So the easy one is the maintenance team. I wanna know when something's broken. So what would, what, how did they know whether something was broken?

Eric: 20:31

Yeah, no, it's, it's a good question and we got more sophisticated over time and, that's a lot of what we do now, with Canopy and, Banyan Hills. for the Redbox example, we would aggregate data across systems. we try to develop the software in such a way that it would send us notifications when it ran into errors. But as you can imagine with software, it only tells you what it knows that it's gonna run into. Inevitably there's, that's just the tip of the iceberg. Everything else is stuff that goes wrong that you didn't expect was gonna go wrong. we would be collecting data from the software system. we would have. uh, our transaction system, feed us data so that we could see, typically this machine does a hundred rentals on a Tuesday afternoon. It's only done two and it's been four hours since its last rental. maybe there's an issue there.

Ryan: 21:18

if we did the same thing in carwash, where it was, looking at the money that was taken in like a six Bay self-serve and you'd say, typically we're supposed to be transacting at least this much money or done this many washes. I don't know what's wrong, but I can tell you that you should go and check it out cuz people aren't washing so something's wrong. Go figure it out. It's the check engine light versus the actual like remote diagnosis. You mentioned Canopy and Banyan Hills, a company that creates remote fleet management of deployed devices and you create dashboards. So I'd imagine as a chief product officer, you, it's I don't know if our roles were diverse, I would suspect I'd be fairly passionate about what you're doing today based off of, if only these tools existed, that were so much more, much more turnkey. So what if you had the canopy product at your disposal today and you could bring that back in time? what could have you done for. the service representatives of the maintenance teams, what could they have life easier for them?

Eric: 22:21

I, I joke about this with the team, at Banyan and also with prospective customers when we, walk into their environment and try to understand, the challenges that they're facing, that, wow, if I would've had, canopy and the Banyan team back when I was trying to do this, and I didn't have to build all this from scratch, that would've been, the easy button to just say, okay, I can, I can focus my resources on, core business problems as opposed to trying to solve for knowing the state of my fleet or the health of my fleet. I, and I think a lot of that just comes down to the thought and the design that's, go, gone into the system. So it's an agent based platform where, you've got this very lightweight agent that can run on just about any operating system, but is able to grab a core set of information about, not just the controller that it's running on, but the peripherals in that environment or other devices that may be on that network. and that's where I think. For a solution like Redbox or what we like to call a non-traditional environment, there's a lot of tools out there that can manage a laptop or a desktop or a printer that are just geared to work in a small business environment. And you need to know the cpu, the hard drive, what patches are installed, those types of things. For an environment like a kiosk, it's totally different. You're probably, you may be running windows, but you're also might be running some flavor of Linux that's been, stripped down to suit exactly what you need to do. So the traditional RMM tools are somewhat light. in that space. and the Canopy agent is designed to not just monitor the device that it's running on, but all the peripherals that are connected to that device. So in the example of Red Box, it could be the, the card reader or the touch screen or the monitor or, even some of the hardware components or the PCB board that was controlling the robotics of vending, those discs. And that's a lot of times where, the errors would occur. knowing that the controller's healthy is, that's helpful. But really that doesn't define whether the system is up or not. You need to know that the actual ability for the customer to return that disc, that the door's gonna open, the picker's gonna grab that disc, put it back in the right slot, and be available for the next customer. the design of the agent to not just monitor what is running on, but be extensible to monitor the peripherals that exist in that device, whether they're, USB connected, serial port connected, or just on the same network. and, we talk to that over an IP address, makes that pane of glass or what we call KPIs about the health of that system, that much more powerful so that you've got the full visibility into that ecosystem as opposed to just knowing it's online. It's got power.

Ryan: 25:07

so I would be able to have, if I were say at corporate, I could see a dashboard that just said overall uptime, overall aggregated things. Maybe even have a map that's like red, yellow, green, that me in these regions we've got more issues and I could always drill down. But then if I was actually a regional rep, I might have a dashboard that shows the, the red, yellow and greens, the check engine light. And it would just say, by the way, these are your priorities. Don't worry about them. and maybe a daily, here's your route.

Eric: 25:43

Yeah. Yeah, exactly. and, as you can imagine, if you're deployed on 20,000 locations and, you look at, you pull that up and there's 20,000 locations, that's just information overload. That's not very meaningful you. while we've got the data on all those 20,000, our ability to then filter and segment that into different, groupings or even classifications. I only want to see, locations that are in the Midwest or in Illinois that have a status of red. That means that they're unhealthy, so that I can prioritize. I need to look at those locations first as opposed to just trying to sort through, an unending list to see what the health

Ryan: 26:24

I would imagine that you've got a whole nother group at Redbox that would wanna look at a very different display, and it might be, around inventory and around, the popular titles,

Eric: 26:35

Yeah. obviously that was, one of the components that caused downtime. inventory was, a challenge for us because we would buy more disks than we had capacity in that machine, expecting at a certain time that they would, there would be some portion of those discs rented out. And if, Sunday evening comes around and, or maybe even more so Monday afternoon where a lot of people are doing those returns and there's not enough slots left, we need to make sure that's on our operational dashboard to get, someone out there to free up that inventory because, As you can imagine, the most frustrating thing for a customer is, I made a special trip to return this. I don't want to get charged extra. And if I can't make that return, that's a really bad customer experience. being able, it just goes back to collecting, aggregating data across multiple systems to really define the health of your network in, in that scenario, everything else could be great. The hardware's working, the software's working, it's connected. It has, all the right, core health statistics, but if there's not an open slot in that machine's actually down. So the marriage of all that information across the different systems, is what you really need as an operator to have full visibility into how to manage that business.

Ryan: 27:52

I keep thinking it's almost like a stock ticker or some kinda like, we need more finding Nemo quick. buy low, sell high.

Eric: 27:59

Yeah.

Ryan: 27:59

I would imagine that a company like Redbox at that time, pre streaming had some of the most valuable information. To the media companies to say, we know exactly which part of the city, yep. this is really big. It's a huge Hispanic community and this title is flying off the shelves. You don't need to know anything about the individual, but you can start learning trends about preferences and what sells and what doesn't and where. So I suppose where you send Overstock is gonna be largely, informed by data as well.

Eric: 28:37

Yeah. I, we were an early indicator for the performance of those titles when they hit the DVD market. So just like everyone else, we got new releases or we were able to, offer those new releases to the customer on Tuesdays, that's when the window came out. So on that Tuesday, the studios could get a bunch of data from, at the time, people would go to Best Buy or Target and buy those dvd. Whereas, but getting, compiling that data from Best Buy, target, Walmart, all these retailers to see, what the performance of that is, was cumbersome. Whereas we could give them an instant, visibility into, Tuesday at 7:00 PM This title has already really started to perform, above or below where we expected it to. And that was a really good leading indicator for the studios, for their production of those, those movies and what, they were going to expect, not necessarily from the rental market, but just from the, that was such a big revenue driver from them, from the actual purchase of those movies, through the retail outlets.

Ryan: 29:40

So walk me through the decision making process for your, typical company that, approaches guys saying we've got lots of data. does it start with them having a very specific question, we need to be able to answer this question on a regular basis, or is it we need to make sense out of a, we don't know, talk to me through like at what point does someone say, yeah, it's time to pull all of this information together and the people we've got back at the office spend way too much time manually pulling it together.

Eric: 30:16

Yeah. and a lot of it depends on just where they're at and their journey. there are definitely stepping stones that we see where, cu customers start to hit breaking points. So for someone operating a kiosk, for example, it's manageable if you have 15 locations and they're all in the same geographic area and you can just, try to keep those up and running and know the status of them by brute force throwing you.

Ryan: 30:40

personnel at that and, overstaffing to ensure that someone's just manually checking on that device, whether the logging in remotely or actually, going to that location to, to see how it's operating. Once you get to a hundred locations, that just doesn't work. there's just not enough time in the day. It's not cost efficient to be able to do that. And then certainly as you get to 500 or a thousand locations, now you're geographically diverse across, whatever territory that you're trying to roll this out to. you need to have better visibility into the health of those devices. for some customers, they're just transitioning, into an environment where they're in an unattended mode or maybe they've, had, a retail experience before, but now they're transitioning to an unattended, type of device where they don't have someone there that can just unplug the device and plug it back in and, okay, now I'm good. That's where, we work with them to understand, okay, first what you need to do is have visibility into this. You need to know what the health of your fleet is so that you're not getting calls from a frustrated customer. before they do that, there's something wrong, and you can start the process to get that back up and running. provide the visibility first. Then let's get sophisticated about what it is that we're monitoring. what does healthy mean? And like I said before, it's not always just the, the computer's on, it's connected and, the CPU is fine. It's what's the overall health of that system based on all the key components that are required to actually perform that customer transaction. and then we've done a lot of work to build in, what we call iot OT automations, which is a. a low-code, no-code type of environment that allows our customers to get really sophisticated about the alerts that they're receiving and automate some of those actions. if I receive this error message that my software, experience this specific type of error, I know that you know the resolution because I have my, support technicians do this all the time. They remotely log in, they delete this file, they rebuild this database, and then they restart the unit and it's back up and running. Let's automate that with our agent so that if I see this, we'll perform those actions, on that target machine. We will then, wait to make sure that the KPIs or the health statistics that we're receiving from that agent now say that error condition is no longer there. and then in the meantime, we've opened a ticket in there, help desk, updated that ticket after we performed that action and then closed that ticket, for the customer. So you can imagine the amount of efficiency that you get from just having that, that system, just watching that and trying to automatically resolve. This reminds me of some of the QA automation, like Selenium and some of these, programs that just run scripts. and giving you the all clear, I know a lot of firmware that I've developed in the past and the boot up process goes through a quick little check. You know, check that, check that, check that, check that. If not, then send up a flag knowing that, we're gonna, we're gonna fail at that point. how often do you pull in additional sources of information outside of just that kiosk, but would there be, other environmental conditions or additional sensing that you would have?

Eric: 33:51

Yeah. Well, and, and that's another, component of the design of the system is just the extensibility of it. like we've talked about, the more data you have, the better your decision making, is gonna be. And if, if you know that the temperature in that location is 120 degrees in Arizona, and. that's a warning condition for the environmental, the components that aren't hardened enough to be able to operate in that environment. That's helpful information. pulling in, weather data, if it's, scheduling, seasonality, type information. Going back to Redbox, the day before Thanksgiving was always a peak rental day for us. So starting to understand previous trends of what we're going to expect, how much we should stock, and then, knowing from a transactional standpoint, a capacity, management, what we need to gear up for just by pulling in, historical data, in our systems now with Banyan, I think a lot of the benefit that we bring. Customers have the ability to take their business data and marry that to the operational data so that it's not just a point in time like here's the operational health, but, layer on the impact of, if my system is down at 11:00 PM on a Tuesday and I operate a kiosk in, I don't know, a Walgreens location, it's probably not doing a lot of business right then, or the storm may even be closed. So the severity of that type of issue isn't as high as if it's a Saturday afternoon, two o'clock where that's peak business hours for, you know, that application. you know, again, it's just marrying a lot of the data and understanding what are the. KPIs or what are the core sets of data that, will, allow you to prioritize, and understand ultimately the service level that you're providing to your customers? and prioritize and automate the outcomes of that based on that information.

Ryan: 35:48

I've thought about the internet of things, especially monitoring type scenarios, much like a utility. so for example, in a water utility, you would have a point of collection. So you're pulling the water out of an aquifer or pulling it in. So you're collecting it, you're transmitting it, through big super myro brother size tubes, all the way to a point of aggregation, which might be a water tower, water treatment facility. and then you've got distribution, which sends us to homes, restaurants, industrial plants, and points of consumption where it, the resources actually being used. And in the utility world, they call it the first mile and the last mile, first mile is point of collection to where it's aggregated. Cuz you need big, giant backhoe and entirely. Groups of specialists to address that, to pull the resource, and last mile is getting it out to wherever it is being consumed. And if we replace water as the resource to data, as the resource, it still of holds up. you want to treat it all while it's a one place. Normalize the data, things like that. and over the years, companies that want to go into smart product development either have a strong suit, they're like, we got the first mile on lockdown, right? we can get the data, we could even get it to some place. We don't know where to put it, but we can, where do you want it to go? But it's that last mile is the value extraction out of the data. The, now that you have it, that seems to be really hard. I make, an industrial pump or I'm making a, some sort of kiosk in some ways, right? It processes, it buys, it, dispenses, it does the thing, but. there's all of those jobs that you don't necessarily consider, from that value extraction. Where is it that you see the blockers when people come in saying, I built a thing, but now I'm having a tough time? what are those, those blockers look like?

Eric: 37:37

Well, certainly from a data perspective, yes, I agree. It's, you're grabbing it from all different, systems and all kinds of different environments, and then you want to pull that together and make it useful. not too dissimilar from, the analogy. There's a scrubbing that takes place of that data to ensure that, these systems are all speaking the same language or you're, aligning that data. when you aggregate it, it's, it's all based off of clean data versus, anomalies that are coming in and could potentially throw you off. getting it to a point where the data that's coming in is scrubbed and flattened so that it can be reported on, in a, in an efficient way, I think is, one of the big challenges that a lot of organizations face because there's a lot of, sophistication that goes into just understanding how to get that data cleaned and making sure that, the anomalies that are coming through aren't influencing the overall outcomes, that are going out there. And then, it comes back to connectivity in terms of making sure that once that data exists and it's been, scrubbed and now there's actions to be taken based on that data, getting that to, the locations so that. they they have that, capability or they have that intelligence now to act on that data. like your analogy, that last mile, it's not just the kiosks sending the transactions to the enterprise, but the enterprise actually pushing, some of that data back down, so that the, the kiosks are smarter. They know how to promote certain titles better at a red box location or, in another scenario maybe, they're discounting a, a certain product because, they know the inventory levels are higher than, where they should be to try to move more of that, and get that out there. I, I think that's a key piece is just having, getting the data, the right data and understanding what the right data is. that's a difficult thing to master.

Ryan: 39:28

well, and that's a good example of that distribution component instead of much water, what flow, what pressure, doesn't need to be portable or not. Now with data, it's who gets it roles, permissions, security settings. you get access to our sales data, but you don't get any of the customer information because p c I compliance, if it were a medical kiosk, you don't get any personal identifiable health information, but you can get data birth, that's it. So it's that, set of series of decisions that gets really nuanced. It's not just, Hey, you have at it and use what you need. Because sometimes regulations force your hand, whether it's compliance regulations or it's just sensitive information someone doesn't need access to.

Eric: 40:11

I, yeah, I think so. and to your point, you almost have to assume that once it's outside of your, Your control that it's gonna go everywhere. that data that you hold core, you need to make sure that what you're handing out to, uh, distributing either to a bunch of locations or to a vendor or something else that's limited to the, just the information that they need. And, yeah, look, I think data science, is, it's a broad term, but that, that's at the core of, technology in the internet. It all comes down to data and how that data is leveraged. And, understanding that data to the point where, now you hear so much about ai. and I kinda see that as just the old careers of data scientists or data warehouse or, analytics is really now just advancing to be, more artificial intelligence because that's all just based on training a system to understand that data and start to make certain assumptions based on that

Ryan: 41:07

Well, I mean, that's exactly it. It's training models. and when you were talking about, the data, flattening the data, I, I see lots of people, even developers, perform aspects of data science. But in my world for iot, there is the whole analytics, the looking into it, crystal ball type stuff, and AI just helps find those patterns faster. but I think there's a very reductive aspect of data science that is far more practical. And it's the person who can say, no, no, no, no, no, 112 degrees on a pump is bad. Oh, that's bad. Because the data is just the data until you apply context and say, here's the threshold. Now. That's how the motor vibration monitoring analysis can predict when a motor's gonna fail, because they had people who understood how to find those readings, set the baselines, tune the algorithm, and now you can hook a little device onto any pump, let it run for 30 days, and it's gonna tell you it's gonna fail within the next 24 hours.

Eric: 42:06

totally agree. and that's as we build our KPIs into our system, a lot of what that means is working with the subject matter experts who may or may not be technical to understand, okay, what are the thresholds that this means? Good, this means bad. And if it's 112 degrees, is that good or bad? that, that could be great. that's exactly the temperature it needs to be. Or it could mean that's the, we're close to this being a meltdown, you know, priority zero, we need to get, all over this thing. We actually have that built into our KPI engine where the user can set certain thresholds based on the data point that's coming back and assign that a category of, red, yellow, green, or, a metric that's numeric or, something that can be used as part of a calculation to get an overall status of health. So I, I think to the broader point, yes, developers end up being data scientists at times because the data by itself doesn't really mean anything. It's when you apply the context of what that data should be or what you're expecting that data to be where it becomes useful.

Ryan: 43:07

Yeah, we, in our, actually the very first interview that we had on the show, was with someone, we were talking about digitizing human knowledge. And it was the guy at the plant who's been there for 30 years. He can put his ear up to a screwdriver up against the cylinder head of a motor and go, oh, We need to check the pressure down the line and then we need to see whether the reservoir of oil's full, because we're only gonna hear this if these other disparate things are happening. And, nine times outta 10 that guy's right because of experience. But those people are going away. And I'd like to say it's because they're retiring and it's great, but we aren't replacing some of these jobs, But these are specialists who need the longevity in these roles to truly have that level of comfort and understanding of what's happening. And We can't train up enough people. We're not making people fast enough with these specialized skills to go in and understand this. So it is this automation. It is the ability to add, we've got on-prem data from the pump, but it's only gonna tell us when it stopped working. So it's the Banyan Hills of the world that need to be working with the other vendors to go that pressure sensor that's down, further down the line. And then the, float sensor that's, looking at the oil reserve, we know that those are key leading indicators for our KPIs to say this will probably happen, right? So what a fascinating world that we live in, that we have the technology at our fingertips, and now it is just a matter of getting people to see the potential and adopt what already is sitting in front of us.

Eric: 44:46

Yeah. I agree. And then, to your point, it's the domain knowledge or the domain expertise of, this should be here. This is always read on Tuesdays, and so you don't have to worry about that. Like incorporating that into the system, but then also the, well, when this happens, this is what I do to fix that. And maybe that was recorded in a notebook previously, but now that's put into a routine or into an automation so that you can audit that, this is how we resolved this. And, it, there's a trail here of this is the fix that we applied when this condition, happened in the past. Whereas, before that it was, we need to call Jim because Jim's the only one that knows what to do when this light comes on. I think that's where we're gonna see more and more of. that knowledge that's, in those types of subject matter experts, heads get systematized. and then that's where it can persist.

Ryan: 45:39

If you could snap your fingers as if you had the infinity gauntlet on your hand and just make one thing happen, change something what roadblock, if it could just get removed, would allow us to fully leverage more of what we already know or the technology we have at our disposal to, make businesses more efficient. Yeah. Um, boy, it's a big question. Like It's think a lot of power, big responsibility.

Eric: 46:10

Yeah. I like selfishly from a technologist perspective, I would say if we could, the types of development that's been done in all these devices and all of the, software applications that we had, if we could have, if that, if things like html where it's a common markup language that's ubiquitous across all platforms was adopted when all these technologies were being built, the integration and the understanding of how all these systems work would just be so much easier because everybody's speaking the same language. But what we have is, this system was developed in c this system's developed, mid COBAL because it was 40 years ago. and so now we have to put all these kind of translation layers. In between these systems so that they, can talk to each other. if I was gonna snap my fingers and I could make all these devices work on one operating system, using one common set of, markup languages or a just standard language, selfishly, that would make my job a lot easier because, we wouldn't have to learn every different, nook and cranny of all these different, custom solutions that have been developed over time.

Ryan: 47:23

That's awesome. you, I'm having a lot of P T S D right now of Oh, the bill validator that actually operates on one language. We got serial coming in over here. We got RS 32 there, cat five running in for, oh, that's ht. Oh, that's great. We got TLS coming on this end. We've got a serial sensor here. A digital sensor and having to make it work. would it be accurate to say, follow me here, that we are living in the age of TVs where we had. A whole lot of extra plugins with the coaxial, the white, yellow, red, and we had to have enough of them, but also an H D M I. And so it just is inflating the cost in order to truly be universal, we're paying a tax of supporting a bunch of other legacy ways of talking.

Eric: 48:09

Yeah. and I think we're getting, we are getting better at that, right? Like the whole adoption, that everyone has just decided, T C P I P is the way that we're gonna communicate. There's gonna be different protocols that sit on top of that and, if, can you use HTTP or HTTPS when I'm communicating, to a web server and I'm gonna use TLS or REST or some other protocol when I'm, building an api. The standardization of that all kind of runs on top of the same thing. So from a network perspective, if I know how to configure an IP address in a router and dns, like ultimately the plumbing's gonna get everything there versus, what you were talking about before, if it's serial or u sb or, some other type of kind of standard hardware protocol, that's going to aboard. So I, I do think that the more that technology advances there are, standards that are evolving that people start to adopt too, and that does make our world, that much easier and it more ubiquitous for us to develop. I can build an app on an iPhone, and that's instantly available to millions and millions of, potential, customers. I see that as a trend, but there's still a lot of legacy, systems out there that, aren't going away anytime soon.

Ryan: 49:23

Last question. Anyone thinking about building a connected product, which job hire are they not thinking about right now That will become mission critical in order to be successful?

Eric: 49:37

Yeah, it's a great question. I think it comes down to that it operations, resource where when you build that product, and I've been in this situation, you think about, okay, what are the features and functionality? What are the bells and whistles that are gonna get a customer excited about this? And you completely take for granted that it's just gonna operate once you deploy it. And there's nobody there to oversee it because you, you build it in a lab where your internet connection is rock solid and you've got clean power and you're there to be able to hit reboot. But as you start to scale this, you start to understand, wow, I'd never thought about in the wild. All these things are gonna happen. having that resource that's focused on saying, look, these are the things that can go wrong. We need to start to plan for those things to go wrong because they are gonna go wrong and we need to have reactive ways to, to make sure that when they do go wrong, we've got the solution to get them back up and running as quickly as possible. Because at the end of the day, it's gotta be the right, service level. if this, if you have the best customer experience possible from a business application, but it's not working, it you're not gonna take off. You need to have the full customer experience, including that satisfaction of when I walk up, it just works and I trust that it's gonna work when I want to go use.

Ryan: 50:59

there is no such thing as a neutral brand interaction. always positive or negative. I believe.

Eric: 51:08

we'll leave it with that one. if you're building a product, know that there are gonna be negative brand interactions and it is within your power to prevent them by thinking about all of the things that could go wrong. Eric, this has been awesome. Thank you for the insight into the, your Redbox experience into sharing, what's possible with technologies for things with Canopy and Banyan Hills. It's very exciting. We're only gonna see more automation in the future. And, looking forward to following your journey.

Ryan: 51:40

Great. I enjoyed it.

Eric: 51:42

Thank you so much.

51:46

This podcast is brought to you by Soracom, a cellular network provider built specifically for supporting IOT devices and things like connected kiosks with reliable connectivity across urban and rural areas around the world. Experienced multi-carrier coverage that benefits from automatic fail over with a single SIM or eSIM. Soracom's network is the top choice for intelligent device deployments, looking to conserve data and enable secure private networks. Now it's similar to even having your own private APN, but takes minutes to set up and not months. Reduce the size of cellular data transmissions by up to 80% and get everything up and running in five minutes or less. Leading companies trust Soracom to connect their IOT devices to the cloud and their secure corporate networks. Wherever your deployments take you experience cellular connectivity, built for IOT without a contract today at Soracom.io.Signing up for an operator account takes less than a minute.

Ryan: 52:42

What a cool time to be, in a world in which everything was getting disrupted.

Eric: 52:48

Yeah. No, it was a lot of fun. Minneapolis was actually one of the first markets we deployed into. so you were, you were a early, customer, but, yeah, we laugh a lot like. we had to build our own data center. Like we didn't have the luxury of just spinning something up in AWS and taking that slider and moving it to the right. And now I've got capacity, right? you also have an appreciation for, what can go wrong and you know that, you have to understand the underlying mechanics of the whole system. that, with those tools now at your disposal makes it even more We, we can totally be business friends. my, my number one hire is an ops person too.

Ryan: 53:28

what a fascinating time to Live through the pain of having to build the tools. And now it's ah, kids look, things you got, like all of these back in my day. be, I could be a solopreneur, I got pockets full of hyper scaling technologies and I could build a product tomorrow. You know, it's like, what do you want? Let's, let's do it. All right. Click, click, click. And it scales from one to a million and like overnight if you