Chris Villi: Working at FedEx Ground / Shipping Problems

Chris Villi has worked at FedEx Ground over the past 21 years. Chris currently holds the role of Operations Research Principal - providing technical leadership surrounding coordination of OR efforts, investment in OR professionals, and advancement of OR.

We chat about Chris' background, difficulties around solving problems in practice versus theoretically, and a current OR project that Chris is working on to optimize some of the shipping done by FedEx Ground.

Enjoy!

Transcript

0:00

Hi there, and thanks for tuning in to optimizing the podcast where we talk about all things optimization. Interested in studying optimization, interested in applications compensation, interested in hearing from current PhD students, faculty, practitioners about what they're doing with optimization. Then listen on my friend. Today I have the pleasure of interviewing Chris Villi, who is a graduate from Penn State, got his bachelor's in math and a master's in industrial Engineering and Operations Research, and is now the Operations Research principal at FedEx ground. Chris, thank you for coming on. How are you doing? Fantastic. Thank you, Anthony. Glad to be here. Of course. How was your day? Good day. Good day. I was actually on the opposite side of the table a little bit ago. We had a we have an analytics community at ground. And we had an event. We had like 95 people there and add three folks actually all three were graduates of CMU and we had a panel discussion with them. So they were the panel. I was the moderator. I was able to just pepper them, Pepper questions at them, and interact with them. And now you get to do that and I get the answer. Oh, jeez, yeah, I'm glad I'm on this side. Don't ask me any tough questions. We could turn it around if you want. Feel free. And you're a part of Fed-Ex ground. How is that different than just being part of FedEx in general? Yeah. So Fed-Ex ground was originally roadway packaged systems, RPS. Actually, when I applied back in what was it like 1999 or something, it was RPS. And I remember my parents are like what's RPS? We don't know what that is. And actually they were acquired by Federal Express, FedEx, Federal Express, the original company down in Memphis. Essentially, they have like an air network. And so they, they they fly packages. It's very tight service, fast service, that sort of thing. And everything about getting the small package industry. And that's what Fed-Ex ground does to trucking network with packages. So basically they had various operating companies and they re-branded at the time. And they said, Let's all be one family of FedEx and will have different operating companies. So Fed-Ex ground is the package, small packaged ground network headquartered in Pittsburgh. Actually, there's a dual a bimodal thing going on now with Pittsburgh and Brookfield, Wisconsin. Fedex Express is the former Federal Express, the air network. They're headquartered in Memphis, and then you have actually a freight network as well. Those are probably the three main there are other operating companies, but those are probably the three biggies. They're headquartered in Harrison, Arkansas and freights doing the trains then. They're doing like the large, big, huge shipments. So think about freight is like I have maybe like ten shipments that could fit into a truck that it may be maybe they're like 1,000 pounds. Each. Ground is like I could fit 1,000 boxes into a truck there, ten pounds each or something like that. Got it. So the scales a lot smaller for ground, it's different. It's different. We're talking about millions of packages picked up every day versus ten shipments yeah. At a time. Got it. Yep. And how how did you end up at FedEx coming straight from your masters? And you've been there for 21 years, which is super impressive. Just like Why have you stuck there? Do you see yourself continuing to stay in this position? Yeah, I do. I mean, it's a great place to be loved. The company, Good work, really exciting from an analytics perspective OR data science, like any kinda branch of analytics, it's very much emphasize and growing. Cool industry, transportation, hard problems to solve. Just good from many levels. But we actually had a guy who graduated from, I think it was a master's degree at Penn State and he might remember my simulation professor was his advisor. And we had a graduate student. I like seminar series, colloquium series. There was one for OR and there was one for IEEE. I can't remember which one. He came back and spoke out, but he came and he spoke at one of those and did a presentation on Fed-Ex ground and showed a video and stuff and kind of piqued my interest. I wanted to be in Pittsburgh and to think about like, where can I go in Pittsburgh and do our work? And it seemed like it just wasn't as well-known back then. I remember doing a kind of an intern co-op type thing and I asked the folks at the time, do you guys do operations research? And they said, Well, what is that? Alright, and so being in Pittsburgh, doing that sort of work. Was really attracted to me. I joined a group called operations research. It's kinda interesting. I'll give you a quick backdrop on that. She was sort of a mix between OR and just like general like industrial engineering. And so we did, I remember I was all excited. I came to the interview, I brought my thesis with me. There are a couple of, there was a PhD from pet and then there was another lady. She was PhD from Northwestern. And I remember they were kinda like trying to take me down a little bit. They're like, Well, just don't get too excited because you won't do OR all the time, you won't do it as much as you want. There's just a lot of other like analysis and stuff that you'll do will turn out to be a really good experience because I think it really grew me and kinda like my business knowledge and my general kinda engineering practice, if you will. And mixed with a dose of OR and a little more like analytics type stuff. I moved to another group a while later. Which kind of guy? He was really smart guy. I think he had two PhDs, one in IT and one OR computer science and alarm. And he had a group that was kind of I would call it a hybrid between IT NOR and she had some real like IT computer science background folks. And then you had a few. We had gotten other guy from Penn State who is a PhD NOR, and then to Voss who are kind of like master's degree, like Pardo, our part engineering. And so it's kind of an interesting mix of a group you almost had like a three-legged stool. You had that business knowledge with the engineering, he had the OR expertise and then had some IT firepower. And so we were able to actually build some like end-to-end applications in our own little small group with having less dependencies on a formal IT team. It's pretty neat. Yeah, pretty neat. Now, that's what we've done is we've really one team. What's that? That's great to have all of you on the same theme, just, you know, yeah, if you run into any issues, you grab the guy next to you and say, we need more, whatever. This proxy on our server, we need more processing power. It was a neat partnership and I learned a lot more. I grew in my programming skills. They're coming in with C plus plus and school back. That's what we studied when we did programming. And then coming into FedEx, lot of the engineer's program that SAS. And then I learned Visual Basic with Excel and some things as, as an engineered automate reports and things like that. But coming into this group, they were using Java and SQL more. So we had our own database server, we had our own DBA in the group. I was learning how to set up schemas and create tables. And you do all the data engineering type stuff that you know it when you think OR person, you just think about methodology and solving problems, but being able to expand your horizons and be able to do that data work, I've actually spent more time, probably my career working with data than I have with like formal solutions. Yeah, wow, I had the same experience realizing that sequel was so important. I was working and learning that skill. I thought that maybe that should be integrated into a degree in operations research or applied math. But you should have experience with databases. You should have experience with Python and C plus plus, even if you're just a math person to have a full idea, even some knowledge about servers and networking and stuff like that. Yeah, it seems like it's all there. Those are all very useful skills, I think so you think about like specializing. And there's this dream that specializing is really good. And I have a dream about that to the say, okay, like we could be a scientist, I could be an OR scientist or a data scientist. I could specialize. I could design and develop a tool and a model and that sort of thing. But then I need partnerships, right? Because I need to have perhaps this data engineers to be able to work with the data and cleansing it and find anomalies and get it prepared and do all that stuff. I need perhaps some BI analysts, business intelligence analysts to do my descriptive and diagnostic work and maybe build out my dashboards and visualizations and things like that. Maybe I need business partners who can go and sell it and train the users and all of that. But what ends up happening in a lot of cases is you do all of that yourself. And sometimes the dream of having all those partners together, I think is a good one. And I still hold to that and say there's a value there because the more, If I could build those partnerships and have those pieces in place, then if you get bigger applications, maybe I need IT partners to software developers and whatever else IT architects. But on a smaller scale, I think it's good to have a little pieces of those skills because I can do more of it and you could turn things around a lot faster. But at same time, building this partnership is really good and creating the swim lanes. Because as you can get the pieces in place more meaningfully. You can have a more robust team. And then they OR scientists could focus in. And it really specialized in that area where they're, where they're strongest. It's like this is my bread and butter. This is your bread and butter. And everybody's focusing and specializing on the thing they do best. But at same time, like I said, that's not always practical or just not always capable, able to do it or you don't have always right people in place. So having that cross-section of skills and experience, that super-helpful. Yeah, That makes a lot of sense. You kinda want to specialize in something, but then also have a little bit of breadth and understanding how that will affect the other teams. And I love to work with them in terms of, I mean, it's solving these huge problems for FedEx. Can you talk about how the different teams have to interact or how you deal with solving your problems from day to day? Yeah. I would say I always say, I just said this today in that panel discussion that we think about ourselves as scientists. And that's really a cool term. I go operations research scientists, right? When I was younger, it was 0 or analyst. And now, now the names, the titles, a lot of that is in flux. In general even with analytics. Because at analytics as a data science, is that OR is it yeah, management science, is it. They're always different than all these labels and words. But at the same time, it's like you have the opportunity to connect with different people, to focus on different things. And even as much as you say, Okay, there's a specialization and if I gotten that swim lane and I'm a scientist and I'm doing research and I'm testing hypotheses and all of that sort of thing. I always like to say there's an art that comes alongside that science. And I don t think of myself as an artist very often. Like My wife laughs at me because she looks at my writing. It looks like chicken scratch. If I would draw it, it would be like a stick figure. But art in the sense that you're, you're, you're kind of like stirring that creative pot. Like I have to come at a problem with a curiosity and creativity to be able to solve it in a meaningful way that intersects and connects with the business. Because it's not always, well, it's never usually a textbook type of a thing. It's like if I come at a, at a textbook problem, oftentimes, my job is just to formulate it correctly and I get it marked correct or I thought through and I formulated it and that's good. But like in business, there are so many other things that just make it ugly. And on textbook like that, number one could be, well, we have all these rules, but there are all these exceptions to the rules. The things don't fit tidy and it's nice little box. Another one might be. And this is what you were alluding to in the question. Sometimes the problem is just really huge, ugly. And I could formulate it really nice, but I can't solve it per the formulation because it's unsolvable. So just to give you an illustration, and I talked about this a little bit offline, but this optimization problem we were working with, we had over, if you really stretch it out and think about it, this is a network problem. And its think about it from a matrix standpoint, you're talking as a dense matrix on a graph. I might have tens of billions of combinations. And actually when we calculated in the mid 20s or something 20 plus billion, that was already like cutting off certain options. So I would think if we really like spread that thing out and said what are all my possible combinations I could build? I might be even more. I maybe 4,050 billion, but that's unsolvable. We're thinking like this is a really great mix integer program. It fits like a classic. Maybe not classic. But if you think about the multi commodity network flow, there's a pretty strong connection to that classical problem. Maybe there's some side constraints, some other things going on that kinda fit the situation. And it's not exactly that traditional thing, but it's for the most part fitting into that. But how can we actually do that in a way that's solvable and meaningful? Yeah, definitely in terms of the multi commodity flow, just, I guess like general setup. What does that look like to you? Do you have a bunch of trucks starting at it? Depot that have to carry different packages to different places. Like what does the formulation look like? Yeah, So that's even a different problem than the one I'm talking about. Okay. Yeah, I'm thinking about is a package sortition problem. So it says, if I pick up my package, Let's just make it up in Anaheim, california. Alright everybody, Disneyland. And I gotta deliver this thing up. Masses here in forums. Yeah, you got to deliver it in Boston, Massachusetts. So I know when I pick it up, the place and the time, we're thinking about time and space network. I know the geographic location, I know the day and roughly even the time of day when I pick up and then my other end point, source sink kind of thing. I know my destination, Boston, Massachusetts. When is it due to? Maybe I picked it up Monday. It's due on who knows, Friday. And it's due on such and such sort is a sub-component of a day. Yeah, it's one we sort those packages. So I know those. What I'm trying to solve for is the in-between points. Where do I sort those packages? In the intervening points in the country? Do I sort it in one place somewhere? Do I sort at two places at three? If so, where and when? And the reason why I want to decide that is because if I put that on a trailer, I need to know where do I load the trailer and then where do I unload the trailer? It's consolidation D consolidation. And theoretically, if I could put if I had enough packages to go straight from west to east coast and just build a full trailer and do it. Why would I even stop in-between them, unload it. I just load it over there and unloaded over there. I just wanted to drop one route at well, there could be a separate problem which we're doing separately in terms of like, well, that's, that's a whole other conversation, but it's interesting is how many problems can you solve it once? Yeah, because if I do the packets rotation and the trailer routing altogether, both of those are huge problems. If I did it all at once, that would be pretty much impossible. So how do I decompose down problem by problem? Well, maybe I saw this rotation and then I saw the routing. After. What we're focused on in this one is actually just where are the points along the way. Now we do bleed in a little bit of routing assumptions and stuff to give it a flavor so that it could drive cost. Because it is a big cost to say how do I pair trailers and route them? So we have some flavoring of that. But at this point we're just trying to say Where do I unload and load trailers. So theoretically just thinking a simplistic heuristic, great. If I could send at point a and say, do I have enough packages to build it all the way over to there? If yes, go ahead and build it, don't put anything in between. And then how I pair that and stuff that we'll figure that out. But at least I've built a nice full trailer, right? If if no, then okay. Can I get to each side? Each facility has it's like kinda parent that's feeding it. Can I get to the parent of Boston and get all the way out to there and build a full trader that maybe it's building 20 facilities, but is that good enough? If yes, maybe I do that if no. No. Maybe I'll build it somewhere closer like in Los Angeles. But we have to figure all that out and we're trying to do it in a much more dynamic way. There are simple ways to think about it. I can think about it in that kind of if-then structure. Or I can think about it as I have these millions of combinations of how these things connect together. And I'm trying to get a meaningful consolidation so that I can fill up these trailer because I don't want to splinter it out because the more I splinter it. Well, let's put it this way. The less packages I put on a trailer, the more trailers have to build. We're trailers have to build more trips. I have to take more trips. I have to take more money I have to spend yeah. Same thing on the flip side, right? If I if I push everything in one place, well, that's going to cause a problem because each facility has rotation capacities. They have only so many doors where they could build other facilities. You have constraints on your network, these capacity constraints. But then you're also trying to drive cost down. And there's a battle that ensues that says, On one hand, if I funnel everything together, it's going to put more packages on buildings and sorts. And that might mean I have to spend more on capital to expand that building and the near future. But if I splinter it out and I start skipping places, well, that's great. But as soon as that causes me to to build more trailers and it gets to splintered, then I have a problem. So it's like how do I balance that in-between? How do I splinter in meaningful ways but still get good full trailers? And I can't do that with my own brain. I have to have the model try to do that for me because of just the combinatorics or two insane yeah, to try to do. So you're hoping some mix of LPs, IPs, and machine learning like what kind of techniques are going to help you solve these? So your brain doesn't have to do it. Yeah, so traditionally this would be more like commodity type of a model. And so we formulated that like super quickly and easily, right? But then the next question is, well, how do you solve it? Because you have billions of options. So this is part of that art is that creativity is say, how do we build a good solution in a meaningful way, because it's its solution quality versus speed. At the end of the day, if I'm in a Cut this. If I'm going to solve this thing, I need to solve it in a meaningful amount of time. But I also don't want to give up too much of my solution quality. So first thing we did is we said, can we apply common sense to this nice? And what I want to do is I want to, and this goes for anybody anywhere. The best starting place is to know your business problem. To understand the problem, to understand the problem structure, to understand kinda the levers of what you can give and what you can take. The common sense comes in and says, Can I, first of all write out of the gate, play with my solution space? Hello, not play with it, but can I, can I play with it? Can I sculpt it in such a way? Better word, sculpt it right here's my artistic thing coming back. How do I sculpt this thing so that I can meaningfully shrink it into a smaller problem and make it more meaningful to solve. And if I could do that in a way that doesn't give up any solution, quality or minimal. I've done a good job. So here's our analogy. Think of the puzzle. I got 100 piece puzzle that I need to put together. I opened up the box and there's 500 pieces. I only need 100. Oh no, I only need 100, but I got 500. So ideally, I can throw away 400. But which ones? Yeah, right. So are there smart ways to look at those pieces right out of the gate and to immediately start pitching some away. And maybe I don't get rid of 400, maybe I get rid of 300 to 50. And now at least I have a smaller set to work with. So okay, Well that one's a color that doesn't seem to match the color scheme of the picture. I'm getting rid of that or that shape is different or whatever. And you start getting rid of like, I don't know, let's call it the low-hanging fruit. And you kinda brushed out away and then you're left with a smaller one that's now more manageable to solve. And so we did that sort of thing. We said, Okay, common sense. So if I'm starting from, take our example, Anaheim have gone to Boston. I'm looking at points along the way. One of the things I could look at e.g. is shortest path or maybe not shortest distance of the path. How far am I going out of my way? How much their acuity. So if I'm saying okay, I'm gonna go from Anaheim, California due north all the way up to Washington or Oregon or somewhere like that and then all the way across the Boston. Well, maybe I've gone pretty substantially out of my way. Maybe I could just get rid of that combination and say, Yeah, that's not very practical, let's not do it. So I could think about that from a time and a space perspective. I could think about acuity and say, I'm only going to allow so much acuity. And I'm gonna get rid of all other combinations. I could also do that from a timing perspective and say, Look, if I have I'm just going to make it up. I have five days to get from a to b, but I can get there in three days. Do I really want to sit at the first point for two days and eat up all my Slack. And then all of a sudden, now everything's hot and it's tight. Because now it's just think practicality again, risk factor. If I add up all two days worth and then everything's gotta go from there. What if I make a mistake? I'll sudden I cause a service failure, right? So maybe I don't wanna do that. I want to let it sit for two days. Maybe you let it sit for 8 h or 12 or day or whatever, right, x. So I can then prune everything else beyond that. So those sorts of things to say from a timing, from a, from a geography. Another good one is aggregation. I could, I could precalculate some of this network graph like the arcs. And I could say, look, I've created all these combinations. But now I can look at a particular arc and say, Hey, if I could assign every package to that arc, that possibly could go on that arc. What's my aggregate sum? If it comes out to be like, Oh, that's like a quarter of a trailer. Okay. Do I want to even keep that as an option or don't want to go for other higher probability, higher potential areas. So if I say, Hey, I just had this odd ball arc that can only get a quarter of a trailer. Maybe I just get rid of all the possible I'm going to call them paths, that point, point-to-point with everything in-between, I'm gonna get rid of all those ones that possibly go through there. Those combinations, get rid of those, throw those off. Those are the puzzle pieces I don't want. So what we did was we came up with these actually, there were nine different ways that we did pruning. And in the, in the data it's a prune as you go approach. So you could build the whole thing and then you can prune it, or you could prune as you go. For data reasons, we did prune as you go, but we came up with nine different ways to do that. And we took 20 billion down to 200 million. Now, whoa, it's funny when you hear 200 million, you're like, Oh, that's crazy. 200 million a lot. There's a lot. But if you think about that as a percentage, that's a 99% reduction and problem size. Now, 200 million is still unsolvable. You can't just say, okay, Gurobi, like here's 200 million variables, go solve it, right? Yeah, it's not solvable, but It's shrunk the problem that we did a lot of cool without decreasing the solution quality. That much problem, yeah, we did a lot of experimentation on the front end with our printing approaches, e.g. instead of k, If I have a test scenario that has all possible paths, I built all these different printing approaches and I run them, and I run each of them. And I tried to run them. The optimality. I say try because not always are they going to converge? You'd cut them off at a certain point. But I can compare my, my baseline of, here's all my one with all possible combinations to my 500 piece puzzle. And then here's my one with 100, here's my one with Ford. And then we compare it. Okay, here I got rid of 80% paths. I solved it in 10% of the time and I only increase my objective by 3% or something. And then you can say, Oh yeah, that's really worth it because it was way faster. And I gave up 3%. But who cares? Yeah, because in reality, you don't always need a perfect solution. You'd need a you need a good solution. Like at the end of the day, does my operation No. If I implemented the perfect solution or not. No. But if you don't know, I'm sure they don't know, right? Do I really need the perfect solution, or is it really just the gauges? Is it better than the one I'm doing currently? Am I moving the needle? Ultimately, that's where we want to go. And so that was one step in a way that makes sense. And you talk about part of that. So you're saving costs to, or you might be giving up some costs because it's probably too big to save on time and solving the problem. How quickly do your problems need to be solved? And are they solved again and again every day, like can the routes change on the fly? What's the time look like for yourself? Yourselves? Yeah, here's a really interesting one, right? So we're just talking about 20,200,000,000 now, let me throw it to the opposite side of the spectrum. What we have implemented right now, we have problems that we put in a runtime limit of 8 s. I hope you guys are small. We originally had five. We bumped it up to eight. And we're solving like 300 problems in 30 min. So whatever, ten problems a minute or something like that. So what we wind up doing is we went a different route and this is kind of a customer in mind type of thing. We said, Look, we want to give a tool that people can use and can find usable. So how do I make an inroad with the user and what is the user doing and what do they need and how do they, what's can be a help to them? And so what that turned into is kinda real scale back. So instead of saying, look, I'm going to solve this entire network with 200 million combinations. Well, you have to think about the implementation at like what does that look like? If I try to solve for 200 million? It's like a soft from scratch, a clean slate, a Greenfield, whatever you wanna call it. How do I actually implement that in real life? Because in real life I have an operation where I'm doing stuff day to day. And if I come in and say, oh yeah, look, I solve this problem. And we got to go make 40,000 changes tomorrow. Yeah. How do you implement that? How do you do that? Because those changes have real effects to say, well, we have a contract with these many drivers over here. And the changes you've proposed is going to change the number of this or that and it's impractical. So we said is let's go from the other end and let's try to be as practical as possible. So let's try to take the smallest increments of change and introduce it and then let that grow over time. So the idea we used two words, interactive and incremental. So interactive is to say, I want my user to see this as a meaningful tool that I can interact with. This isn't taking your job. It's not telling you what to do. It's coming along your side. It's a tool in your tool belt, In your interactive with it, you're going to tell it, Here's what I want to solve, here's how I want to solve it. And so it's parameterized. And what's really cool is, I think of a lot to think about like a Google search. And you type in a word like here's the thing I want to search for, and then it optimizes searches and it gives you options. That's kinda that mentality that were coming out at. We're saying first of all, this was a change of paradigm for us as a paradigm shift. Because we said instead of just solving the problem for objective, the way that we formulated it, Let's get out of that paradigm for a minute and let's allow for different types of objectives based on the user's desires. Let's make it a little bit more user-centric and say, What are you trying to do? What do you want to do? And then we'll have an option menu of options that says, okay here your potential objectives, one through ten. And eventually they might be checkboxes to where I could run multiple of those together like multi-criteria. But just to start, what do you wanna do? A, b, c, d, or e? Right? Do you want to fill up trailers as full as it can be? Do you want to get volume off of that sort or out of that building, like what are you trying to do? And then I could use the same infrastructure with maybe a little modifications to solve different versions of that problem. Nice. Would that just be scaling your Like coefficients on maybe different variables in your optimization problem. Or it could be doing a variety of things with constraints and it would be different. So let me give you an example. The one that we've been working on so far has been like, let me add a new direct load. So direct load would be like, I'm in a, normally build it from a to B to C. Maybe I skipped B and just go straight from a to C. So how can I build something more direct? Give me a new arc. This gives me, say it in graph language, right? Give me an arc that's not being used. And let's try to use that and take volume off with some other arcs. And the idea is we're gonna get it, they're more directly, we're going to maybe save on rotation or something like that. Yeah, so that's, that's the use case where we're solving for. So when we do that, what we're doing is we're taking local subproblems and we're saying, okay, first of all, here's my data exercise again. Like let's look at all the input data and let's try to first query and see whether different options that I even have. Like maybe there's 100,000 of those things that are built in the preprocessor and everything else is pruned away. But even after all that, pruning out of the 218 million things, like those are end-to-end paths. If I look at those individual arts, how many of those are not being used that meet this criteria of a direct load that I might want to try adding. And okay, there's 100,000. So my next step is then out of that 100,000, how many can I try to send to my optimizer in the amount of time a lot because that user interface, not only do they have the ability to choose their objective, but they have the ability to choose their parameters. And part of the parameters is, one of them is, how long do you want to let this thing solve? And if they say 10 min, well, I got 10 min. And I have a sense I could do about

10: 34:41

10 of these per minute. So okay, that's, that's 100 runs. Maybe it will send a little more just in case we get lucky. So we're going to get, I need to get 120 of those, 150 whatever and pull them in. Don't want to bring too many because that's extra data storage and stuff that I don't need it. The more you have the clunkier get. So I want to build, I want to bring that in. And I want to run these things. And what we're doing is we're essentially taking that and we're trying one at a time, and it's not a simple math enumeration. Each one is an optimization because I'm saying, if I try this one, I want to open up all the different variables that come in and out of there. And also keep the incumbent of like what do we currently doing? And what about if I turn on this combination? But that combination has upstream and downstream impacts. And so that opens up a lot of variables, even though it's kinda localized. And say, boom, run that through the optimizer, give it 10 s or 8 s. Let it figure out, is it worth it or not? If it's worth it, it's going to tell me, here's how much money you can save. Here's how many changes you have to make. And it's a finite number, because that's another parameter. It's how many changes are you going allow the model to make. And let's just say maybe they say ten. So ten would be like, I got to build this new one here and that new one there and get this volume in from here. Like a small subsection of the network. Yeah, it's manageable to do maybe. But back to the Google analogy because I ran, I had 10,000 to pick from somehow. And we could go into this in a minute because that's where the machine learning part comes in. We actually pull in. I have enough time to run 300. How do I choose 10,000 or 100,000? Choose 300. How do I do that? That's internally to us. Then the 300s, those all go to the optimizer. And out of that, Let's say I get, let's just say I get 200 that have good cost-savings. Another hundred or don't do anything, it's zero savings out of the 200. Now, which ones do I show to the user? Because another parameter they said is, how many do you want, like in the Google search? For Google doesn't ask you this, but we do. How many recommendations do you want to see or how many options do you want to see on your screen? Yeah. And they could tell you that, well, show me ten. So when they hit the button and they wait 10 min, they should have a population of ten different options to choose from when you get back. So, come back to my example. I get 100,000 to start with. I picked 300, I run this 300, I got 200 good ones, but now I get to pick my best ten to serve back to the user to say here your top tend to choose from. So we have a lot of like logic at each step of that process. And the optimization is that as one piece. But we have to figure out those wastes to this other pieces too. That's really interesting. And who, who's the user in your case? Because like do they have all this knowledge about what the optimizer is doing and everything that's happening or what did they do after they get those ten possible changes from you? Yeah, this is where, like other types of analytics come in. So when I think about analytics, I think of it as. Broad umbrella. And you probably have heard this paradigm before, but you have the descriptive analytics. Yeah. Some people use three-pronged descriptive, predictive and prescriptive. I like to break out the descriptive and have diagnostic as well. I see that as a separate branch, but nevertheless, those descriptive ones that maybe sometimes people aren't. If you're thinking in an OR, or data science mindset, like maybe you're not as interested in the descriptive side. Butt. Those are very practical and there's an entry point there. So when we're thinking about this tool and application, we gotta have dashboards, we gotta have visualizations. We got to have ways for people to access. The, you gotta, you gotta turn data into information. How do I tell the story, right? Because it's not just a dump of here's the outputs. Good luck, right? But now I got to, I got to know like, what are those outputs mean and how do I pull them together in meaningful ways so that I can identify KPIs and key performance indicators metrics. I can maybe bring this up on a map before and after snapshot. So we have UI, User Interface, UX, or user experience UI UX designers and people involved and thinking about what does that look like and what does the user need to see and how do we have an experience that's meaningful for the user? That, and part of that, again, you're back to your art of like, how do I do that in the most meaningful way? And how do I get from point a to point B with the least number of clicks? And way that, that's kind of intuitive to the user. And yeah, it's, it's its own challenge. And I know the folks that I'm working with who are focused more on the modal logic. We're not into the nitty-gritty on UI UX design, but yet we're kind of consultants in a way because we understand the data and we're producing the outputs. And because we're kinda thinking of it a little bit more end-to-end. We don't have the BI analysts and the data engineers and all the other teammates there where we're thinking more on the front end, where we're building a model. We're thinking more on the backend. We're talking about those types of interfaces with them. But at the end of the day, engineers have to plan. So you could have people who are centralized or thinking broadly for the entire network. You can have people who are localized, who are thinking about, hey, my local area, What's my next change I want to make just for the Midwest or just for their dates or something. Yeah. And how can you give them that tool so that instead of them going out and writing their own stuff and trying to figure it out and everybody is doing it different ways. Now we have a centralized intelligence and ability to use a sophisticated tool to serve up these options to you so that you can now go in and analyze them. These are still really important, right? Because they gotta be able to analyze the outputs and say, I got these ten options. Maybe I don't pick number one even though it shows to be the greatest cost savings because there's some kind of knowledge I have about the way we're operating that that's not as easy to pull off for, that's not as practical or whatever. Because I told you I want ten options. I can look at that and evaluate it and then I can say I don't want it and go to the next one. That's the interactive part. That's super interesting because I'm all the way over here and PhD land where it's like we think about a theoretical problem and we solve it. I also like to do some practical things and maybe like you're saying, prune off some solutions that we know it might not be the best. Like, I'm fine with doing some heuristics work and then having the solutions having to be useful and having to be practical and the real-world, or there might be some uncertainty or are different things going on across all the different regions? Yeah, that's like a total next layer that I feel like we don't get too much experience with in the classroom, but, but definitely sounds like it's very useful. Yeah, yeah, it's really fun. I mean, we have to live in that tension of practicality and academics, sophistication and how do we marry the two together? And it's never going to be like perfectly on one end or the other. It's gonna be something that combines both the certain degrees. Hopefully, right? The machine-learning pieces, kinda cool to me about that, a little bit about that. Yeah, that's that's kind of a newer piece that we've put in there. So we're thinking about this optimization and saying back to that idea of like, hey, I could pick, I have 100,000. Options I could run, I can send to the solver. Well, if I did that, even if I limit it to 5 s each, yeah, we did it. We did a calculation at one point. I don't know. I'm just using 100,000 is a place or I can't remember who it was, but it was like if you took all the options. Yeah. It was like if you took all the options and you sent them there and you just told them back-to-back, not parallel multi-threading and that it would take a year to run the whole thing, even though it's like a five-second time that I got to figure out like which ones am I going to try to choose? There are some ways to user can help narrow that down because we can ask them questions like geographically do you want to solve for the whole network or not for the whole network, but what's your scope? Is it like network wide that we can pick anything? Is it like in a certain region or certain smaller area? But even after all that, you have your options. And then when I, when I choose the options and then send them to the Solver, what we found was if you just choose randomly out of that, okay, I could run 300 iterations and I just pick a random 300 out of my group. We found that 80% of them came back saying, there's zero changes that we're going to make another words. The current solution is the best. Yeah. Because we're always starting in. I don't think I mentioned that, but we're starting from a current state. We have a current plan. And that becomes the starting point. What we're trying to do is find a change in the way we do things. So we're not starting from scratch, we're starting from a current plan. So when we send in this alternative, it would solve and it would come back and say, No, don't change anything. This is a $0.01. So if I run a model for 30 min and I get through 300 iterations, but 80% of them come back as zero. Okay? 300 times 20%, I still get 60 to choose from. And maybe the user only wanted to see ten. So I still have enough to serve out. But thing is those other 80%, it's a waste of time at the end of the day because I knew that they weren't worth doing. I wouldn't do them. If I could get that 20%, that's 60 to be bigger. Well, if I can get 80 good ones to choose from her, hundred or 150, like the more I have to choose from. Theoretically, if I'm only given the use of the top ten or the top five are the top three. The more that I have to choose from, the higher probability, the higher likelihood that my top three are gonna be overall better. So what we did was he said, Hello, This is a great potential opportunity to use machine learning. So what we did is we said if machine learning can be used to predict the likelihood that the optimal, that the optimizer is going to say, yes, Change, make changes and save money or no, don't make changes. It's a zero change, zero savings solution. If I can predict that with any meaningfulness using machine learning than that 20% can increase to some higher number. So what we did was we, we tried that and we said, okay, let's train the data and we went, we solved like thousands of iterations and the optimization, yeah, and we draining on, we're like, Oh, what, what kind of characteristics for the problem are you looking at? Yeah, so we did a bunch of feature engineering. There comes again like your business knowledge, what you think you put your engineer hat back on, he say, What is it from a business perspective that might make me want to build a direct or make it worthwhile. And so we started thinking of all these features like, well, the distance, you are away, right? Like Is it close or is it far away? Because that has cost implications and where you're located in the country and how many potential trailers or potential packages you have. Is it something that could potentially get on the rail or drive on the road? So they were like 14 different potential features and solve the problem. Did, did some feature engineering and saw like which ones were meaningful, which ones weren't. Narrow. That down to four. Tried some different models and approaches. But first of all, it's a classification approach to say, I'm going to treat it as a binary. It's either a good or a bad. If I make changes and I saved money, good. If I make zero changes, don't save any money back. So I had like a binary prediction, binary classification. It's either a yes or no, good or a bad. And solve all that. Do your feature engineering, do all that. So at the end of the day, what was really cool Is there was some predictive power there because the random scenario that had 56 out of 300, let's just call it 20%. By the time we got done, the best machine-learning algorithm tuned up with the feature engineering. Got us 199 out of 300. That were good thing. Yeah. Yeah. It's pretty cool because there are different things that you look at. You look at your confusion matrix and you say, Okay, I got my precision and my accuracy and so on and so forth. And I think precision was our main driver because it was like Trying to understand my positives and false positives to say like I attended today, I'm trying to predict the good ones. Yeah. I want those to be as good as Paul. I don't care about the bad ones because look, if I got 100,000 options and I can label all these and my, my algorithm, even if it really has 20%, okay, well that's 20,000. Then I have to pick out of that 20,000. I say, okay, I've labeled those 100,000, 20,000 of them machine learning things as good. Well now I can pick 300 out of that 20,000. And if we were perfect at predicting, all 300 would be good. But we're not perfect. So I got 200, 300, which is way better than my random 20%. Yeah, and so if you look at the math, well, not the math, but just the basic metrics at the end. When I talk about, if I were to pick top three, the total savings from my top three more than doubled. Now, because now I had 199 to choose from rather than 56. And out of that, you wind up finding some better ones. So that's your total possibility for cost savings. Yeah, it was, it was really cool. So it's essentially like one model to predict the behavior of another model. Machine learning. We're still using the optimizer, but it's just the machine learning is helping us think about like what am I going to send to the optimizer? That's very cool how you're using them, like hand-in-hand machine learning to help. I'm guessing the other ones in LP or an IP, it's mixins your program. Yeah, wow, that's really cool. The machine-learning people would be very happy to hear that. That's working nicely. Something that we had on, we had been mostly on the last episode who deals with when you have to solve a problem, like over and over again every day trying to learn something from the structure of the problem that you're solving. In this, this sounds like maybe a similar application of that where you're training on the past instances and that's going to tell you something about what's going to happen with the solved for today's instances. That's our hope. Yeah, I mean, we're not in production yet. But the original research that was done, the guy who did it, he ran thousands of instances and he trained himself and then he did the testing and figured it out. But the way it's going to be implemented is whenever people have this tool and they're going to run it, perhaps weekly. And there could be over 100 people who've had this wouldn't be installed, I guess now everything's through the web. But David, 100 people who have this application that can be running it probably on a weekly basis. But what we could do is now we can collect the information that comes from their runs. So we're going to actually create a new master scenario weekly. Say you and I are both users. You go and clone this scenario and you got a scenario one-on-one, Anthony and I have a scenario one-on-one, Chris, then I'm working off mine, you're working off yours. And then anything that's saved into the database. Like if you run your model and you say give me my top five, blah, blah, blah. And you run it for 10 min and it runs 100 iterations. We're going to save the high-level data from this 100 iterations into a table somewhere when, after you run it, right? And it's gonna go to the table, I'm going to say, Okay, I'll give it 30 min. Maybe I've run 300 iterations. You're going to take my 300, save it to that table. So what's going to happen is we're gonna get a bunch of data that we have from people's runs. Then we're going to have a job scheduled, say over the weekend when nobody is doing anything, that job is gonna go and grab all that data that we wrote into the table. Or maybe if not all of it or it'll figure out some subset of it, whatever is meaningful for us to do. Yeah. And then we're gonna take, and we're going to retrain the model so we can have like, you know, instead of having drift and also I think we're learning from the scenarios like what worked, what didn't work, what was good, what wasn't good. And then hopefully we can get even more sophisticated on what we learn off of that and how we better make decisions from it. That's really cool. Have you found that in terms of, I mean, you're, you're saving costs. Planning all these routes are the deltas, big words, this machine-learning thing helping make the deltas even, even bigger. Are you trying to keep things afloat? Or like say you're spending $300,000 a year, super small number doing all the routing. Like how much value-add is the current processes having or you kinda trying to keep it around the same baseline. Are you still working on saving a lot of money? There's different ways to go about trying to make an impact in an operation where we started and we were talking about like 20 billion in this huge thing. Like you could think about it theoretically and say, Oh, well, you know, if I would take my current Full network and try to solve it. I could come up with some theoretical idea of like here's how much dollar savings so I can get, here's how much percentage if I do it that way and that could look really incredible. But what it's saying before as I can, in realistic kind of practice, that's not always gonna be, it's never gonna be obtainable. What we did, we kinda came the other direction. So let's start in this problem with a, an entry point. So I got these 100 people who are localized people. If you're going to run it once a week, let us just say that you can run it and you could find one recommendation. Yeah. If your recommendation says I can save you a 1,000 bucks a week, doesn't seem like that much. But you multiply that out by 52 weeks and then that compounds by 52 weeks the next year and the year after that. But if you do that every week, okay. I had 51 52,000 savings from that first decision. Then the next week I got 51,000 if it is just do it that way and then 50,000.49 thousand, so on down to 1,000. You do that every week for a year, like just you have saved a tremendous amount of money by a 1000-dollar a week thing, which is like in the scope of operational costs, that seems very small, but there's a real possibility for like a compounding effect over time. Definitely. Yeah. I mean, where I used to work, I used to work on a trading team at a hedge fund, ended. I was working on just making these changes that might save exactly what you're saying. Like a basis point, a day or something or a very small amount of $1,000 a day. And then like you're saying, incremented, that helps so much. One thing with ours was maybe if we're making these incremental changes, it does level off to where things can kind of run status quo and you're not you're not getting back as many changes. Do you see that the number of possible changes is decreasing or slowing down? So we're not in production yet. But I mean, that's a good, that's a good thing to anticipate and to ask from an anticipatory standpoint. I don't ever see that being an issue because we're the world, the culture or the business is never static enough that you would ever catch up, right? It's like trying to shoot at a moving target, like you could make a change. But the business is also changing. And as fast as people are going to change and implementing things, there's going to be new business opportunities and think like just even from a predictive forecasting standpoint, we have a team that does like pickup package forecast and their time series information that they're looking at. There's a lot of predictability there. But like fast-forward the COVID, and all sudden, like now, the history is different than what anybody expected. The history has changed and that changes the business and they're just things all the time and a growing and dynamic workplace. Can you talk more about COVID changing like FedEx is operations. Just, I could just say generically. We had always had like a peak and a non peak. So right now we're in peak. Peak is between Thanksgiving and Christmas, right? Mom's got to get her her packages. Shipping increases drastically. Right. And so you do a lot of planning for those three weeks in a year because there are significant capacities, all sorts of things. Buildings route like there's so much that goes into it, then your non peak is pretty standard. And so if you think about it as like a line graph, you're kinda cruising along and then boom, you've got this big spike, right? Well, with COVID, it was kinda like a never-ending peak in a sense. It kind of kicked up. And if think about e-commerce in general, e-commerce kind of really went into high gear. Which is reasonable if you think about the situation of people doing more online shopping and stuff like that. Yeah. When you're quarantine, you can't go to the right. You have to buy online, right? Right. But the mix of business changes to like, we have business, business, business to residential, that's sort of mix changes within the changing culture and economy, those sorts of things. So I've been there for 21 years and ever since I've been there, even if I had the job, a job that did the same thing over and over again, like C or planning the same way every year. In a dynamic and changing and evolving workplace. Even that changes, even if I'm doing the same kind of planning every year. There's different flavors to it, right? Like you come to this problem again, I mentioned you earlier that we have the ability to have different objective functions. And even we finished this first use case and say, Okay, you got your direct from a to B. And if people were doing that, doing that, let's say like nothing changed everything. It's kinda static. And now they start to get diminishing returns. Well, there's all sorts of other use cases that are down the line down that those radio buttons lists. Okay, what if I want to solve for this or that or the other? Like one of the big things now is like with all that increase in surge in volume, now we have well higher utilization, right? So if my capacity is x and now my utilization of how much I'm seeing on a given day at a given facility location. Maybe a lot closer to x than it was formerly. But how can I engineer that? How can I use analytics to now move those packages around in a smarter way so that I opened up that gap a little bit and I give myself a little bit of relief. Those are big problems and those get more complicated the more complex the network becomes. And the more maneuvering that we do, the harder it is for people to solve that manually, right? And so now I have to say, well, I'm just going to like, do this work around. Well, maybe that was easy ten years ago, but maybe it's harder now. And we could use this type of sophisticated solutions for the model to find things that maybe are outside of the way my brain works just because I can't deal with that many combinations. Or who knows? I mean, you know this too, but I think it's kinda rewarding actually. What's my brain can deal with them. Or the idea of like even changing paradigms, like what happens in the model suggests something that nobody's thought it because it's stretched our paradigms and it's different than the way we've done things. Maybe that in some cases will be rejected because he can't do it that way. Maybe in other cases, it puts us in a whole new dimension of how we think and operate. Like maybe you can do it that way. Yeah. Maybe it's just Hey, we just never thought about that because of whatever we were locked into this way of thinking. I think just even for that problem. And then that's just one problem of many. There are so many different things to solve and do. So yeah, we'll never run out. We're just tip of the iceberg. Nice. Well, we play a bit of a fun game here on our podcast called name that stat. I know we've been talking about the problems of FedEx being quite large sometimes. So I have some statistics and I'll ask you questions and you just answer a number, maybe a general range that you think the answer might be. Okay. So how many Fed-Ex ground hubs are there in the US? 30 ish. Oh my gosh, You're good. 39 was on your website. And what is a hub? A hub would be a consolidation facility. So think about it this way. Here's the simple model supermodel be a hub and spoke model. So I have facilities that pick up and deliver packages. So let's say I have a Pittsburgh station and there are P and D are pickup and delivery station. And maybe I have 40 drivers who have different routes. So you have a route and you have these 15 zip codes, and I have a route and these ten ZIP codes. And you go through these neighborhoods and you drop off, you deliver and pickup pagans. And 40 of us and we all bring it back to the station at the end of the day. At the end of the day there's a sort where they sort two packages, you unload them off the advance, you sort them and you put them on trailers. So that's called our outbound sort. Well then those trailers go up to what's called a Parent Hub. So maybe Pittsburgh is one station and there's 30 or 20 or whatever. And all of them go to Columbus. And Columbus is apparent hub of Pittsburgh and all these other ones. So all of these ones that have 40 drivers or whatever and they unload their vans and put them on trailers. They go to say Columbus, the Parent Hub. Columbus is then a consolidation point to take all those packages on load this trailers, mix them together, and then you have your hub, the hub 40 to 40309239 network, right? And they get them to there. So you have your origin hubs, building your destination hubs because you have that consolidation of volume you have, you have your, you have a Fuller, fuller, denser kind of build. And then you get to your destination. So Columbus gets it from those other 38 places. And then it's their building, everybody in Columbus. And then you unload those trailers and put them on the vans and then the vans go out in the morning and deliver. So that would be the hub-and-spoke networks. Everybody has their parent. And you build from the origin, the origin hub to the destination hub to the destination. So that would be, this is very simple model, right? And then I like how it touches the model that I mentioned. Just as a framework for thinking, yeah, tie it. And so if you started with that as a very simplistic way to operate, you would operate, right? But then at some point you say, Wait a second. Instead of me building to Columbus, what if I could build? Anaheim directly from Pittsburgh. Or let's just forget that and say Anaheim is parent is Rialto, California. What if Pittsburgh, instead of going Pittsburgh, Columbus, Rialto, Anaheim. What if I could build, say your say router has 20 stations? What if I could build those 20 into a trailer straight to reach out to and I feel that trailer why unloaded at Columbus? Yeah. So now I could skip the origin hub, go origin straight the D hub. Or maybe I built all to Columbus and then Columbus has enough to skip three outs and go straight to Anaheim. And until it's now going from. So you see how a basic hub-and-spoke construct origin, origin, hub, destination of dust that fills a basic network. But as volume grows and density grows, now I have opportunities to say, hey, I could splinter off and build origin straight to desktop or origin hubs straight to dust, Kip assort point along the way. Or if I'm really if I really got a lot, I could go straight from origin, the desk. I started getting more and more combinations. And that kind of opens up the network more from a combinatorial standpoint, then you'd go and say, well, now what if I change what I, what I think about who can sort of package in-between the hub is like the big consolidation point. Yeah. Like what if I could use a station as a hub and so many more different? Or what if instead of building it to Columbus geographically, what if I built somewhere in the middle of the country? So there's all these other questions that you can do to get trickier and to build it in different ways. But yeah, there's what I have is 39 hubs. There you have it. We have one other fun game for you called this or that, okay? And I will give you two things. And you can choose one as your preference to explain why you like that. All right, Would you rather be president of FedEx or President of the United States? Fedex. That's probably a hard job, but maybe simpler than that country. Do you prefer heuristics or exact methods? Oh, wow. So this comes back to where we live, right? So where I am. I think in my idealistic mind, exact methods are wonderful, but nothing is ever an exact method right there. A lot of different dimensions, even that we didn't get to talk about, about the problem we're working on. Like we broke up into large neighborhood searches and we had to do all these other things. And even just like in the world, you have convergence. So even if it's an exact method, do you get an exact optimal answer? Well, only if it converges the full way, but with our time limits like hey, sometimes I saw, I have an eight limit, eight second time limit and I solve it in three and it's great. I get a perfect answer. Sometimes I don't. So, yeah, I think I'm going to have to go with heuristics just out of pure practicality. Because where I am, it's often impossible to any exact method in a real implementation. That makes sense. I figured that would be your answer. Well, what's cool is I can take the chocolate and vanilla and make a twist. This is what we're trying to do. We're trying to take exact methods and embed them within heuristics. So how can I use this iterative heuristic to run instead of a pure heuristic? Can I lean into the backbone of the optimization model and use that as the backbone of my algorithms. So that's exactly what we're wanting to do, is like how can I use a greater level of sophistication as the basis for my solve. And yet at the end of the day, it's still a heuristic to some degree. Yeah. Fair. As a good rest of the two of them? Yeah. Because I had to cut down the problem size. I have to break it up into large neighborhoods. I have to solve iteratively all of these things I have to do that makes it not as exact, but at same time I still if I can get some improvement, I went through my last one is for Christmas presents. Are you a dad? Yes, sir. I assume so. Um, so this is maybe a dad kind of present choice. Would you rather have a scented candle or a striped tie? I take a striped tie. Very nice, very nice. What kind of colors you want on that? Oh, man. Wow. I have a go-to tie. I mean, I got several go-to ties, but one of them has orange stripes like orange and blue is pretty good combo. That is a great guy. But as a FedEx guy, I'll have to say purple and orange, brown and yellow. That well, is there any other anything else you want to talk about? Any other career advice you have for people or anything else. Yeah, we could probably go for hours here, and this is a whole lot of fun career advice. I'll just throw out one thing and then that'll be, that'll be good, I think. But now I think it's really fun to think about partnerships where I work. Like I obviously did a master's degree and then went straight into industry. I thought about a PhD and I remember I asked to chair the department at the time and said, Should I do a PhD or a master's? And he said, What do you, what do you look into? Do I want to go into industry and he said, well, you know, maybe think about a masters and that's okay. But we have a lot of PhDs that we're hiring in and, and I think there's just good diversity of thought and ideas. I think about analytics, and I think about analytical diversity too. We have an analytics community if Fed-Ex ground now and you have a whole spectrum of people, some people who are doing just stepping into dashboards and things of that sort. If people who are PhDs, who are, who have more theoretical knowledge. Another interesting one is you have like OR, and data science about them as like disciplines. But at the same time there's so much crossover, an intersection and I think building a team of people who can think together and work together and who have their own strengths and skills. I liken it to sports. Take any sport like football, I need the big strong linemen. I need that like the fast skills players. Baseball, I need power hitters, I need people who can hit for average. There's all sorts of aspects and dimensions of a good team. And I think there's, there's a diversity there of skill set and thought. The problem that we've talked about for a long time. Yeah, I contributed some really cool ideas to a lot of other people. And it wouldn't nearly be as good if one of us did it as it is with a team of people working on it. And then you can extend that team concept even outside of like our group to say, okay, we have our core group. But then we did some consulting actually with Dr. Van Hove here at one point, we did. So you have the merger of like academia and industry and working together on thinking through problems. Being part of the informs community is a really cool idea because you're getting to know different people. And just the, the panel discussion we had before this, we had three people on there. Like one was from an oil and gas background, and one was banking, one was consulting, and they just different industries. But yet there's commonalities, analytically speaking, and different backgrounds and education, different backgrounds and experience. So I just think there's lots of opportunities and there are different layers of that. To say like, not just how do I set it at a desk and bury my head and just do things on my own. Like there isn't independent thought and learning. But there's also a partnership that says, how do we do things as teams and all the different layers of Teams? And then even outside of analytics, like how do I interact with upper management? And can I present my material at a level that is corresponding to the hearer? Can I speak in a way that's meaningful? And can I change that if I'm speaking to somebody who's in the weeds, I can talk with my technical jargon and if I'm talking to an executive, I can speak on a level that makes sense to them, that matches, that aligns to the business. And I just think there are lots of dimensions about whether a person goes into academics and does research, and they're contributing ideas that people like me are going to maybe try to implement. Or even like working together and a consulting context or being an industry and doing different things. Teaching, like they're just so many different elements, but they all fit together in various ways. And I just find it all exciting. Let's say one last thing is, I think there's a sense of continuous learning like where I've been for 21 years. I've learned a lot in the workplace. And whether you're professionally doing research and you're always learning, or whether you're an industry, It's not as if you're ever going to be anywhere and just take your current skill set and use it and never go beyond IT. Technology changes. Technology is changing at such a fast rate. We moved things from on-premise to clouds and different servers, different technologies. When I came into industry, Python didn't exist or we didn't know about it. Now Python is like pretty significantly used, so it's just always a lot of fun, I think. Doing something that's meaningful. But also you're growing and you're learning and you're changing and you're doing it together with other people at the same time. Wow, that, that's phenomenal advice and life insights. Thank you for sharing and thanks for coming on today. It's absolutely. Have a good rest of your day. You too.