Future-Proofing Procurement with Emerging AI Technologies

Unlock the secrets of integrating cutting-edge AI into state procurement processes with insights from Amy Glasscock, Program Director of Innovation and Emerging Issues at NASCIO and Megan Smyth, NASPO's Director of Legal Education.

Curious about how AI like natural language processing and facial recognition is already shaping our daily interactions with technology? Amy breaks it down, explaining the importance of differentiating between traditional and generative AI (Gen AI), and why state governments need tailored policies and safeguards. Hear firsthand how these advanced technologies are transforming government functions and why it's crucial to stay ahead of the curve.

Click here to view the transcript of this episode: FutureProofingProcurementwithEmergingAITechnologies.pdf (naspo.org)

Follow & subscribe to stay up-to-date on NASPO!
naspo.org | Pulse Blog | LinkedIn | Youtube | Facebook

Transcript

Telice Gillom: 0:05

Welcome to another episode of the NASPO Pulse podcast, your source for all things state procurement. I'm your host, talese Gillum, and for today's episode I've got a fabulous co-host joining me, the illustrious Megan Smyth, Smyth NASPO's Director of Legal Education. Megan, thank you so much for joining me today.

Megan Smyth: 0:23

Well, thank you for inviting me, Telice. I'm so excited for today's episode. Who are we talking to today? On today's episode, we're talking to Amy Glasscock, Program Director of Innovation and Emerging Issues at NASCIO, the National Association of State Chief Information Officers. Amy specializes in emerging tech, privacy and IT policy, so we're talking about AI today.

Telice Gillom: 0:47

Ah yes, An incredibly important current issue.

Megan Smyth: 0:50

Amy talks with us about the types of AI available for use, what issues attorneys in the state should look for when contracting for any services involving AI technology, and what procurement officials should know as we move forward into procuring this new tool.

Telice Gillom: 1:06

She also talks with us about practical usage for AI in the public procurement process and gives us some examples of great and not so great usage for AI in front-facing interaction.

Megan Smyth: 1:16

Yeah, this topic can inspire some fear and doubt in people, but we are here to demystify the fear-mongering and talk about the real issues.

Telice Gillom: 1:24

So get comfortable, because we've got a lot of great information to share.

Megan Smyth: 1:28

Telice, I for one welcome our robot overlords. So let's dive into this topic with that kind of enthusiasm.

Telice Gillom: 1:34

Remember to subscribe to the NASPO Pulse podcast so you never miss an episode, and feel free to email us with your questions.

Megan Smyth: 1:41

All right, let's take the pulse us with your questions.

Telice Gillom: 1:48

All right, let's take the pulse. Thank you so much, amy, for being with us today to share your knowledge on this topic that we know is at the front of mind for many state procurement professionals and the attorneys who support them. Please if you would introduce yourself and give our listeners some information about NASCIO as an organization.

Amy Glasscock: 2:05

Absolutely. Thanks for having me today. I'm Amy Glasscock, I'm the Program Director for Innovation and Emerging Issues for NASCIO and I also co-host and produce our podcast called NASCIO Voices, and as part of my role in keeping up with emerging issues, I've been dealing a lot with the increased interest in AI, as you can imagine. And then just to explain a little bit about our organization, NASCIO, the National Association of State Chief Information Officers is the membership organization that represents the chief information officers of the states and territories and, as you guys know, we've worked really closely with NASPO over the years. As you can imagine, there's a lot of IT procurement for state CIOs.

Megan Smyth: 2:45

Yes, NASPO, over the years, as you can imagine, there's a lot of IT procurement for state CIOs. Yes, I was going to say thank you for being here, amy, and that we have what I think is a special relationship with N ASCIO, especially because we all used to be in the same office together back in the day. That's right, and we've done several collaborative projects together and I think it's always super beneficial for both of our memberships when we're able to do that, because there is such a necessity for the CPO and the CIO to speak to each other and have regular conversations, regardless of who owns IT procurement in your state. Do you agree?

Amy Glasscock: 3:22

Yeah, absolutely. There's a lot of important work that overlaps between our two organizations and a lot of challenges that we can work on together.

Megan Smyth: 3:33

That's in common All the same problems. Yeah, yes, all right. So, like you said, it is at top of mind and I don't know if we're going to get burned down on this topic or not, but I know that everybody's hungry for it. We had a webinar yesterday on AI with the most registrants I think I've ever seen for a N ASPO webinar. So this is what everybody wants to know what to do, wants to know what to do and I wanted to discuss a little bit. Before we start saying AI and AI, this and AI that, what are we exactly talking about? And I think that people struggle with that as well. There's a lot of things in the media and the common vernacular that aren't necessarily accurate from a technical point of view. So could you talk a little bit about the types of AI that there are and what we're usually talking about when we talk about AI in state government?

Amy Glasscock: 4:34

Sure. So artificial intelligence is, you know, sort of your plain old, basic, old school. Ai is the simulation of human intelligence processes by machines, and it includes things like natural language processing, speech recognition, robotic process automation, facial recognition, and we've been using AI every day for many years. So if you have a smart speaker like Alexa in your home, if you use Siri on the phone or you use your face to open your phone, if you use Siri on the phone or you use your face to open your phone, if you use Google Maps and it tells you to take a different route because there's an accident ahead, or you get targeted ads on Facebook or Instagram, or these ads follow you around you know your internet world then AI is already part of your life without you having to seek it out.

Amy Glasscock: 5:20

Now, what we've really been focused on over the last year and a half or so is generative AI, or Gen AI for short, which is a type of AI that generates things like text, images, audio, video code, language models or LLMs like OpenAI's ChatGPT or Google Gemini, and they can really help with creating efficiencies in both your work and private life if used correctly and with some guardrails in place. So that's really the kind of AI that people have been talking about. I mean, we're still talking about regular AI and a lot of states are putting out AI policies which are broader than just GenAI. But I do think in conversation people need to be specific. Are you talking about Gen AI or are you talking about AI, which is a much larger bucket?

Megan Smyth: 6:13

Yeah, agree, and I think there's so many products coming out and every week it seems like there's something new. There's a new photo generator new, there's a new photo generator, and I myself found it interesting to figure out which limitations the the different systems had. And there is even specialty right you can specialize your AI toward a certain topic or or profession yeah, yeah, a lot of tools out there.

Amy Glasscock: 6:44

You know. So, like if you want help writing a document, grammarly now has AI tools Maybe it was just a website, before you looked things up which I find really useful Help you. There's things to help you write emails more clearly and get to the point quicker, or if you want to create photos for your PowerPoint presentation, and and I know you know, some of the big companies are rolling out tools for their clients and things that I don't even know about, but there's definitely an AI for everyone.

Megan Smyth: 7:12

Yeah, you remember Clippy, the little thing that used to come up, and say are you writing a resume Like that was? Kind of the first, like the computers realizing what you're doing and interacting with you. I mean, that was obviously very basic, but it is that same feeling that somebody's there watching you, and I think there's a lot of paranoia behind this that could be dissipated with information.

Amy Glasscock: 7:42

Yeah, and we've been talking to CIOs a lot about you know kind of what people are thinking around Gen, ai and at least in the state government technology workforce and you know for technology folks there's a lot of excitement in general. There's not really a lot of fear around it. You know, if you get down into other agencies and other lines of business there might be some people that aren't really interested in learning about it or think it's kind of weird or creepy or going to take their jobs and things like that. But for the most part there's just a lot of excitement, at least in state government, around it at this point.

Megan Smyth: 8:17

Yeah absolutely, and you guys, nacio, has been at the forefront of this issue for many years, not just now and just now everybody's noticing it, but it's been there all along. And you guys, actually, in December of this last year, released 12 key considerations as states develop their AI roadmaps, and that's exactly what we've been seeing them do this legislative session is create these advisory groups and task forces. They're trying to figure out what to do, and I think that you make a lot of great points in that article. It's a great resource. I highly recommend it to the attorneys especially, and at one point you say and I love this don't assume AI will solve every problem or help you reach every goal. In addition, don't go looking for a problem to solve with AI. Can you talk about that a little bit and give us some examples of how you could mess up by putting the cart before the horse when it comes to AI policy?

Amy Glasscock: 9:21

Yeah, absolutely, and this is something that I talk about a lot when it comes to using AI or even other emerging technologies that might be bright and shiny at first. So one example of this is just the regular traditional chatbots on websites that you might be using with your airline or things like that. So before the pandemic, only a handful of state governments were actually using those kinds of chatbots on their websites. A handful of state governments were actually using those kinds of chatbots on their websites, and that was really for two reasons based on our research and we've been researching AI in state government since about 2018. So the first was that they didn't really feel like the technology was ready to the point that they felt using it as a state government. You know governments don't necessarily want to be the first out there to use something. It costs money. Citizens don't need to be frustrated. So you know they want to make the first out there to use something. It costs money. Citizens don't need to be frustrated. So you know they want to make sure it's ready for primetime. And the second was that they really didn't feel like they had a business case for AI in general. You know a few examples here and there, a few states using it here and there, but it wasn't really widespread.

Amy Glasscock: 10:24

And then the pandemic hit and at that point, chatbot technology was really kind of ready for primetime and there was a huge business case. So by June of 2020, 75% of states had chatbots on at least one state website and they were using it to answer questions about unemployment benefits because, you remember, the unemployment offices were just completely crushed under inquiries and the call centers couldn't keep up and then just general questions about COVID, maybe to the Department of Health or things like that. So that was a good example of, you know, really using it to solve a real problem and this was the answer to the problem at the time to solve a real problem, and this was the answer to the problem at the time. And it's funny to think about those kind of chatbots now because now that we have generative AI and chatbots based on large language models and things, they're so much more advanced and better and it's only been a couple of years. But, yeah, that's an example we like to talk about.

Megan Smyth: 11:20

Absolutely. And you know the attorneys who I work on educating here at NASPO, who support central procurement. I think that they see themselves as risk managers, as I don't want to say gatekeeper, because that's not accurate. It's more of a guard at the gate right. We're going to let you through, but we're going to pat you down first. Through, but we're going to pat you down first. And what would you say to a state attorney who says I don't really quite know where to start here and how do I protect my state when I enter into a contract with a vendor who I know is using AI, but I don't know how they're using it. I don't know what they're doing with that data. What advice would you give on building these contracts with this new AI in mind?

Amy Glasscock: 12:14

Yeah, so I will just say, like any smart policy analyst since I'm not an attorney and get nervous about giving advice to attorneys I reached out to some of my colleagues who are attorneys and privacy experts and heavily involved in studying policies around AI in their states and really involved in procurement as well, to help answer this question. So you know, knowledge is definitely the best offense and they have to be on offense when it comes to these contracts. So you know, there's established boilerplate procurement T's and C's for products that governments buy, and that's been around for years and years and years. But now we have these unique issues around AI development and use and now we have to revisit the old boilerplate and update it for AI, just like we've had to do with cloud contracts over the last several years.

Amy Glasscock: 13:02

So, yeah, so it's up to state procurement folks to advocate for what's best for the state. They're really the only ones that can do that and it can be really easy to just keep checking some of the same boxes that we've been checking for the last 20 years. But these new technologies require a revision, obviously, of AI specifically. We do want to think about, like you said, if it's a contract for building a model. We want to know whether or how the state's data is going to be used to train that model. We want to consider terms about testing the algorithms. And then finally, sort of your original question, some federal agencies have already adjusted contract language around AI, which can be a helpful resource that states can build on and make more specific to their situations too.

Megan Smyth: 13:47

Absolutely. I love the connection you made to cloud. I was just thinking there's a lot of the same issues data privacy, storage, secure it's a lot of the same issues, so that's a good comparison, I think, for them to start with, because everybody's got a cloud contract at this point right. Absolutely yeah that's a great point. Yeah, thanks.

Telice Gillom: 14:09

Switching gears a little bit. Let's talk about the usage of AI in the actual procurement process. Procurement professionals should be aware, if they're not already, that there is AI in some of the products that their agencies are already using Microsoft Office products, for example. When you're typing in a Microsoft Word document, it gives you suggestions for text that would be coming next in the paragraph that you're typing and in the publication that NASCIO put out in this past December. Does NASCIO offer any guidance to states around that?

Amy Glasscock: 14:51

Yeah, definitely. We, as part of that guidance, recommend that there be an inventory for states to find out what AI is being used by the agencies, either knowingly or unknowingly, because, as you said, ai functionality is being rolled out into systems all the time that we've been using for several years, but then all of a sudden they have these new features and with generative AI, that's happening a lot. Connecticut was actually required by law to do that, and so they took an inventory from their agencies on what AI was being used and had a short questionnaire and had them fill out and have all of that available publicly. So you know it's a good starting point, especially, you know, like I said, some of these Gen AI tools are so accessible.

Amy Glasscock: 15:36

Before, if a state agency was using AI, it was probably a specific product and they'd bought licenses for it and it was only for a few people to use. And now anybody can go out and, with a login or not, and use generative AI for free in many cases, and you know, users don't need a license and they're able just to do it on a browser. So there's a lot of easily. You know we call it shadow IT shadow AI happening in states, and so it's just really good, I think, if you're developing some policies to wrap your arms around what's actually being used out there.

Telice Gillom: 16:17

Would you say that they should disclose that information? For example, if I am a procurement officer who is writing up a solicitation and I know that this document, you know, has Gen AI in it, I'm typing and it is giving me suggestions as I'm typing, should I disclose that to potential suppliers? This document was edited for grammar with the usage of AI.

Amy Glasscock: 16:44

Yeah, I do find that those disclosure requirements are a common requirement of state AI policies, and I think it has a lot to do with just kind of ensuring citizen trust in government and not wanting to erode it by having something inaccurate come out and passing it off as your own or you know, things like that. You know, of course, it's going to depend on each state's AI policies and what the requirement is. I do think, as time goes on, though, it will be hard to use any tools that are in some way touched by AI, and so I could see that requirement easing up a bit in the future, but it's just kind of my personal prediction. We'll see.

Telice Gillom: 17:24

Speaking of those, policies, but it's just kind of my personal prediction. We'll see. Speaking of those policies, I think you're right as it becomes more prevalent. I don't know that it will be necessary to disclose that information, because it will be assumed that it's there and not necessarily assumed that it's not there. And what questions do you think states should be asking themselves as they develop these policies now, presuming that they may be obsolete in the future?

Amy Glasscock: 17:52

Yeah, you know, I think the kind of questions that you know your folks should be asking themselves are kind of the same that anybody should be asking themselves when they're using AI. You know specifically around it being obsolete in the future. You know, just kind of thinking back, like I may have like text predictor on my phone I don't disclose that. You may have suggestions in your Word document. You're not disclosing that. So you know, I think maybe if it comes to like a summary of a Zoom meeting, those get things wrong all the time. So it's really kind of helps you to put it on there and disclose it. And if it's like something that you're not an expert in, I like to talk about this a lot. I got this feedback from our CIOs If you're using it to write something that you're not already an expert in, it's hard to see when it gets something wrong. So you know that's sort of a good way to think about it at this point. But then, beyond that, you know asking yourself is this use of AI? Excuse me, is this use of AI low risk?

Amy Glasscock: 18:58

You know, are you using it to write a document or to make someone's healthcare decisions, which is not a procurement function, I understand, but it's an example. Is it going to save me time or is it going to make my final product better? Because if the answer is no not really then not really much point using it. Will it make the process, especially with procurement, more fair? Will it make the decisions more fair? Will it reduce bias or will it create bias that I wasn't expecting? Is there a chance that it will get some things wrong? So you know, those are all kind of questions that we like to think about when deciding to use an AI tool.

Telice Gillom: 19:37

Megan, I would ask you some of the same kind of questions from a legal standpoint because, speaking of fairness in the evaluation and award to identify where the decision points are made- and ask who is making that decision.

Megan Smyth: 20:06

So if I am an attorney on the supplier side and I'm told that I did not win an award, I'm going to want to know why. I'm going to want to know why and when I start diving into that if your system isn't set up to tell me where the decision points were and whether it was made by review these proposals and tell me which one is the best. That's a very different question than add up this score sheet and then make a spreadsheet and tell me what all the scores are right, and I think that distinction is what will probably make the difference from the legal perspective, and so that's something to think about now in your policy.

Amy Glasscock: 20:57

Good point.

Telice Gillom: 21:00

I would say at this point for that particular process. I don't believe and I could be wrong that AI is ready to make those kinds of decisions. Because of that, if you are using it to check grammar in a solicitation document or give suggestions for headers, to give suggestions or citations for documents and even for citations not ready because it has been generative. AI has been known to hallucinate things like citations. There are good usages and bad usages and at this point, using generative AI to make decisions that human beings will later ask about from a procurement standpoint, I don't believe it's ready because there inevitably will be someone who will ask and having to defend the decision that a quote-unquote robot made can land any agency in a world of trouble that I don't think they would like.

Megan Smyth: 22:22

Yeah, yeah. The only other thing I would add is the bias problem. If you're pulling, if your generative AI model pulls from all the information on the internet, well, there's some really horrible things on the internet and you don't want your ai to take on the biases that we all have. Because we have eyeballs and we live in society. You'd prefer that it be a pure environment, but if you don't give it the right input data, you're not going to get good output. Um, and so I think, from from a legal perspective, if I'm thinking about the issues that someone would bring up, I would also think about that one. How are you going to ensure that the decisions your AI is making are not discriminatory in any way?

Amy Glasscock: 23:11

Yeah, I know there's been a lot of talk around the use of AI in HR too, for those reasons or college admissions, and so, yeah, big questions.

Megan Smyth: 23:24

And we're going to solve all of them on this podcast right o On this podcast. If you listen to the end, we'll tell you all the answers.

Telice Gillom: 23:33

Keep listening, we'll figure it out, we'll figure it out. So we're talking about some examples of, you know, not recommended usages for the procurement process a good usage of generative AI for customer interactions and a not so good usage of generative AI for that same thing.

Amy Glasscock: 24:07

Sure. One of our examples that we've heard about in our community is the state of South Dakota has a generative AI chatbot and the model that it's pulling from is the state government websites. So if someone asks a question, it's pulling it from the government websites, giving the answer and then, you know, showing the location where they can go and read about that on the website. So that has worked pretty well, according to them, and has been a lot better than the traditional kind of chatbot that they may have had access to before.

Amy Glasscock: 24:44

Then there's another story of a major airline that their chatbot was agreeing to things that the customer was asking for that were actually outside of their policy, like a flight change or a refund or something like that. The chatbot agreed to it and so it led to a lawsuit because an airline said, no, that's actually outside of our policy, and you know the airline ended up having to agree to it in this case. But you know again, that was not like a huge risk to the airline. It maybe cost them a few hundred dollars Well, plus the legal fees, I guess, but you know it's not like it was making decisions on fixing the engine while in flight in the full plane it's embarrassing, but that's about it.

Megan Smyth: 25:24

Yeah, it's embarrassing.

Amy Glasscock: 25:26

Yeah, but not life or death.

Telice Gillom: 25:31

So here on the podcast we're just starting a fun little segment. At the end, Bad jokes and puns. Megan, have you got some puns for us? Toss them my way.

Megan Smyth: 25:42

Well, Telice, I love a good dad joke. And do you know how to tell when a joke is a dad joke? It becomes apparent.

Telice Gillom: 25:53

That's a good one I like that.

Amy Glasscock: 26:02

Okay, I think I have one. That's a good one, I like that. Okay, I think amy's got one, okay. So, uh, yeah, the other day, um, this fancy box was delivered to my house. What was in it? Well, we're looking into it. In it, well, we're looking into it I was going to say I have an eight-year-old.

Megan Smyth: 26:26

Most of the best jokes are like inappropriate and have to do with the bathroom, so I wasn't going to say any of this. That's an eight-year-old joke. That's a classic. I like it.

Telice Gillom: 26:34

We like those, we love it. Here's one I heard the other day and it's a Christmas joke, but I'm just going to toss it back out there because hey, why not? What is Santa Claus' favorite genre of music? Rap.

Megan Smyth: 26:51

That is right there.

Telice Gillom: 26:53

Yep, right there, right there for the taking. That's a good segment, amy, thank you so much for joining us. That's a yeah, it's a good. We'll bring that one back out in December. Thanks so much for joining us today.

Amy Glasscock: 27:11

Thanks for having me. It's fun.

Megan Smyth: 27:12

Yes, amy thank you, and I'm sure I don't know if it's a blessing or a curse that you are the AI expert that you are, but we appreciate you sharing your knowledge with us n Never a dull moment.

Telice Gillom: 27:25

That's for sure. Megan, my co-hostess with the mostest, we'll have you back on. Thanks for joining me today. It was a pleasure. I'll work on my standup. We can do a whole routine. If you want, we can go back and forth like Abbott and Costello. You let me know, let's do it All right.