Innovating Math Education through AI with Dr. Jeremy Roschelle of Digital Promise

Dr. Jeremy Roschelle, Executive Director of Learning Science Research at Digital Promise, conducts research on the future of learning with technology. Dr. Roschelle is Fellow of the International Society of the Learning Sciences, with >100 publications, 25,000 citations, and 10 patents. Dr. Roschelle serves as a subject matter expert on Artificial Intelligence to the U.S. Department of Education and on the Advisory Committee for Education at the National Science Foundation. He actively communicates to broad audiences about how findings and insights from research can shape the future of technology use in education.

Recommended Resources:
🔗 Dan Meyer
🔗 EDSAFE AI Alliance
🌐 TeachAI
🔗 Ethan Mollick
🌐 Stanford HAI

This season of Edtech Insiders is once again brought to you by Tuck Advisors, the M&A firm for Education Entrepreneurs.  Founded by serial entrepreneurs with over 25 years of experience founding, investing in, and selling companies, Tuck believes you deserve M&A advisors who work just as hard as you do.

Transcript

Alexander Sarlin: 0:26

Welcome to Season Eight of Edtech Insiders where we speak to educators, founders, investors, thought leaders and the industry experts who are shaping the global education technology industry. Every week we bring you the Week in Edtech. important updates from the Edtech field, including news about core technologies and issues we know will influence the sector like artificial intelligence, extended reality, education, politics, and more. We also conduct in depth interviews with a wide variety of Edtech thought leaders, and bring you insights and conversations from edtech conferences all around the world. Remember to subscribe, follow and tell your edtech friends about the podcast and to check out the Edtech Insiders substack newsletter. Thanks for being part of the Edtech Insiders community. Enjoy the show. Dr. Jeremy Roschelle is the executive Director of Learning Science Research at Digital Promise where he conducts research on the future of Learning with Technology. Dr. Roschelle is a fellow of the International Society of the learning sciences, with more than 100 publications, 25,000 citations and 10 patents. Dr. Roschelle serves as a subject matter expert on artificial intelligence to the US Department of Education and on the Advisory Committee for education at the National Science Foundation. He actively communicates to broad audiences about how findings and insights from research can shape the future of technology use in education. Jeremy Roschelle, Welcome to EdTech insiders.

Dr. Jeremy Roschelle: 2:01

Hey, Alex, it's great to be here.

Alexander Sarlin: 2:03

I'm really excited to talk to you, you know, I have followed your work with Digital Promise with Sri for quite a while you're really a giant in the field of learning sciences. Let's start by talking about your background in the learning sciences. How did you get into the learning sciences in the first place? And how did your journey influence your current research role focusing specifically on AI and mathematics and other current topics like that?

Dr. Jeremy Roschelle: 2:27

Yeah, I'd love to share Alex, you know, I was naturally good at math in high school. And I got into MIT and I thought I was going to study physics. That's how this journey started. And I couldn't stand the way they taught physics, the guy would start with equations on board number one and fill up nine blackboards full of equations without explaining once what it was all about. Wow. And like, that was a real puzzle to me in my early undergraduate years, like, here's a subject I really loved. And I was good at math. And why do I hate learning this subject? Well, I went over to computer science, which they taught exceptionally well at MIT. And then I had this like, amazing experience where I discovered a music professor, her name's Jean Bamberger. And she was using technology to flip a class. And it was a music appreciation class, instead of like listening to music, and having the professor tell you how great it was, you would listen to something and you would try to recreate it using a computer music program. Wow. And so you would do a try to make the music yourself and you would discover things because would make you listen really hard when you're trying to recreate it. And I was like, Whoa, this is an amazing way to learn. Learning by constructing something blew me away. And I loved music, too. And that contrast set me on my path between here's a subject I should love, but I can't stand the way they teach it. And here's a real surprise. And so I decided after my MIT experience, actually, I thought of pursuing AI at that time. So I hung out in the MIT AI Lab, read all their working papers, everything. This was circa 1985 or so. But I decided not to pursue AI. But to go into Cognitive Science and Learning Sciences. Started out did my dissertation about how to teach kids physics using a tech? Because like, I still trying to figure it out. Isn't there a better way to do this? And I'm gonna keep the story moving by saying eventually a great professor came along Jim cap, and he said, you know, you got a good future ahead of you, Rochelle, but you're wasting your time in physics, but only a few kids make it to physics. Math is where the action is. You've got to refocus and help kids learn math, because that's where our big national problem is. And so I accepted that and ever since my research is a learning scientist, has been about what are the effective ways to use technology for mathematics learning. And I pursued that at SR I have pursued that it digital promise. I've pursued it across different kinds of studies, like design studies, and then randomized controlled trials to really try things at scale, and all kinds of things. I've worked with companies, I've had lots of experiences, but a lot of them around. How do we use technology effectively for Teach You're learning math. And

Alexander Sarlin: 5:23

you're starting in the mid 80s, with this story, and here we are in 2024, you know, 3540 years into this journey? Can you give us some of the highlights about what the research has revealed in these decades about the effectiveness of technology in general, and maybe even a little bit about what we are learning now about AI? In enhancing and teaching the learning of mathematics? What are some of the pivotal findings that have come out in the decades you've been really deep into this field?

Dr. Jeremy Roschelle: 5:53

Yeah, that's fascinating. It's something of course, because I kind of care about what are the big ideas, it's that could help us do something at scale. So I've thought a lot about that over time, and guided by experts in the field. So you know, one thing I think we've learned, is really on the back of the move to give kids graphing calculators, so I'm just gonna start there, that's probably the oldest math educational technology that made a difference, and offloading some of the minut calculations to a calculator. So you can focus on the big picture, quite literally, the big picture, our graph really helps kids develop concepts. And so when I think of learning principles, with that told us, it's a little bit about representing the math using visual graphs, but also that cognitive load, helping focus their cognitive effort and not getting students lost in the details. So that's one big area of principles. Second big area of principles, is using visual and interactive representations for conceptual understanding. And you see a lot of this now Desmos is a great case that's out there, you see it in all sorts of commercial products. But I had the pleasure of working on it really, in the early days, you know, when the Mac came out, and it had great visual capabilities. And we could draw vectors on the screen, we could draw animations, we could have graphs that controlled animations, it was an exciting time. And now a lot of that stuff is commonplace. And then the third area is where we see most of the advances in AI. And that is getting good at mathematics does take practice. And you can assign practice in smarter ways. And also, you can give feedback in smarter ways. And so actually, like I like to remind people, there's a tendency, think like, when was aI invented yesterday, Alex, was it yesterday or the day before? Because, you know, Gen AI arrived on the scene. And it seems like that's when it started. But the history of using AI for math, teaching and learning goes back 2530 years. And a lot of that research was how do we help kids do smarter ways of practicing their math? And that's in products like the cognitive tutor from Carnegie learning? Yep. So actually, all three of these kinda have products with them, the graphing calculators, Desmos, practice with feedback. And the visual representations is all great examples. They all have sound learning principles behind them, there's RCTs, for each showing, they really work.

Alexander Sarlin: 8:30

One of the things that we I think, in the EdTech world always wrestle with is how much, you know, deep randomized, controlled trial, you know, how much real research do we need before we take a product and try to scale it out to the world? You know, you're mentioning the Carnegie learning, which is one of the best researched edtech products out there. Desmos comes from a whole line of graphing calculator research, but you know, for everyone you just named there's 100 different math edtech tools that have less research backing, I'm curious how you see the field in relationship to research, and how you know, a newer edtech product might think about whether they want to build on past research or do their own current research? How should they think about that?

Dr. Jeremy Roschelle: 9:13

Yeah, absolutely. And I resonate with your remark, you know, researchers are as aware as anybody that research is really slow, and expensive. And the idea that you're going to come out of the gate with a new product, and have a gold standard kind of evidence, it's just not realistic. So I would say a couple things. One of the things that call for these days is make sure you're using some modern learning principles, I can get pretty frustrated when I hear even some bigger figures in the market say, Well, you know, this is what learning was like an 1800. And we've got to do better. And how's here's how we're gonna do it. But yet, we've learned some things since 1800, Quickstart, a little bit closer to today, you know, in some learning principles, and so, you know, I mentioned a few with conceptual understanding, helping kids build a robust mental model of those key elements and figures and the graph or whatever it is, so we have better principles. So in the tiers of evidence that the government describes, these would be called rationale. How good is your rationale and rationale is not that expensive to produce. And you know, some of my colleagues at Digital promise with the learning variability navigator, they do a great job on helping companies with rationale for special ed. So there's different all different kinds of rationale, that's where I'm going with that. But I would ask companies to kick the tires on their rationale, and make sure they're using the available knowledge the best they can. But beyond that, certainly, it's great as you get started to do some pilot studies, almost never works to go all the way to a gold standard study as your first thing you tried to do. Start working with some researchers early but don't set the bar super high. Think of the research as formative improving your product, a lot of times helping you understand what teachers are going to need to use the product well, with students to do that work. And then once you've got pretty comfortable that you've got a really solid working system, and you're gonna have to implement it really consistently, with hundreds of classrooms to get a good thing, then you're ready. But even in the development of a product, a lot of times in the early days, companies aren't ready to implement it consistently in one way, like, we want students to use this an hour a week. And here's the routine, it's going to be to a good controlled trial, you really need to have establish that protocol, that approach. So it takes a while. But I think research can follow companies along basically, it can work at all the stages that they're in, and actually developing a great product. At the end of the day. It seems like technology moves fast, and research moves slow. But at the end of the day, product development actually takes time, too.

Alexander Sarlin: 12:08

And it's a really good point. You mentioned a couple of tools that are really solidly researched and evidence based in the field Desmos my first apartment in San Francisco, I live right next door to Eli Luber off from Desmos. And it was fantastic. Oh, yeah, brilliant guy. And obviously, Carnegie and you've seen some really amazing tools. When you look at the landscape of tools that are teaching, mathematics education, and the issues that you just name, you know, cognitive load visual models, you know, mental models? How do you see the landscape? Do you feel like we've developed a pretty robust landscape of solid tools? Or do you feel like there's still a few that sort of really meet the bar of research principles and the rest sort of fall behind and are just sort of trying to figure it out?

Dr. Jeremy Roschelle: 12:53

I think we're in a much healthier place than we were like a decade ago, I think the kind of things that were just sugarcoating mathematics to make it fun, you know, write an equation to blast something out of the sky, but there's no real learning value there, it's trying to motivate you to write an equation, I don't see so much of that anymore, I see a lot more that's trying to make the motivation, you know, more intrinsic to the work. And, you know, I see a lot of really high quality companies out there putting out products that have all these different dimensions in them, they'll have the calculator part, they'll have the practice part, they'll be doing things to support kids with special needs. There'll be doing stuff to develop conceptual understanding. So that's rewarding to see that. I think one big area of growth that we'll get into as we talk about AI, some more you and I is the researcher side of the world would say there's been a social turn, and how we think of mathematics, really a social cultural turn, and how we think of mathematics beyond cognitive. And I think AI gives us opportunities to move towards the more conversational. And in my research, going way back to the beginning, conversation is just essential to making the math meaningful. That's where meaning making happens. It's when people talk about the math. A very few people might do meaning making all by themselves in a quiet room. But most people, it seems to me do it when they're talking about and explaining, justify justifying, arguing critiquing these kinds of things that we call 21st century skills are intrinsic to understanding math. And that's where I see big opportunity for growth. So I think they have a pretty solid base. But we're missing this one really important element that we could do some more work on. And this is part of where teachers are going to is following that social turn in mathematics. So it's the technology and the teachers together.

Alexander Sarlin: 15:06

It's a really interesting and I think, perhaps counterintuitive point for some of us who think about mathematics education. It may be a little bit of an antiquated way. You know, you mentioned calculators and practice sets and that all sounds very familiar, but the idea of you know, I Guess they call it mathematical discourse or talking and articulating and you say, debating you know about math, it doesn't always feel like I don't remember doing a huge amount of it in my mathematics education know that

Dr. Jeremy Roschelle: 15:34

I pray well, and you know, one thing we learned in research that's really cool is that the math classes you and I took, they would teach you the abstract thing first, and then give you an application of that. And one thing we've learned from the learning sciences is that's almost exactly the way to start is activating something that is familiar to the student that is close to the mathematics. In essence, this is why we value story problems instead of just the bare bones math, because the story is meant to evoke an everyday understanding that will help you see the purpose of the math. So if you take that to the next step, you know, you really have to be creative for different students in different settings. What kind of understandings are they bringing to the table? And what kind of setting what kind of story what kind of situation would activate that knowledge? When I was doing the conceptual understanding work, for example, we were going to do something we knew, with something moving across the screen, according to a graph. And we put a lot of thought into what that scenario of the thing moving across the screen would be. The very first draft, I'll tell you was on the moon with aliens. And we thought that was just really fun and game like, but you know what, it didn't get students really actively thinking except for, you know, a few gamers who liked it. And so we switched to soccer, a soccer player running across the soccer field, it was much better. Kids love thinking about races and running and catching up. And they had all this knowledge of the phenomena of motion on a soccer field that they could bring to bear to start to make sense of the mathematics, you know, faster, slower, catching up, getting farther apart, accelerating, these words came naturally to them in a soccer context. And so that's, I think, for people who say, well, math cultural, you know, it's not it's just the math, from a learning point, it's not the way we learn these abstract concepts, is to start with something familiar. Yeah.

Alexander Sarlin: 17:45

So I think that brings us in a few different ways to a natural segue to talking a little bit about the future of learning math with AI, you've mentioned it in passing a little bit here. But one of the ways that AI we know can influence learning, I won't say necessarily improve yet, is changing the scenario of a story problem, for example, or basically adapting any kind of learning resource to the interests of particular learner. So you know, the gamer learner might get an alien, the sports learner might get a soccer ball, a different kind of learner might get a race car. I'm curious if you see that, as one of I'm sure many different ways in which AI will influence the future of teaching and learning.

Dr. Jeremy Roschelle: 18:26

Absolutely, yeah, I'm gonna walk you quickly through like sort of three levels of change and math that I think is coming and the one you describes me the middle one, but I'm going to take it one beyond there that I spent some time thinking about to the most basic 1x is just the math we've already got. But giving kids a better as their learning it experience, more an assistance, a tutor, a guide on the side, but we're not changing the math at all. We're just giving them a better support resource than conventional ed tech, which kind of pops up a balloon with a hint at it. And we're making that more natural. That's the most basic level. And it's not a bad level. I'm not saying that negative way. And actually none these levels. I'm not saying negative way. I think the second one you're right on. I'm very excited with AI, about how good it is for storytelling. And I think story is a neglected resource in all of modern education. We started as human beings learning by telling stories around the campfire, it's it's in us that stories grab us. And in the math classroom, like I said, kids like telling stories about motion. They like scenarios where they're, you know, managing the lemonade stand or whatever it is that helps them connect, what's the purpose of this math? Why am I learning this? So I think the ability to customize the stories, super key, so changing the story, it's not just to make it be what students want. It's really in math, to activate prior knowledge activate ways of understanding that will help them move from familiar to abstract in their development of a math concept. And the third level is very far reaching, which is the need changing what math itself is in school, and why I think that's important is really AI can do all of school math, not necessarily your chatbot AI. But other forms of AI that have been around for quite a while can do all of algebra and calculus. And it's, it's all machine doable these days. And that should make us question, what is the human role in mathematics, until we want to spend so much time getting students to do manipulations that really computers are going to do for them for the rest of their life? And then the simple analogy, you know, one analogy is to, like, I learned the square root procedure, which is kind of like long division. And like, nobody teaches that anymore. You're going to use a calculator or a computer, if you're going to do a square root? Well, there's a lot of things like that, where the calculation part of it you're going to use a computer for. And that should make us question, what do we want the human understanding of math to be. And I think that's going to move more into parts of like using mathematics to model a situation and make predictions, it's going to move more towards the use and application of concepts and less towards you have to calculate quickly and accurately. And I think lots of people over time and pushing for that, I just think that has to accelerate. Because in a race to calculate quickly and accurately, Alex, you and I know who's going to win that race, it's going to be the ai ai can't do is look at a model that's meant to be a good estimate of how much it's going to cost to build a campsite. Or build a house and say how good a fit is that model? Did we leave something out, just gonna be human choices. And so it's really that kind of back and forth between math in the real world, where I think I would like people to still be learning a lot. So I think our goals and teaching and learning mathematics, what we consider to be worth 12 years of K 12 Education is going to change in the next 10 years. So

Alexander Sarlin: 22:38

that is it's a really thorough and very well articulated vision of some of the major changes in math, I'm just going to read back what I'm hearing, because I think there's a lot of really interesting pieces of this, I'd love to dig into. You mentioned these sort of three. So you'd already mentioned mathematical discourse and the social aspect of math, the ability to you know, talk about math debate, collaborate around it 21st century skills as they relate to math, very interesting. And then there are these three tiers. One is about taking existing math and using artificial intelligence or other sorts of tutoring type systems to help people break down problems, move forward, understand the math better in real time, we just saw, you know, Google put out learn LM last week, which specifically was trained to be able to break down math problems, and teach them step by step. And I think, you know, we're definitely heading there than the second one is about inserting storytelling and application scenarios in maybe a customizable sort of adaptive way into the math. And that's a huge part. And then the third, which feels very related, is, you know, when you have these storytelling scenarios, where you have these application scenarios, the story problems, how might we move from the calculation model of math to more of the application? Like how can we use this math to actually do something meaningful in the world that goes beyond the rote algorithms of you know, long division or the quadratic formula and actually start using the math to do things or to predict things in the world?

Dr. Jeremy Roschelle: 24:04

Yeah, or support an argument you want to make about some change in the world? Maybe? No, a lot of students, they're very worried about climate change. Are there ways in which they can be making arguments in their own community about what the cost benefit would be of doing something that would help with climate change, we want students to feel that power associated with math as a tool to help you work with other people to make change in the world. Students get excited about that, especially as they're moving into high school. They're thinking about who they're going to be in the world. And if they can see math, not it's as this arcane set of things they had to do to get a credential, but rather is a power tool for who they want to be in the future. Well, that's exciting to me. Definitely,

Alexander Sarlin: 24:53

you know, some people might call that sort of interdisciplinary, but it feels like it's just very intuitive. The idea of you mentioned going from the familiar to the concrete. And issues that students care about, like climate change are incredible motivators to get people to care about tools like math and data analysis to be able to make a difference. I love that vision. So So let's back out these are fantastic ideas. How do you think the actual classroom is going to evolve to keep pace with some of these changes? Is that you anticipate that we'll all see in the next, you know, few years, two decades around how math is taught? How are teachers going to be trained? How are schools going to adapt curriculum?

Dr. Jeremy Roschelle: 25:33

That's really the million dollar question. A few minutes back, you and I were talking and you're asking me to assess, you know, where are we and I said, pretty good. However, you know, all these technology resources that can help students learn math, they're not always well used. And so we're behind already in teachers being able to use technology well for math, teaching and learning. And so we've got a long ways to go. Teachers are essential to this process, but we don't invest enough in our teachers, and the romance of the technology can take away dollars from the teacher side. And that always worries me like, if a district or a school board ever asked me about their use spend, they asked me about their spend for technology and math. You know, I'm always going to ask them, how much are you putting alongside this tech spend into teacher support teacher development, time for teachers to get together and figure out the best ways to use this? So I do think there's a lot of change. I think already. I'm glad the teachers are excited about AI. And they like the storytelling part, I think there's still a lot of work to do in helping teachers develop math classrooms that are oriented towards the goals, we're talking about, to social use of mathematics to conceptual mathematics to, you know, mathematical argumentation, that's still very much. Look, we know from research. I don't want to make it sound like it's undoable. It's doable. We know from research, it's doable, but it's not yet what's happening broadly. So I think the most important thing, and how classrooms are going to change is changing our image of what we want to be going on in those classrooms. Do we want this emphasis on accurate fast getting through the problems? Or do we really want classrooms that spend more time on fewer problems, but really get to conceptual understanding, I want that. That's a change. That's a change. And I think it's changed lots of people have been talking about for a long time. But the advent of AI, and the societal changes are going to mean we have to push harder now for students to be learning the conceptual and social parts of math.

Alexander Sarlin: 27:55

And some of your comments make me think of you know, Dan Meyer's TED talk about using, you know, really complex story problems and sort of having a whole class revolve around, you know, taking a story problem and deciphering it and talking about it and arguing about it, and really changing the whole paradigm from you know, we're doing this set of 20 problems or, you know, to we're taking a problem that seems meaningful and weighty and piecing it all together. There are so many different great ideas coming out of this conversation, I think, coming out of the research, do you see some of this manifesting in the current uses of AI, some of the AI startups that have come in the last couple of years with generative AI or some of the, you know, edtech companies that have been using AI, you know, regenerative ai ai, advanced over the last, you know, decade and a half? What are some of the most promising uses, you see that are moving towards the vision that you're laying out here?

Dr. Jeremy Roschelle: 29:14

Yeah, you know, well, I think you have to look at Kimiko. And what the Khan Academy is doing with that, because they're putting so much well intended effort into trying to do the right thing there. And it is a more conversational approach to math. And, you know, as we've seen, they're getting the earliest releases of the newest models and trying to figure out what to do with those. And I think they're very aligned goals. And yet, it's out in exploratory phase. One caution, I want to say with any of these things, that sometimes people are equating AI with large language models. And as you hinted in your question, there's more to it. If it's not all large language models, large language models are sort of about predicting which words go together. That's not a great way to think about math. And so as we think about AI and math, we do have to make sure we don't get in a trap of thinking. It's about large language models only. I think it's impart about the art language models because they enable this conversational capability. But if you look at the work of Khan Academy there with Kimiko, they're doing work to check the mathematics because they know the large language model doesn't get the math right. And so yeah, so komikko is great to look at not only because of what the product looks like, but because they're openly talking about their engineering process. And so they're a model to others of what Getting it right is kind of look like. Yeah. So I like that one. Another thing I like that's a little really removed from this, but I really like it is a product called Teach FX, which helps teachers look at the conversations in their classroom and improve them. So it's something that listens to the conversation in a classroom, and confines after the class is over, can show the teacher some highlights of that conversation and help them reflect on when they made really strong teaching moves, that did the things we're talking about kept a conversation about a concept going. And maybe when it didn't go so well, and the kids shut down. And, you know, I was talking a second ago. There's a lot of teacher change here, and teach effects and other tools that help teachers learn from their experience in their classroom. I think there's a lot of promise there. And so I'm excited about those things. I'm excited about simulations for teachers, where they can practice new ways of teaching with simulated students, I think that's a great way to give teachers a density of experience, doing things that if they're doing a real classroom, it might be uncomfortable, because it's risky, to try something new in front of your students. Or it might just not, the opportunity might not come up that often. And so often simulations for teachers to learn is a really cool use of AI as well. So let's not leave our teachers out is where I'm going with us. Right, because I've made this play before. The teachers developing teachers, supporting teachers should be a big part of the equation, what teachers deserve good tools for their learning. They deserve them to

Alexander Sarlin: 32:33

100%. And I think teacher effects is a terrific example of a use of AI that feels a little bit outside of our normal consideration of what AI does, especially in the context of teaching. Because it really is about taking the talk data, as you say what is being said in the classroom and who's saying it and turning it into data that can be synthesized and then analyzed and put back into the hands of teachers. One other tool, I'd love to hear your thoughts on that one of the tool that's caught my eye that also has a sort of unusual use of AI, in this context is a nascent, pretty growing tool called OCO labs. Have you ever run across that one, it's another another a a couple of X amplify, folks. And basically, it's a teaching assistant in the classroom that facilitates small group work around problems, especially math problems. And really, it's about facilitating discourse. It has students all doing the same problem and then debating and arguing and discussing how they got the answer, or whether they got different answers until they sort of come to a consensus around what the answer is. And so it's really even though it's AI, it's AI in the service of actually making more human connection and more human conversation, it feels like that matches some of the principles you've been naming in this call.

Dr. Jeremy Roschelle: 33:47

Absolutely, Alex, and I just, I'm smiling, you and I must be twin separated at birth, because I actually had three products written down in my notes for this conversation. And that was the third one makes sense. And I actually did research in about 2005, on that model of consensus. And it's very clever. It comes from a professor in Chile, named Miguel Nussbaum, who's a brilliant technology designer. And what I love about the game is kids look at a question, they each try to independently answer the same question. And the software instead of saying who's right and who's wrong. It only says you agree or disagree. And if you disagree, it says talk to each other figure out which is the best answer, and then resubmit as a group, and it worked brilliantly. And it has all these social things. And I stay in touch with Matt. And I think that's a really interesting direction, because he's using the AI to orchestrate a small group to be productive. And that's something teachers, they don't have enough time to go to every group all the time. So it can really help out. So yeah, very excited about that product. I

Alexander Sarlin: 35:01

love the idea of maybe using storytelling or that idea of you know, climate change or other motivating contexts to be able to have that conversation, maybe could take it even to the next level. If it's, you know, five students sitting around thinking about how to use math to make a case to tell a company to reduce their carbon emission, I mean, just takes it to a whole new level of intrigue and interest. But I'm very excited about that kind of model as well. And I think people who think about AI and math often think about what do you call it, you know, tier one, you know, AI as a math tutor, or, you know, we see all of these apps that just call them scan and solve apps where you know, you take a picture of your math homework and it does it for you, or, if you're lucky, at least teaches you a little bit of the steps to solve it. And that's so much scratching the surface of what You know, can actually happen in mathematics. I love how you're laying it out.

Dr. Jeremy Roschelle: 35:50

Yeah,

Alexander Sarlin: 35:51

I'm a huge optimist around AI. But I have to ask because this is, you know, such a fraud moment everybody's trying to figure out how to use AI in classrooms and in education in ways that don't, you know, reduce teacher salaries, I don't feel like they put students at any kind of risk or have data privacy leaks, you know, what do you feel are the main risks when you talk to educators and districts with using AI in education? What are the risks that come to mind first, and what do you warn people about? And How seriously do you think we should take them?

Dr. Jeremy Roschelle: 36:21

The more time I spend thinking about AI, Aleks, the more seriously, I think we should take the risk. So I'm going to start with the most obvious ones, and then introduce maybe a couple other ones that are maybe a little less obvious, but privacy and data security are intrinsic. And the reason is, if we want it to adapt to the student, we have to know something about the student. And that implies you have some private information about the student, especially if you're trying to adapt stories to be based on their interests, well, you're going to need to know those interests. Somebody might want to know those interests to sell something to that student. And so there's there's real risks of that. And we still have work to be done. on that. One thing I'm excited about. brief aside here is I'm working on a National Science Foundation project called Safe insights, that's trying to figure out the right sorts of barriers to both protect the privacy but allow the research use of private data. Anyway, so that's a big trade off, if we want it to be personalized. If we want it to be adaptive, we're going to need to know about the students and then how do we protect that data about them, I think that one's the most obvious the fields knows about it. Second one is making sure the math doesn't have errors in it, especially if you're just using an LLM. And I do get annoyed at products that are just a thin wrapper on an LLM large language model. I think there's much more to it. And so we do have to be, you know, we can't be giving kids math that has errors in it, that's just bad. A more subtle one is I just worry about the reality distortion field, Steve Jobs kind of term associated with chat bots and AI can make us forget some of the basic principles of how students learn. When you mentioned, Dan Meyer, He's great on these kinds of topics like reminding people, when you start to use AI, don't check everything you know, at the door, and walk into the room with just the AI. So I do worry about overemphasis on the AI. And the last one I'll mention is AI is what my colleague, Judy Fosco calls an answer completion machine, it produces outputs quickly. But what we value in teaching and learning is the process. And I think we're going to find ourselves unfortunately, with a bit of an uphill battle, to keep students motivated to think about the process, when getting to the end result can happen in snap your fingers. And so I do worry about this, the overall output orientation of generative AI make it hard. And I know Khan Academy with comm ego, they spend so much time trying to get Kumiko to stop giving kids the answer. And make them think and that's what I mean, is we're gonna have to work Tripoli hard to avoid the innate tendency of the AI to want to get to the end result and slow down and do what's good for teaching.

Alexander Sarlin: 39:34

And it's really hard. That shift from answer to process is something that I think is a really fascinating shift in how how any education but maybe especially math education, where the answer is often the you know, the whole show your work paradigm of math, right that has been around for decades, is really trying to shift to that, you know, to that idea of the process is the is the important product. It feels like in a perfect world AI actually could open that up in a whole new way. I can imagine a world in which you know, what math homework is, is sitting with an AI bot and the entire session is the homework, you know what you ask it, what it says to you, what you graph what you upload. It's not that you're trying to get to a single answer, and that's the only thing you're submitting. It's the entire process of sitting down and trying to figure it out, is the submission and then you know, that would be too hard for a teacher to grade at scale, but I could get it graded pretty easily. So you could imagine something coming out of that saying, hey, this student had a really deep thorough conversation. They had three misconceptions, but they spotted what they were and they they discussed them with the AI and they came out with you know, a really powerful process, not product. Do you see that coming?

Dr. Jeremy Roschelle: 40:44

Absolutely. I Think both parts of what you said are key, in a show your work is going to be even more meaningful. And record your work will be part of that. But then giving teachers tools, I wouldn't go all the way to where AI has to be created. But AI can make it help easier for teachers to examine a longish work record of what show your work looks like, maybe help them identify some places that they want to comment on what the student did. And, you know, I even heard people from Adobe talk about this that it's going to be silly to look at whether images were partially created from Ai, or 100%, created from Ai, it's gonna get silly, because every image is gonna have some AI processing in it. But knowing the audit trail of an image, where did it start from? What tools were used along the way. So I think it's gonna be an industry trend to they'll need to be waste capture the trail of steps that went from an input to an output and to be able to browse and review those steps. So yeah, I look forward to that. I don't think a lot of people have started building those tools, yet. The only

Alexander Sarlin: 41:55

one I've seen that's even close is a tool called snorkel, which allows students to basically record their screen while they're solving problems. And then the entire recording becomes the submission to teachers, and they have a whole panel and it's moving in that direction in a way. That's pretty exciting. I have one more really strange question for you. It's sort of out of left field. But I'm very curious when you mentioned the simulations and the power of, you know, simulations for teachers to be able to practice teaching on simulated students. One idea that has sort of come up in the background as we navigate this AI world is, is there a world in which you could do research on simulated students? You know, we know that research on real students is very time consuming, slow and has high risks, because you never want to, you know, use substandard models on on real students. But if we had simulated students that were very realistic, and in a number of ways, could you, you know, put 10 different interventions in front of 10,000 different simulated students to see which one works best, do you see a future in which that's possible? Or is that just completely nuts.

Dr. Jeremy Roschelle: 42:57

And that future is here, researchers, like the researchers I work with in the Engage AI Institute, face a dearth of data often about how students are going to interact with a particular simulation or product that the students using. And within a particular product, you can start to make simulations of different students with different characteristics, use that to generate a huge volume of data and then use the huge volume of data to train your model. It sounds a little weird, but it actually is a promising approach. In those cases where a company is early with their product, they don't yet have 10,000 students banging on the product for six months. And if they could use simulated students, they can more quickly, you know, generate some examples of the path students are going to take. The one that I think is really hard is the idea like a general AI student, that you can put in front of any product, and it's going to do reasonable things. And that's a little out of reach. But yeah, people are working on simulated students as a way to deal with the relative dearth of data. That is

Alexander Sarlin: 44:07

incredibly exciting. I really thought that was a science fiction idea that might not be possible, the fact that it's already in action, just, you know, proves that we are we are through the looking glass in a really exciting new technological landscape. Dr. Rochelle, you have such incredible insight into so many aspects of this field, especially into the sort of the research aspects of it happening in all different areas. What is the most exciting trend that you see in the Ed Tech landscape right now that you feel like our listeners should keep an eye on what's something that you see coming that maybe what people aren't quite realizing is, is coming quickly or is going to be bigger than they expect?

Dr. Jeremy Roschelle: 44:46

I really think this business of storytelling and pivoting learning from a very task orientation, to an immersion in stories and doing your learning and stories becomes so much more possible. It's such a great model of learning. So I would ask listeners keep an eye out for ways they can ask students to be involved in story and different points of view in stories, not just stories told from one point of view. The flip side of that is developing comprehension is a natural thing that happens in stories, and is something that has to be learned skills in comprehending. So I think that uncomprehending is the same as what we were talking about before with conceptual understanding. So I really think one interesting trend is narrative centered learning and comprehension as real growth areas. And the other one, I think, is better supporting our students with disabilities, because AI is multimodal and we can Now really do Universal Design for Learning gives students different ways to receive input and provide their response. And that's exciting. So the disabilities area and the narrative centered learning area, two hot ones to keep an eye on. Yeah, fantastic.

Alexander Sarlin: 46:02

I love that term narrative centered learning. I've never heard that. But it's a story is, you know, sometimes talk to educators and say, you know, after we all leave the classroom, teachers and students, what does everybody do? They all hop on Netflix, they all hop, go read a book, they all you know, listen to a murder podcast, or watch a cartoon, everything people do sort of voluntarily is almost always story based. So it seems very fitting that it's coming into the classroom as a motivator. I love that term narrative centered. And then, you know, I'm sure you have a ton of resources. What are some resources that you would recommend for the listeners of this podcast? Who want to dive deeper into any of the topics we discussed today? Whether it's math, education, AI technology research?

Dr. Jeremy Roschelle: 46:46

Well, as a place for resources, I would really advise starting with LinkedIn, there's great conversations going on there. We mentioned Dan Meyer, I think he's writing some terrific things on the issues of safety. Ed, safe AI is active on LinkedIn, teach AI is active on issues of like, what is AI? Really, it's definitely worth following Ethan Moloch and seeing what he's writing all the time, because he explains it in pretty clear terms. Also, Stanford H AI has some wonderful resources broadly about AI if you just want to know where the field is going. And what's happening. Stanford Hai is great to follow. There's many more. And well, I'll say if you follow me on LinkedIn, I often retweet and we post all these things. So you'll discover it all pretty quickly. And I'm pretty easy to find just by typing in my last name and LinkedIn. And so yeah, I do invite all your listeners to follow 100%

Alexander Sarlin: 47:44

As somebody who has been following your work for quite a while you are well worth following, you're always queued into incredible insights into what's happening in this in this world. And it's obviously unless it has not been said yet, but it's definitely worth following all of the incredible work that Digital Promise is doing. You mentioned the learner variability project, there are so many interesting projects happening and coming out of digital promise that are just vital for people trying to stay on top of what's happening these days. Jeremy Roschelle, Executive Director of Learning Science Research at Digital Promise, and as I said, a giant in the learning science field, more than 100 publications, 25,000 citations 10 patents in the learning sciences really an honor to speak to you today and fascinating conversation. I feel like I'm walking away with a lot of big new ideas. Thanks for being here with us on Edtech Insiders.

Dr. Jeremy Roschelle: 48:37

It was great to talk with you Alex. You just have such a great impressions and intuitions and it was just a super natural conversation I really enjoyed myself.

Alexander Sarlin: 48:47

Thanks for listening to this episode of Edtech Insiders. If you liked the podcast, remember to rate it and share it with others in the EdTech community. For those who want even more Edtech Insider subscribe to the free Edtech Insiders newsletter on substack.