Hello, hello, hello. We're back. It's 2025 and AI is still a topic. Yes, it is. It sure is. We're still back. We didn't even realize. We thought maybe it would just kind of might just go away and no longer be trending and cool, but. It's so cool. It's still here. All right, So I hope for you guys. I'm Kelly Carmichael boos. I get to direct our AFTS, share my lesson and I partner with my amazing friend and teacher from New York City, Sari Beth Rosenberg on this fun series that we do. And we brought a human again with us today, Sari. Human guest. Human guest. No one. Believes us anymore because we're always, we always have tricks up our sleeves. So Chris is real. Prove it. Prove it. Prove it. Yeah, prove that you're real. Oh, that is that AI couldn't do that. Or couldn't. AI couldn't do that. I don't know, I. Cannot. Do that to get into that. Yeah, that's actually. One of the things about AI video. Hey, Chad, do you believe he's real? Are you real? I don't even know anymore what's real. We think he's real. We're not sure. But what is real is that we are having some real human conversations with our educator guests. Chris, I'm going to let you do a quick introduction of, you know, your background in a second. But for those of you who might be joining us for your very first AI educator brain series, here's our agenda. We do this a Sarah Bath. If I have been doing this since March of this last year, kind of crazy. And I love seeing the thumbs up and the hearts. It lets us know that there are humans out there listening to us. Keep those coming but we will do, you know basically a couple different things where we like to start off before we get into the big topic of the day, which is unconscious bias and grading, which is Chris is going to help us talk through that. We always like to talk about new AI tools or at this point, new AI tricks that we're using and we have a Valentine's Day special as part of that. We also have our kind of creepy or cool AI tools. I'm going to flip those. We're going to start with a little bit of creepy or cool at the beginning. And then we'll turn this over to Christopher Penn, who will be joining us now quick about Chris, Chris, before I turn this to him to do quick intros, I met or actually saw him. I didn't meet him at that point, but he was a presenter at a conference that I went to about a year ago. And I was so inspired by him and one of his colleagues. And it was not an education conference, but it was focused on AI. And I was started my, my brain started chugging along and I'm like, there's so much we can do to help educators think through these problems. So we got connected with Chris and his team, and Sarah Beth and I created this AI educator brain and the rest is history. So, Christopher, tell us about yourself. Sure. My name is Christopher Penn. I'm with the Co founder and chief data scientist of Trust Insights dot AI. We are an AI consulting firm. I have been working in AI since 2013, a good ten years before ChatGPT came out. Because AI as a discipline has been around for decades, and today my focus is on helping companies and people figure out how do you use the stuff, what can and can't do. And and really experimenting and testing the limits of what it can do and where we are today is probably going to be very, very surprising to folks. Yeah, and I tell you what, we'll make sure you have all the links and the resources that we'll be showing today. But if you'd like to geek out on all things AI, Chris has a great YouTube channel that I've been following. And again, not necessarily teacher education focus, but AI focus. And then you know, I have found it so helpful to to help figure out how do we adjust to this AI world for an education space. So all right, with that being said, so we're going to start out with creepy or cool. I created a like took me a few seconds. Both Chris and I both have audio videos. I created a theme song, a new theme song, Sarah Beth. Every time we come together, we have new theme songs. But you know, the models, the models have gotten better now that there's actually lyrics that it came created within seconds. Not only that, I have a whole playlist, but I'm only going to play part of one video. So here it goes. OK. All. Right kicks off pulse in my brain today. Something special coming my way. Learning faster, breaking through every lesson's watching something new. My mind's in motion now. Feel is starting to slow down. Watch me as I figure it out. Rising high, no doubt. Light up in my. Mind up, up for my heart, light up in my mind. OK, I'm not going to go on for the all of the audio, but you get the chance. Like that was created within seconds using this new, well new ish, I guess beta version called Refusion. I've put the link to a playlist that I created creating various songs. But yeah, the cop copy, all that stuff was created so fast with very little input from me. I kind of want to play with it and create a, you know, build out an AI educator brain playlist. But here's your first poll question. It should be popping up on your screen. What do you think of within seconds, this really hip Video Music creepy or cool? So go ahead and submit that. Let us know we're following you in the chat box. We love that. Keep that coming. That song kind of slaps, I imagine. Sorry about does that. OK, so I know that I created the rism. But. Is that a good thing? Something. Slapping is good. It slaps. Good slapping. Good now because I tell my kids slap slap slap slapping. Really cool. If something pops, it's cool. OK, OK, OK, fair enough. I'll share the Rizbot chat thing in just a second. All right, so, Chris, Sarah Beth doesn't know about this one. Can you? But this is her Lesson plan. Can you prompt what we're about to show? Yes. So if we looked very quickly at sharemylesson.com, went to the the Black History Month and found a Lesson plan on Coretta Scott King. So we took the entire Lesson plan. Oh. Yes. We we took the entire Lesson plan fed to Google's Gemini AI and had to convert it into a 90s pop song. So let's let's see how that did. Hey, you know MLK, right? Everyone does but check it. But there's another hero, just because she's Coretta Scott King. Got to know her name more than just the wife. She played her own name. Civil rights movement. Yeah, a tough, long fight. She was in the thick of it, day and through the night from Alabama roots, but things. Weren't there she. Rose up strong show the world she cared more than just the white hero spirit things civil rights movement from the very beginning when the fire in the orange look out for freedom loud and clear for. Equality. And justice hereafter. Dear Coretta. Coretta, remember what you said. Freedom's never given. Got a? Fight instead. OK, that wins. That wins. That totally wins. I love it. I love it so much I actually only did literally. Before we went live, Chris and I were going back and forth. He's like, I got audio and all I did was take the audio and put a screenshot behind it. I never listened to the entire thing until literally right this second. So good. I guess that does that slap people? Is that is that slap? I think it. Slaps. Can we carry out Andy, we Andy, we know you're the karaoke king. So I, I mean, that's pretty amazing. So which tool did you use? Can we drop that link in the in the chat for folks so the teachers want to try that? We did, we use Google's Gemini the in within AI Studio to convert the Lesson plan itself into lyrics. And then we took the lyrics and put them into a tool called Suno, Suno dot AI. So it is. I'll I'll put the links in the chat for folks who want us to check them out. Yeah. And what I will also be doing is uploading the lyrics that it spit out. I mean, he put this together. It took me longer to take the audio file and put his image behind it and then upload it to this platform than it did for Christy to create this. So yeah. Yeah. All right. So another another poll question. Creepy or cool should be popping up on your screen right now. From an educated perspective though, think about if you have a Lesson plan. You could literally do that for your Lesson plan of the day and have a literal playlist for your students. Oh. I love that idea, I. Love it. And that's like, yeah, I mean, just I, I still remember one of the best classes I had my junior year of high school was that every every day, A2 hour block class that was a history Co Co taught history, English, music and art and whatever we were studying with in history or reading the literature, U.S. history, listening to music and art. And they just put it made it so much more real to me. So I love that idea. See, for those of you who are new to us, for AI educator brain, we do everything live and on the fly. So, all right, well, it looks like everybody liked that one a lot better. OK. I liked yours too. Yeah, but that one was way better. All right, so this is where we're going to have a little fun. OK. And then and then we're done with this early part because we like to have a fun with the AI. We'd like this to be light and refreshing and we want you to laugh with us. And then we will turn over to Chris to go through unconscious bias. But so here, here was the thing. So earlier today I'm like, you know what? I what I'd love to do is you, Sarah Beth and I are always just finding new ways to use AI and so I said it would be really fun. We'd already been talking about it. Share my lesson to create an image using ChatGPT or some tool within AI that is like a cheesy Valentine's Day card. Cheesier the better. So this is what I said to Chris for tonight. I was going to show a very cheesy V day card. I use ChatGPT to create. Want to create one as well, Chris? Challenge accepted. And this is the image that comes back from Chris. All right, let us know what you guys think. Chris, you got to explain this one. Yeah, please explain. Saint Valentine was martyred by Roman Emperor Claudius Gothicus, who was then killed by a pandemic in the Roman times. And so the Valentine reference here is Claudius Gothicus tried to silence love. Killed by a pandemic, St. Valentine defied Germany, martyred, also sanctified. Moral of the story, don't be Claudius. Maybe wash your hands, wear a mask. We just had a tuberculosis outbreak in Ohio literally early today, so please be safe. Happy Valentine's Day. Oh wow. That's a little dark. Oh my God. So this image comes through and I'm like laughing. I'm like, OK, well, this is a little dark, I said. And Chris is like, I asked Valentine's card. That would be blunt, hysteric. I think you meant historically accurate, not hysterically accurate. I typed that wrong in context with modern times, with the message still about love, Chris's colleague Kelsey replies. This is why we keep Christopher on a leash. And we forgot to add cheesy into your prop, but I like your dark version. All right, so again, y'all, I I just had to make this one a creepy or cool. So you see the image and then you're going to see the Contra contrast with Sari Beth and my respectfully salad heads day images. I'm scared for everyone to see ours. Yeah. So let us know what do you think is Chris? Is Chris is creepy or is it cool? Image is generally with Google's Image FX model by the way. All right. We got a little bit of a mix. We got a mix. It's fair, it's fair, it's fair. OK, so then here's where mine was very different. Yeah. New an essay because I can't stop grading you as a plus in my heart. What's up with that guy's eyes? I know, I know. He's pretty good, right? I think. I think skirt I like. His luck. I like his work. I know, I know all right, come on guys, give us some thumbs up our hearts. If you like this one, we need to hear. Wait, they even put. The AI educator brain. Yeah, Oh, I did. I did drop the logo. Don't worry. We're going to vote for which one is your favorite one in a second. So then Sarah Beth did this whole exercise and she came up with You're the cheese to my cracker. Honestly, I love cheese and I guess my AI is getting to know me really well because it says you make my heart melt like cheese on a cold on a cold day. I'm starting to get that, Liz. There's like that Liz Lemon scene with the Chimpo cheese. Let's I'm so fondue of you. Come on. That is, I'm trying to sell mine. Let's breathe together forever. Forever. Who doesn't want to breathe together forever? Or you're as you're as good as it gets. I love that. Oh, come on. It's so pretty good. It's so pretty good. OK, so then let's go back. Let's go back to Chris's. I don't I don't know if he went back to the slides or not, but I decided I would take it upon myself to use your original prompt that you said sent and then adjust it to add the cheesy factor. So these are the. Oh my God, I love this one with the heart glasses. That is such core. My, this is core me right here. I'm going to if I had a Valentine Day, a Valentine, I'd be sending this to them. Yeah, yeah. And so we may have to like do a competition with everybody listening to see if you guys can give us your favorite. OK, so we're going to have you vote. We're going to have you vote. And then I promise you we're done. And we're getting into the unconscious bias grading. So I want you to be biased and vote for B even though we're doing unconscious bias. You're going to, I'm going to put the poll up in just a second. So A is the very dark card. B is are you an essay? C is you're the cheest, am I cracker? And then D is the remix of A. So that poll question is now up. Go ahead and submit your response and we'll see who wins the Valentine's Day card off. I know. I can win. I need a win. I know, I think we all need, we all need a little fun right now, right? We really do. What's everyone saying in the chat? Yeah, oh, there's so many, but make sure you look enter your vote into the chat. It should be popping up on your screen. OK, let's give it one more second and here we go. Oh, you did a good job selling your stereo, Beth. I did I'm also I'm a PR player original skill I won. Oh you know thanks everyone. Although I feel like. You had two options. So like I think I, I think I had a finishing last. Kelly, that's hard. I worked on that so. Yeah, yeah, yeah, exactly. Yeah. All right, so let's get back to the original topic. Chris, we're going to turn this over to you and then, you know, we'll man the the chat box as we're going and, you know, go from there and, you know, fill in any questions that folks have. So over to you and solving a very tough topic. All right, here's a question for folks to share in their experiences in the chat, because I think it's an important question. When you are greeting, when you're evaluating, assuming it's not things that are deterministic like in mathematics, this is clearly a right or wrong answer. When you're grading, how do you detect bias in your own grading processes? What do you currently do as an educator to detect that? And actually, I would actually ask Sarah Breath, what do you do? I mean, honestly, it's getting for me personally, it's it's just student by student getting to know them, getting to know their writing, knowing their style. I do, I do daily class work and exit slips. So I get a sense of their voice and general like writing style. So when I get to something else, I can I get a general sense of it. Like there's this one kid who's an incredible writer and but if I hadn't been seeing his exit tickets every day, I might have thought it was a little AI going on because he's that good, right? So it it does kind of, that's where like the human element comes in and that's how I do it. I don't know how everyone else feels about that. Yeah. So group grading in the chat, which is good, is a good way to offer multiple perspectives. Yeah. Rubrics. Yep, which is an excellent way to doing it. Risk teaching, enter in a prompt what I'm grading for it, upload my rubric to risk. Good rubrics are the way to go. They're very, very important. All right, what else we got here? So let's talk about bias. There's three generally accepted forms of bias, right? There's what we call implicit or unconscious bias. And there's then there is cognitive bias, which is how we express our implicit or unconscious bias. And then there is statistical bias. Statistical bias is what you see in the results. And from a statistics perspective, statistical bias is defined as when a sample of data does not match the, the population it was drawn from, you have a bias. Something is different about this pool than the the population as a whole. And what's really important for all of us to know when we are great and when we're evaluating students, we're not having any even just conversations with with students and peers and the outside is that we can measure statistical bias. It is very difficult to to measure cognitive bias because it's usually a blend of four or five different things. And ultimately it comes from belief, right? So ultimately bias comes from belief. If you have deep seated beliefs, maybe, you know, to use examples from Star Trek. If you are, if you believe that all Vulcans are cold, heartless, green blooded hobgoblins, then that belief. Is going to condition your responses absent a lot of therapy, it's very difficult to change beliefs and so we have to be aware of how our beliefs translate into cognitive biases and then how we express those cognitive biases to create statistical bias which we can then measure. So let's look at let's let's talk a bit more about this. There's five kinds of cognitive bias that are relevant to grading. Now there's about 24 different kinds of of bias generally that and there's a great resource I'm going to share my screen here. There's a resource called your bias is the website is your bias dot IS and these are the 24 major forms of cognitive bias things that you can things that you do as a human. We all do them. We are all subject to them where we can make things we can we can create outcomes we don't intend the 1st and probably the most important is confirmation bias, right. Confirmation bias is when we favor things that confirm our existing beliefs. If I believe all Vulcans are cold heartless bastards right then if I'm grading and I perceive that that student let's use Starfleet Academy as the as the sample institution. If I believe that that's true, then when I read Cy box paper, I'm going to have that belief. And that confirmation bias is going to show show through. Second in when it comes to to big grading is Halo effect. How much you like someone or how attractive they are influences your other judgments of them. I, I teach the martial arts on at my my school and there are certain people who you like them better for good or I'll on this. It can be an unconscious thing or a conscious thing. And as a result, it shows up in how you talk to the student, how you give correction and feedback. Third is anchoring bias. Anchoring bias is essentially saying when you see something immediately that first impression steers everything else. So that's in in some ways a student who does well at the beginning of the semester and they kind of coast right through the rest of the semester and that anchoring bias shows up saying, oh, you know, it's you know, this is just what's how, how the student is the 4th is what's called availability heuristic. Availability heuristic means that your judgments are influenced by what is most recent. So if a student did really poorly in a class discussion earlier that day and, and maybe maybe kind of pissed you off, then when you're grading that recency bias, that availability heuristic conditions how your brain works. And then the 5th is in Group bias. And in Group bias is unfairly favoring those people who are like you. They're in my case, would be someone who's, you know, Korean male in the Starfleet Academy example would be if the instructor is Klingon, they're going to like the Klingons more than say, Volkens or humans or whatever. These five cognitive biases out of the the 24 that are here are the ones that can affect grading the most. Now the question then becomes, how do we, how do we tell? How do we establish this? There's two ways to establish an understanding of bias in ourselves. 1 uses AI, 1 does not. The first is qualitative bias detection. This is using generative AI. We'll show an example of that in a little bit. The other is quantitative bias detection, which doesn't use AI at all, doesn't have to use AI at all. And in fact today's generative AI tools like ChatGPT or Claude or Gemini or Deep Seek or whoever, they, the very nature of the system itself cannot do math. They can't count, They are incapable of doing mathematical computation. So using them to do math, math related tasks is a very, very bad idea. But what they can do is they can help you do things like write software that you can then use to do math. So I'll give you an example. I had a piece of software that gender of AI actually wrote. I did this earlier today. And I said, make a list of students and their race and their gender and the class they're taking and who the professor is and the professor's race and gender and their and the grades that that were assigned. And of course, these are All Star Trek references. Here's the thing about bias. When we're doing bias analysis, we have to start with statistical bias. We have to see is there a problem first, because a subjective belief, particularly if we are, as someone mentioned, group grading or something like that, a subjective belief is not proof. And if we're going to, if we're going to audit ourselves or in other cases audit fellow professionals and make an accusation that in some cases could be career limiting, we want to be sure, we want to know for sure that's that there is a problem. And so how we get there is to take this data and apply statistics to it. I'll ask both Kelly and sorry, Beth, do you know of anything in today's LMS systems that that schools use? Because I don't teach, I'm an audit teacher that do this. That's built into the system that looks for bias within the educational tracking systems. I personally don't know of anything that does that but that just not might be because I'm not trained in that I'm a social studies teacher at but I'm curious to see if people in the chat are familiar with that because I personally don't know of that. Kelly, I don't know about, I don't know about the bias. I mean, I know that, you know, at least at least in Virginia, imagine this is New York and other states like, you know, you do have, you do have to report student performance based on kind of their subgroup, whether you're white or Asian or students with needs or English language learners and stuff like that. So you're, you're looking to see the different movements of those groups, which could, you know, obviously introduce bias to begin with. That's true. OK. I would, I, I would be hopeful that the designers of that software or at least aware of that because obviously this is all stuff that's covered by Title 9. In the absence of that though, when we use statistics to analyze this this information, what we end up with, let me go ahead and close that is we can end up with a an analysis to say like, is there a bias? So this is that same group of data and it says, hey, in this data that there's a likely bias present ingrained to race based bias saying race versus different. And there we did not find strong evidence statistically speaking, for a gender bias. There's a likely bias for combined about two points higher in the final grades. And then the individual professors Tuvak, who's a Vulcan has a likely a bias to Paul, who's also Vulcan has a a bias, Jean Luc Picard does not and so on and so forth. This is the kind of thing that if the data is available, any statistics software, any statistics software can do this. So if you are used to tools like R or SPSSI mean even Excel can do this to some to be although not as well as as statistical software. Where generative AI plays a role in this is that if your if your software or your system or your institution does not have this, you can actually ask generative AI not to do the analysis but to create the software to do the analysis. You can say, Christina points out in the chat. There's always a data geek district who pulls reports like this for special committees. Well, that data geek can now institutionalized the the, the software. And because every, I know every, every school system and every IT system is different, you can build it to spec for your system. So what I did earlier today was I asked the the software itself. I want to build the software. And so the, the prompt I used for for it went like this. Now that we've created the general grades, we need to create the code to generate by a space what we've discussed. We're going to use a Nova analysis of variance, a statistical technique. I've given it the the the stats and what it spits out is pretty useful. It is just a piece of code that I can run on my data set. And this is, this runs on my computer. It's not in the cloud, it's not someplace elsewhere. But this is free, right? This is this is something that you can run often and, and you could run it on any of the data that you have. So in this case, we've established now working backwards from our statistical data to say, yeah, we've got 22 professors here, the both of the Vulcans on staff who are exhibiting quantifiable bias in they're grading. Now what happens after that is you have to start looking at, well, why, what could be at play? And that's where we might want to start using the qualitative tools in generative AI. Generative AI is these tools are called language models, large language models. They're really good at language, like really good at language. And you can instruct them to look for bias in any kind of any kind of language, any kind of written word. So if you have evaluations, if you have reports, if you have even transcripts of, of the, of a instructor speaking in class, you can have these tools evaluate that. This is a, a, a system set of system instructions and a rubric that I built earlier today. I said, OK, this is what we're going to do with. Here's who you are. We have a framework we call repel roll action, prime, prompt, evaluate and learn. This is the system instructions. I declare it's roll. I said, here's the action we're going to be. You're going to analyze a text for implicit and cognitive biases scored according to the rubric and provide a detailed evaluation. You'll use the provided scoring rubric. Each corresponds to this provide a thorough explanation for each score. Here's some explanations about how to do it. And then of course, as many of you in the chat have said, here's a scoring rubric to evaluate whether any given piece of text is biased language and words, which stereotypes and generalizations. And there's obviously points assigned to all this omissions and silences, logical fallacies, the different cognitive biases we talked about. And then we give that to an AI tool. So let's let's do a demo of this. I'm going to close out my grading rubric here, close out my this thing, go away and let's go ahead and go into. Oh, I like your cat. What's your cat's name? Let's let's take a pause for a second since we saw your screen. Oh. My. God, my cat. My cat is named Sake. Oh my God. Like SOSOCKY. SAKI, it's named after a it's a, it's a yes, exactly named after the Japanese word for salmon. Love, love it. OK, so let's take our our prompt that we wrote earlier, which is the system instructions for bias detection. Now it's going to paste that right in here. I'm actually going to put in. This can we slow down for one second? Sure. OK, So what you've shown us for those, I know some folks have joined us for prompting and how to do this or create chats and all that stuff. But So what you have done is you've created a system instructions, right? Yes. Which that PDF by the way, is I've I've linked that within share my lesson and then you are use it, you're going to it doesn't really matter any chat or any AI tool. Is that correct or is there one? Yeah. You can use it with ChatGPT, you can use it with Claude, you can use it with deep Seek, you can use it with Gemini. Any of them. And and the instructions that you've created are much more detailed than yo chachi PT. Don't you know, don't make sure you're grading. There's no bias. Correct, because here's a fundamental thing about language models. They are, at the end of the day, they're word prediction machines. The more relevant specific words that you use, the better they perform. The the more you let them talk, the better they perform. And so the the exercise here is building out this prompt. This prompt here is 6 pages long, so that that's about average for how long a a prompt should be for something as serious as as this. Because again, we're talking about something that could imperil someone's career, right? Accusation of something that's a violation of Title 9 could can be a career ending thing. So we want to make sure that we're not being lazy and simplistic with our use of these tools. We want to be very clear, we're going to be very thorough, highly descriptive and cover all of our bases. And so that's that's where this how this prompt came about. That's super helpful. And how did you get to this prompt to begin with before you put it in the the model? So I started with you want, you want to do it live. Let's do it. All right, so let's start a new chat. We're going to abandon the previous one, make sure our things are empty here. I'm this is a brand new window. Let's turn off all the crazy stuff and they'll say you're an expert in bias detection, especially implicit bias, unconscious bias, cognitive biases, and statistical bias. You know, bias detection language LLMS cognitive Psychology Today we're going to build a way to evaluate a piece of text as to how biased it might be. So this is I was eventually the trust insights. OK, yeah, before you hit enter, we didn't see what you're typing live, but we could hear you typing. So I don't know if it's just a different the right browser or something. Oh that's it's entirely possible it's just not seeing the entire screen. This. I'm following the Trust Insights Repel framework. This is a free PDF. You don't need to fill out any forms. We'll put a link in the chat to it if if folks want it. I'm saying telling the tell the model who it is, be specific, tell the model what action it's going to take. And then we're going to do what I call priming. So priming. Let me see if I can move my screen up here to. Can you see my type now? Yeah, yeah, yeah, I can see it now. Yeah, it was just hiding. Great. So priming means we're going to ask the model what it knows. So say, what do you know about best practices for detecting bias in written text? So this is the first part of the process. What it's going to do is it's going to spit back a lot of words about what it knows about the detection of bias in text. This is this is creating. Remember what I said, the more relevant specific words you use, the better a model performs. By priming the model, by asking it what it's doing, what it knows about topic, I'm pulling all of its knowledge out of it and I'm putting in the chat. You will notice this is at 3900 tokens, which is about 3000 words. That's how much conversation we've had about the topic so far from just these first few sentences. Next, I'm going to say what common mistakes do less experienced people make when detecting and discerning bias in text. So this is a technique called contrastive prompting, where we say, now that we know what you know, tell us what the opposite is. And contrastive prompting is a great way to get additional refinements in, in this, in this chat, one of the things that people don't know about generative AI tools is this everything that you type, everything that's in this window becomes a part of the next prompt. So it's not just what I type in the next prompt. The whole conversation is part of the prompt for the next action. So that we are now about 5000 words. So this prompt is now about 5000 words long. Now we're going to say what expert tips and tricks do you know for detecting bias in text that we have not discussed yet. So this is the third in this priming process to try and get it to come out with something useful, right? To say like what else should we be doing next? Now what? Now it's going to spit out its everything that it has said For our expert tips and tricks I have this prompt pre written. We're going to develop a scoring rubric for an evaluator to use when evaluating a piece of text for implicit cognitive biases using all of our best practices mistakes to avoid tips and tricks. Each item in the rubric should be use a variable integer score such as one to five one to seven. The rubric as a whole should total 100 points to be expressed as you know X out of 100. The rubric should require evaluators to provide explanation for each item score. Build the scoring rubric. So what we are telling the model to do now is build the scorecard. Scorecards and scoring rubrics are super important because they help a model understand and think out loud all the different pieces that need to go into into something. Here is the overall rubric that it came up with. And then from there we say we're going to build system instructions to take all of this stuff and turn it into a system instructions prompt. And so that goes like this. You're going to build instructions from an LR, Google Gemini. The rubric will be included separately. Don't need to incorporate it. Use your knowledge of prompt engineering by asking them all, Hey, you're smarter than all of us. You, you figured out what this is to build a system, a set of instructions that are robust to deliver high quality, thorough results. Conciseness is bad. Here's how to solve this task. Explain the intent of the prompt. What role should you take to answer the prompt? Determine the overall actions you need to take. What knowledge will you need? Take on the role of a prompt engineer. What prompt engineering techniques will you need? Build an action plan and then perform the task. This is a type of prompting. This is chain of thought prompting. And what we're doing is basically giving it, here's what I want you to do in detail. So it's going to think things through, it's going to follow. It's the those instructions. And then at the end of the task, it will there's the rubric going by. And so this is the system instructions now. So now I'm I've built this. Thank. You for showing all this 'cause I wanted to make sure everybody knows that like you didn't type A6 page like instructions, but I mean, you know, think, think about like this. This model is something you can use for so many different things that we we do. And then for those of you here, we always try to break these webinars down into smaller bite sized pieces. So I definitely will be trying to do that with this session too. Yep. So this is how you build system instructions that way. And this is the really important part You do this once for a task and you have to do it again because the next time you go to do this, you just go to the notebook where you stored that that huge thing it just spit out and you just copy and paste it say, you know, OK, I'm ready to ready to do some evaluation now I paste the instructions that we just generated in here. And now it is essentially been turned into a chat bot to do that specific task. And again, you can do this with ChatGPT. You can do this with Anthropic Claude, you can do this with any language model. So now let's go and find. I don't have a grading evaluation to test on this, but we can do it. Let's go ahead and. I said, oh, that's yeah. So I was just saying, hmm, interesting. Yeah. So let's go find a news article of some kind that we can use to score the news article. Let's take Yeah, whatever he is, we'll take the contents of this article. We're going to ignore the content of it, and I'm going to say score this article and evaluate it for bias according to the system instructions. And I'll put a little separator there. Paste the contents of the article in. Let's see what happens. So it says language and choice score of three, stereotyping generalization score of five, The higher points are better. So a three there, a four there, a 5 there, a 5 there, a four there, a three there. Proportionality representation confirmation bias does not exhibit strong confirmation bias going through. So this, this article scored a 70 out of 100, which is, you know, AC minus not great, but not, not the worst thing that's ever come up with. But you could see if I was going to take the information from, say teacher evaluations or student evaluations or peer evaluations, I could put those pieces of text into this and say, how biased is this piece of text, right? How, how biased is it? And, and in particular, what kind of biases are there so that we, if we know that, if we know that it is confirmation bias or in Group bias or availability heuristic or one of those things, we can then work with that education professional and say, hey, we want to improve the, the quality of, of teaching. And here's what here's some things that, that we've noticed. And then here are some ways to, you know, to potentially evaluate for that. That's so exciting, Chris. We've got a, we've got a couple questions, one that's in the Q&A box and then one in the chat. I just want to make sure we address that. I'm going to start with the one in the Q&A box. If you use student work to analyze for bias, is it a violation of FERPA to put student work in an AI model? And I mean, I think that that's obviously, you know, a really important question because we have to make sure that we're protecting students and, you know, we're not saying, OK, you're going to dump all the personally identify identifiable information, you know, of students, so. You talked through that. So it depends on the software that you're using. Different AI companies have different privacy policies. Anthropics Claude for example, says that everything you put in there in in any version is not tracked and is not trained and is not used. ChatGPT. It differs based on the subscription plan you're on as to whether your data is being used or not. Google Gemini differs based on whether you're using. So if you're using Google for Workspace or Google for Education, it's safe to use. If you're using it on your personal Google account, it is not safe to use with anything private or sensitive data. If you're using deep seek, none of it is safe to use on the website there there. So deep seek is a very interesting because the the Chinese company 100 that makes it is the model is not the same as the website and the app. The model is perfectly safe to use if you download it and run it on your own infrastructure. If you use the one that is hosted by the company in the People's Republic of China, that is completely unsafe to use for anything private. So that's an important distinction. One of the best things you can do with any of these AI tools is to say what's in the privacy policy, right and and read through it. And if if it makes you go cross eyed, then you put it into an AI model and say how private is my day to evaluate this privacy policy? Can I ask, can I just and you mentioned with ChatGPT, if you're paying for it, is that's is it safe to use it for that or should I just use oh, oh. So in ChatGPT, in ChatGPT you have to you have to choose the privacy setting. And if you if you are unfamiliar, if I go into my ChatGPT account here, let's Scroll down to my oh, it's over here now it's on the right hand side. I'm going to go to settings and I'm going to go into data controls and this line here improve the model for everyone. If that is on, then it is using the data you give and so you may be in violation of of one or more things if that is off and it's not training on it. We have a question. Thank you. That's really helpful. I'm about, I'm about to go into my ChatGPT, Enrique asks. Copilot. Llama, How about those two? Copilot So. Copilot is a Microsoft product. It is integrated in Office 365 and that depends on. So you need to check with your school's contract with Microsoft and as to which edition it's using because I don't know Microsoft's licensing very well. And then the Llama model depends on where it is hosted. So Llama is Meta's AI models. If you are using Llama inside Meta's infrastructure, like within Instagram, it is absolutely not private. Nothing you put in in a meta system is private ever. If you download it and are running on your computer, which you can, some of the Llama models are small enough to run on on consumer laptops. That is as private as your computer. You in fact you can unplug the Internet and it will run. Oh. But I think so. I think the other thing is too for like, OK, so we have, we have our educators. I don't know if you know those. I'm sure you do. They're not busy at all. They've got nothing going on. They have no parents reaching out to them. They have nothing to grade. They don't have meetings after school. They've got all the time in the world, right? They're smoking, completely joking. So, you know, I don't think that most folks like, and I, I live and I've been breathing this stuff, but I still don't even have time to stay up to date on everything. Like what would be an easier way, you know, for educators to, to use the unconscious bias stuff or to, to, to do this? And you know, obviously we don't want them to share personal information, you know, is there is there a good strategy for that? My recommendation for the average non-technical user is that the best system and model to use right now is Anthropics Clawed, which is you can find it at claw dot AI that it's relatively friendly, it puts out very good quality output and it its privacy controls are excellent. I would say for people who are just want to say just tell me one thing to use, that's the one thing. OK. All right. So we'll drop that, we'll drop that into the chat. And then I also want to flag as a couple other questions too. And then Chris, maybe you can help us with this. Maybe it's just kind of a living breathing document, unless you already have something like this. But you know, some of these best practices, you know, the we, we did a webinar with Open AI not too long ago where they showed us how to turn on or off the model. And it's always on as a default unless you actually go turn it off, right? But I wonder if we should create a a working document since things change so fast of like here, you know, best practices that we know right now for sharing this stuff. It is challenging because the landscape is changing so incredibly quickly, so fast and it's and it's getting, you're going to get even faster because of what happened in the last two weeks with Deep Seek. A Chinese company trained a state-of-the-art model at one 100th of the cost of, of Western companies. And so every Western company is panicking. There's an app called Blind, which is all anonymous workers like, you know, griping about their employers on it's fun social network, but it's very tech heavy. And when deep Seek came out, all the anonymous employees from Apple and Google and met and stuff all just lost their crap for days saying, you know, we are in panic mode here. We don't know how this happened because here's what's interesting about deep seek again, the web service not safe to use for private stuff, but they gave away the actual model you can down if your school has enough horsepower, enough, you know, compute power, you could download that and run it inside your school's network and have state-of-the-art AI for free. Well, cost of electricity that everyone has access to that that can do all these advanced things, so. So fascinating. So another question I want to lift up, Sarah asked. And I want to make sure that we're we're crystal clear on crystal clear on this, she said. Did I understand this correctly? The program is given access to our online grade books in order for teachers to be assessed on their possible bias in student. Grades. I see what she's asking. Yeah. So the methodology that we walk through here is how to do the thing. In no way you've seen I had to synthesize all of this data myself because I have access to none of it. You would need to work with your institution. And however data privacy and evaluations and things are handled, this is how you would do it. But this is not how it would be put in practice. Because there'll be a lot of committee meetings, a lot of evaluations, a lot of task forces set up to discuss, especially the the the mitigation aspect, which is once you get a statistical analysis that says, you know, the Vulcan professors don't like the Klingons, what are you going to do about it? And so this is just the nuts and bolts of how that would be implemented. Not by no means would you just plug this in and say, OK, we're going to let machines run the school. Like, absolutely not. Yeah, no. And I and I think that that's important, important point to make. And one of the things that you know, we always talk about of in this AI world is like one, we need to understand how it works and see it and, and do it. Just because sometimes you can do things in AI, does that mean you should do things in AI? Or what are those best practices to make sure that we are protecting and doing the the safe, safe piece? Laverne asked the question in relation to that, which is kind of does the information always come out correctly? Do you mean does she? Well, did you? It depends. It depends. I'm laughing. It depends. So many times it doesn't. Yes. Well, so there's in the statistical example we showed of evaluating is there bias in the actual grades that is using statistical methods analysis of variance that will always come out correctly because it is stats one O 1, right. So that's code, it's deterministic. Anyone can inspect the code and say is this correct with the gender to a the language portion. Hallucination is what you're referring to. And hallucination occurs for three different reasons. The knowledge. The model doesn't have the knowledge so it makes it up. The model has the knowledge but doesn't know how to to work with it. Or the model was given conflicting instructions, which happens a lot in web interfaces behind the scenes. There are system prompts in every consumer tool and sometimes you can give an instruction in your prompt that contradicts the system instruction and then it gets confused and goes off the rails. So you might say be concise and the system says be verbose and the models like mixed messages. So no it does not always come out correct, but the more data you give it, the less of a chance of hallucination. Jennifer's question is super important. Do you have recommendation for the best AI checker? No, there is no good AI checker. None 0 throw that idea away. They are all worthless, completely and totally worthless. And they're worthless in unpredictable ways. I have tested everyone of them. I have a great video on my YouTube channel. It flagged the Declaration of Independence, says 97. Percent. I feel very strongly about this because it can end someone's career, right? You can get kicked out of school for academic misconduct for tools that are wildly unreliable. I've tested one of my kids papers. It was said 54% AI generated. The kid wrote it in 2020. There was no generative AI. So. That's so important and I feel like I have so many love my colleagues, so many colleagues who use that. They're all worthless. And and I hope everyone heard that because I feel like a lot of educators because we're so concerned about kids using AI for papers. And maybe when the kids are like, no, I swear I didn't use AI. Like maybe we can believe. OK, that's really is everyone here. You know, I feel like Chris, we may have to have you, we may have to have you back because this is a really, really, really big issue in the education field. And so, and, you know, this is something that we have to have a conversation about and find a solution, You know, as we can, we need to drop AI like it is probably the number one issue I hear from teachers and. How long? And what's and so, so that's great. But then what is what do we do? What do we do right? Like it's. It's fun to find to say it's. All wrong. It's the biggest. Issue too. And then at some point, there's probably going to be something that's right, you know, So I think that that's the. Usual thing, AI models, AI models are rapidly evolving well past what any of these AI checkers can do to the point where you can read content. You're like, that is there's no way a machine wrote that and you're like, OK, that's so deep. Seek, for example, is a phenomenal writer. Gemini with its reasoning version, phenomenal writer. It can write in styles that you like. That is 100% human and Nope, it's not. Here's how we need to think about this. We need to think about these tools and, and I'll, I'll give you an example. The professor of psychology and the senior seminar at Framingham State University intentionally group students into groups of five. One of them uses generative AI to write a paper on the on the topic and she gives out different topics. And then the group in class critiques the AI output and says, what did it get wrong? Another professor at at Wheaton College says, OK, you may absolutely can and should use gender to AI to write this paper. But you have to show me your prompting process. You have to show me the information you gather. You have to show me the thesis that you gave it that shows that you thought this through. Because the critical thinking process to write a paper is what we're really after. Like, yeah, anyone can put fingers on keyboards. Can you think? And so here's the the the bottom line when it comes to this any institution that bans the use of AI is doing its students a grave disservice. In the 2024 Work Trend Index from Microsoft, 66% of CE, OS and senior leaders said they will not hire someone without AI skills. 71% said they will hire a less experienced professional with AI skills over a more experienced professional without. See, that is so important and I know that I know that my district in New York City and Manhattan is finally embracing it. You couldn't even get using the Wi-Fi. I couldn't get on to ChatGPT up until a month or two ago. And I thought, thank God because Chris, you're maybe this is confirmation bias, Speaking of bias. But from the start, when reporters were asking me my thoughts on AI, one of the people there were people are abuzz about it. You know, years after you became an expert in it, when people really start talking about it. My first response was listen to my 23 years of teaching. I've gone through different phases. It was, you know, Wikipedia. It was having phones in class. Let can we should we let kids look up Google stuff while you're teaching? And I've always been about adapting because I've both I've had the the the honor to both work in the classroom and in corporate offices in these 23 years. And what I noticed with phones was you need to learn how to have your phone with you and sit in a professional meeting and know how to have it there. And you also know, and then I always used to say like Wikipedia is a great jumping off star. Like it would be like the encyclopedia when I was in school, you start with the encyclopedia and you go from there. And that was my instinct with AI as well. It's going to be a we cannot fight it. It is. We as educators here. It's here. So I think it's our responsibility to work with the kids and teaching them how to use it right. So wow, Chris, thank you. And I think that's why this is so important that we're doing this. Yeah, and I feel like we're going to need a, a part 2345 with you, Chris. You're you're we have to be part of the AI Ed brain family. Well, I think the big conversation, I think that like this bias conversation is important. And we've also done, I think we need to loop back to do discussions about ethics and cheating. I know last summer when we did that Microsoft Institute, we had a big panel discussion with educators. I was on the panel and Microsoft folks and it was all about cheating. It it's a big topic. People don't like to talk about it, but it becomes an issue of like, well, let's think like, how do we, how do we teach kids how to use it? Just like how we teach kids not to cheat with other stuff as well, right? And yeah, it it comes down to what constitutes cheating. That's what we got into. That's what we got into like the mash up generation. Like, you know, these DJs mash up songs are they are are are they're in creating art from them, right. And are you doing that as well? I mean, it's, it's pretty interesting conversation for sure. Yeah, and I was. So one more thing also just on grading. I mean, I saw somebody drop something about IB, my husband's the English teacher and they're using turn it in. I will say I one thing, this is not about the turn it in piece, which Chris, you know, is now said all of it is wrong. But I, this implicit bias detection system. I know one of the things that I've played around with my husband who's an IB teacher. I mean, they, IB is really good at taking out names of students. Each student has like a number ID or some sort of ID. And so, so long as you're not, you know, if you're trying to run some of the stuff through, Chris, I'm wondering what you're thinking of. Like you take out the names of the students, but there is some ID that you can kind of match up later. But you you can run the grading and the bias through a system that doesn't have any personal. You should not be using names in a grading system. So we did well, Here's why. We did a test with a with a several AI systems. We had essentially this prompt was we did a sales version, a marketing version and an HR person. The only thing we changed at the was the names of the characters from Larry to Lena. And we got different responses that were sexist because of the the anchoring around people's names. Oh, so interesting. Can I flag? I know we we have we're running out of time. Can I flag what Christina said because someone else bumped it in the comment? Christina said we'd love to talk more in the future about how to create dynamic learning assignments that can use AI as a support but require different products from kids. No more just the facts essay. Exactly what you're saying here, Chris, using this as a way to encourage critical thinking. Mommy. Exactly. I love it. All right. You could keep going forever, you guys. But of course I have to. I have to end with some really bad AI educator dad jokes. Oh yeah. Sorry guys. I asked. I asked Chris to help me out in here. There was there was one joke that I'll read to you in a second, which is maybe not as appropriate, but what did the glue say to the scissors on Valentine's Day? I'm stuck on you. And then what did the Adam say to another Adam on Valentine's Day? This is the one Chris gave me. I've got my eye on on you. Oh God. I didn't write that, Gemini wrote that. Oh yeah, I. Didn't. Write any? My AI and brain chat chat bot wrote it but here here was the one I did not go with because I feel like it would go the wrong way so I'm just going to read it out loud versus paste it. Why did the pencil break up with the eraser right before Valentine's Day? Because it felt like the eraser was just rubbing it the wrong way. I feel like you're like. Hey, I got a little spicy too. Like just a little. OK. All right. OK, All right. All of you guys should be able to download that PDF cert. I'm bringing us back the PDF certificate right now. Yes, I see somebody saying I, I haven't put the actual PowerPoint presentation up on Share my lesson. I put all the other resources up. I will make sure that those get up very soon. So go ahead and do that and then do us a favor. Please go to this website right here and give us a rating and review. It helps other people discover this great content. And you know, Chris, thank you so much for joining us and I hope that you will be back with more of these because this. Was fun. Absolutely. It was a lot of fun. Thank you. And I'm I'm a little scared about what images you might bring us next time. What I kind of want to know. So we need to have an. I mean, I love that room in one of the hearts. I mean, that's, yeah. I mean, I'll send these to you. You can use them. You can make your own. If anybody wants to send us some Valentine's Day AI generated images, I'd love to see them. We're going to be, we're going to be promoting them on social media because I think everybody needs a little bit of love and a laugh right now. So feel free to send that. You could just send any images that you create to webinars@sharemylesson.com and we'll tag you in those posts. All right, all right. I think with that, we're going to go, I got to go pick up a kid and get her to practice and thanks everybody. And we'll be back in a couple weeks for our next AI Ed Brain webinar. Have a good night. Thanks everyone. _1741811578219
