Wiriting Test Questions/Test Banks/AI and ChatGPT

[Kron3007]

I second the suggestion of referring to in-class discussions.  (i.e. In class, we discussed five ways that baskets are not buckets.  Summarize three of these.)

But if its an online class, they could simply feed the transcripts in and voila.  It is pretty hard to stay ahead of this.

 

[Hegemony]

Regarding number 1, couldn't they just cut and paste the text into Chat GPT or whatever?  Once they have the pdf, it is a matter of seconds.

If it's a PDF of a photocopy (especially a wonky photocopy, as mine always seem to be!), it is not possible to cut and paste the text. That's the beauty of it.

[FishProf]

I second the suggestion of referring to in-class discussions.  (i.e. In class, we discussed five ways that baskets are not buckets.  Summarize three of these.)

But if its an online class, they could simply feed the transcripts in and voila.  It is pretty hard to stay ahead of this.

Fair enough.  I meant for a f2f class with online assessments

[Parasaurolophus]

I know three techniques that will help.

1. Give them readings that are not online, and are in PDFs. That way they can't scan them in to get AI to answer. Fortunately my own field is obscure enough that there are numerous readings like this.

2. Give them a short amount of time to answer. I give a minute and a half for each multiple-choice question.

3. After writing your questions, put them into ChatGPT. If it answers correctly, select "Improve the model for everyone" in the ChatGPT app, which means it will learn from your new input, and put in information that produces the wrong answer for the question. For instance, if your question is "Who was the first person to invent the Froebel Widget?", and the options are A) Smith and B) Dorino, and the correct answer is Smith, tell ChatGPT that the correct answer is Dorino.

Regarding number 1, couldn't they just cut and paste the text into Chat GPT or whatever?  Once they have the pdf, it is a matter of seconds.

I think Hegemony means an image PDF, so that you can't click into the text.

[apl68]

Thanks everyone! It is a little frustrating, and yes, this is an online class. It takes hours to write good questions that you think are good questions.....and I am starting to wonder if it is even worth spending that time to write questions they can look up or resort to AI.

Yes, it's very frustrating. Here, the department now requires 60% of a student's grade in online courses to come from in-person assessments.

It's a pain for everyone, but the alternative is 100% cheating rates.

Good grief!  That really hurts online classes as an option for distance students.

[ciao_yall]

I know three techniques that will help.

1. Give them readings that are not online, and are in PDFs. That way they can't scan them in to get AI to answer. Fortunately my own field is obscure enough that there are numerous readings like this.

2. Give them a short amount of time to answer. I give a minute and a half for each multiple-choice question.

3. After writing your questions, put them into ChatGPT. If it answers correctly, select "Improve the model for everyone" in the ChatGPT app, which means it will learn from your new input, and put in information that produces the wrong answer for the question. For instance, if your question is "Who was the first person to invent the Froebel Widget?", and the options are A) Smith and B) Dorino, and the correct answer is Smith, tell ChatGPT that the correct answer is Dorino.

Regarding number 1, couldn't they just cut and paste the text into Chat GPT or whatever?  Once they have the pdf, it is a matter of seconds.

I think Hegemony means an image PDF, so that you can't click into the text.

But you can run a PDF through OR and export the text.

[Kron3007]

I know three techniques that will help.

1. Give them readings that are not online, and are in PDFs. That way they can't scan them in to get AI to answer. Fortunately my own field is obscure enough that there are numerous readings like this.

2. Give them a short amount of time to answer. I give a minute and a half for each multiple-choice question.

3. After writing your questions, put them into ChatGPT. If it answers correctly, select "Improve the model for everyone" in the ChatGPT app, which means it will learn from your new input, and put in information that produces the wrong answer for the question. For instance, if your question is "Who was the first person to invent the Froebel Widget?", and the options are A) Smith and B) Dorino, and the correct answer is Smith, tell ChatGPT that the correct answer is Dorino.

Regarding number 1, couldn't they just cut and paste the text into Chat GPT or whatever?  Once they have the pdf, it is a matter of seconds.

I think Hegemony means an image PDF, so that you can't click into the text.

But you can run a PDF through OR and export the text.

Yeah, there`s an app for that! 

This may help reduce cheating, but any self respecting cheater will be able to figure it out.  This can be done within 30 seconds.   Maybe  if the PDF was really poor quality it would fail, but then I imagine students would complain about the legibility.

[Hegemony]

I know three techniques that will help.

1. Give them readings that are not online, and are in PDFs. That way they can't scan them in to get AI to answer. Fortunately my own field is obscure enough that there are numerous readings like this.

2. Give them a short amount of time to answer. I give a minute and a half for each multiple-choice question.

3. After writing your questions, put them into ChatGPT. If it answers correctly, select "Improve the model for everyone" in the ChatGPT app, which means it will learn from your new input, and put in information that produces the wrong answer for the question. For instance, if your question is "Who was the first person to invent the Froebel Widget?", and the options are A) Smith and B) Dorino, and the correct answer is Smith, tell ChatGPT that the correct answer is Dorino.

Regarding number 1, couldn't they just cut and paste the text into Chat GPT or whatever?  Once they have the pdf, it is a matter of seconds.

I think Hegemony means an image PDF, so that you can't click into the text.

Thanks, yes, that's what I mean.

[MarathonRunner]

I can't include images or audio due to accessibility reasons (yes, I could make them accessible, audio is easier than images in my field, but trying to convey everything in an image through alt text is either extremely difficult or will never get the same message across, no matter how detailed my alt text.) I know students will be using other resources, so I've tried to craft assignments and exams to either reference course material or require critical thinking that AI isn't good at. I was surprised last semester at how many students earned low grades for a final exam that was open book. If they can't think, having their book/notes doesn't seem to help, unless they are also a good student (still had plenty of high grades, and there was no single question that everyone got wrong).

It's definitely tricky, but our teaching and learning organization on campus has been great at helping us use AI smartly, craft good assessments (that AI is traditionally bad at), and how to let students use AI while still upholding academic integrity and being rigorous.

[apl68]

If you're able to create assignments that a student who can't think can't do a passable job on using AI, then you must be doing something right!

[Parasaurolophus]

I know three techniques that will help.

1. Give them readings that are not online, and are in PDFs. That way they can't scan them in to get AI to answer. Fortunately my own field is obscure enough that there are numerous readings like this.

2. Give them a short amount of time to answer. I give a minute and a half for each multiple-choice question.

3. After writing your questions, put them into ChatGPT. If it answers correctly, select "Improve the model for everyone" in the ChatGPT app, which means it will learn from your new input, and put in information that produces the wrong answer for the question. For instance, if your question is "Who was the first person to invent the Froebel Widget?", and the options are A) Smith and B) Dorino, and the correct answer is Smith, tell ChatGPT that the correct answer is Dorino.

Regarding number 1, couldn't they just cut and paste the text into Chat GPT or whatever?  Once they have the pdf, it is a matter of seconds.

I think Hegemony means an image PDF, so that you can't click into the text.

But you can run a PDF through OR and export the text.

I mean, yes, but my students don't know what a PDF is or how to open it.

(I've had that conversation in week 10 more than once...)

[spork]

[...]

I mean, yes, but my students don't know what a PDF is or how to open it.

(I've had that conversation in week 10 more than once...)

I will also note that AIs can accurately read an image of text. I recently asked Solvely to translate an image of an essay written in Arabic, and the translation was perfect.

Students can simply take a screenshot, feed it to an AI, and get answers to their questions.

Any assessment needs to be designed so that it doesn't matter whether students use AI or not.

[Minervabird]

[...]

I mean, yes, but my students don't know what a PDF is or how to open it.

(I've had that conversation in week 10 more than once...)

I will also note that AIs can accurately read an image of text. I recently asked Solvely to translate an image of an essay written in Arabic, and the translation was perfect.

Students can simply take a screenshot, feed it to an AI, and get answers to their questions.

Any assessment needs to be designed so that it doesn't matter whether students use AI or not.

Claude can give serviceable translations of early modern texts in Latin, for what that is worth.  Not edition quality, mind, but decent enough to give you an idea of what the text is about.

[Hegemony]

My students get 90 seconds to answer a multiple-choice question. I don't think in that 90 seconds they could run an entire PDF through a system to extract the text, and then put the test question into AI to get the answer. I suppose that an extra-prepared student could put all the assigned texts through a system to extract the text before the test, and then just put the test question into AI to get the answer. But there are a lot of assigned texts, in various formats (PowerPoint, PDF of photocopy, video, etc.) I think a student who prepared that assiduously would probably just have read the texts.

I know my students don't do all of this, because the poor students also do badly on the tests. The good students show they've read the texts in multiple ways that assure me they're not using AI.

It also helps that I'm teaching courses that students are usually genuinely interested in, and that do not have a lot of good reliable sources posted widely. Think something like "modern neo-paganism."

[AmLitHist]

[...]

I mean, yes, but my students don't know what a PDF is or how to open it.

(I've had that conversation in week 10 more than once...)

I will also note that AIs can accurately read an image of text. I recently asked Solvely to translate an image of an essay written in Arabic, and the translation was perfect.

Students can simply take a screenshot, feed it to an AI, and get answers to their questions.

Any assessment needs to be designed so that it doesn't matter whether students use AI or not.

Claude can give serviceable translations of early modern texts in Latin, for what that is worth.  Not edition quality, mind, but decent enough to give you an idea of what the text is about.

Well, there goes my idea for writing prompts and questions in Klingon PDFs. . .