Scientific literacy course for non-scientists

[Caracal]

This sounds like an outstanding idea. I second the recommendation to discuss HARKing, and also what Gelman calls the Garden of Forking Paths.

I wonder whether it might be useful to organize these topics not in terms of sub-disciplines (social sciences people use statistics and like to extrapolate data too) but in terms of something more structural about the process of quantitative reasoning. For example:

Unit 1: What can we learn from numbers
-causation vs. correlation
-probability (Type I vs. Type II error)
-randomness

Unit 2: How can we learn it?
-double blind and observational studies
-from individuals to populations
-polling, surveys

Unit 3: What can go wrong when operating in good faith?
-measurement uncertainty and error propagation
-cognitive biases and experimental design
-Data modeling and extrapolation; specifically how widely model extrapolations can diverge
-selection bias
-underpowered experiments and Type M error
-garden of forking paths

Unit 4: What can go wrong when operating in bad faith?
-p-hacking
-HARKing

Unit 5: The publication problem
-file-drawer effects and publication bias
-preregistration and open science

Seems like its quickly going in to areas that are of concern to scientists and are certainly important to teach to majors, but aren't things that everyone else needs to learn in a particularly exhaustive way.

The proposed Unit 5 is a good example. I have to admit harking and p hacking are both the sort of terms  that as a non specialist who reads statistical stuff sometimes, I see all the time and have a vague sense that they both mean "messing with data in a way that makes conclusions look more meaningful than they are." And ok, I looked them up, and have a clearer sense now, but I don't actually need to know more than what I just read in the first two sentences of the wikipedia article.

The basic concept is just from intro statistics-any particular series of results could just be from chance-and if you're selecting variables after the fact based on them correlating, you can end up with things that look convincing but are actually meaningless, like "every president elected in years that are multiples of 20 from 1840-1960 onwards died in office, so there must be have been a curse which was broken when Reagan survived being shot in an assassination attempt." Outside of certain disciplines, nobody else needs to spend two weeks going through the ins and outs of this. Majors in those disciplines will benefit more from methods courses in their own discipline.

I'd be all for core courses that take an interdisciplinary approach to studying some sort of topic or issue that allows students to explore some of these ideas within a broader context.

[Parasaurolophus]

(Sorry for the double post.)

Sadly, most people who talk about Ockham's Razor get it seriously wrong.

But yes, I would have added a section on the scientific method. And, in particular, on different accounts of what that method actually is.

Frankly, I'd also have to add a unit on evolutionary explanations, because there's an awful lot of really bad appeals to them out there, including and especially!) by scientists who ought to know better.

But here's the thing: courses like this already exist in a lot of universities. But they're taught in the philosophy or HPS departments. No one course hits all the topics mentioned, but many come close.

The problem with this is that it gives the impression that these issues are somehow esoteric or arcane, rather than fundamental considerations that scientists and others thinking about science need to do consistently and deliberately.

Would a course on science fiction be better taught in some STEM department than in an English department since it's "science" fiction? Or would that understate the importance of the literary considerations that apply to all kinds of literature, and are automatically and explicitly examined by English scholars?

I mean... most philosophers of science have at least a bachelor's or master's degree in a science field. Many have a second PhD.

The real issue, to the extent it is one, is reputational, since nobody knows what philosophy is or what we talk about. And, to be sure, one way around that is to strip away courses from such a disreputable department, or duplicate their offerings until they can't compete anymore, Starbucks-style. Another would be to work with them and funnel students through the department for that pre-existing course or courses, and thus signal to students that it's perfectly respectable.

Look, all I'm saying is that this already exists in many universities, especially those with some research clout, and it's a perfectly respectable course taught by someone with real credentials. (It does take a trained philosopher of science to do well, and they're fairly ubiquitous these days, but you'll still find more of them at R1s, R2s, and fancy SLACs.)

Similarly, you'd probably usually want courses in the history of science to be taught by someone from the HPS department, if you had one, rather than any old schmuck from the science in question, because the sciences, for all their virtues, don't do a great job of teaching their history.

There are lots of different ways to set up a great course like this. I'm just observing that, in a lot of instances, a high-quality version already exists. There's nothing wrong with designing more or better wheels. I just don't think we should pretend we're inventing it from scratch.

[polly_mer]

There's p-hacking and then there's the scientific method of looking for patterns that prompt interesting questions for experimental design.

If you're teaching "the" scientific method, then use something like https://www.pinterest.com/pin/425519864773938497/ instead of Problem, Hypothesis, Experiment, Obervation, Conclusion.  That's one huge change in k-8 science education in the past decade.

[polly_mer]

There are lots of different ways to set up a great course like this. I'm just observing that, in a lot of instances, a high-quality version already exists. There's nothing wrong with designing more or better wheels. I just don't think we should pretend we're inventing it from scratch.

How many of the gen ed requirements in science are met by a philosophy of science course?  I know the one I took decades ago counted as a philosophy elective, despite being taught by a psychologist with relevant expertise.

If we scale up to requiring a philosophy of science course as the required science gen ed, how will those sections be staffed?  There aren't that many philosophers of science, unless we go to straight up enormous remote lectures with self-paced online work.  That's content delivery, but misses all the important translation from academic knowledge to daily use in reading the news and evaluating information for one's own life.

I agree we're not short on existing gen ed courses.  I'm still going to bang the drum for the fact that gen ed doesn't deliver on promises, unlike a true liberal arts education chosen by people who want to learn and invest the effort to do so.

[ergative]

This sounds like an outstanding idea. I second the recommendation to discuss HARKing, and also what Gelman calls the Garden of Forking Paths.

I wonder whether it might be useful to organize these topics not in terms of sub-disciplines (social sciences people use statistics and like to extrapolate data too) but in terms of something more structural about the process of quantitative reasoning. For example:

Unit 1: What can we learn from numbers
-causation vs. correlation
-probability (Type I vs. Type II error)
-randomness

Unit 2: How can we learn it?
-double blind and observational studies
-from individuals to populations
-polling, surveys

Unit 3: What can go wrong when operating in good faith?
-measurement uncertainty and error propagation
-cognitive biases and experimental design
-Data modeling and extrapolation; specifically how widely model extrapolations can diverge
-selection bias
-underpowered experiments and Type M error
-garden of forking paths

Unit 4: What can go wrong when operating in bad faith?
-p-hacking
-HARKing

Unit 5: The publication problem
-file-drawer effects and publication bias
-preregistration and open science

Seems like its quickly going in to areas that are of concern to scientists and are certainly important to teach to majors, but aren't things that everyone else needs to learn in a particularly exhaustive way.

The proposed Unit 5 is a good example. I have to admit harking and p hacking are both the sort of terms  that as a non specialist who reads statistical stuff sometimes, I see all the time and have a vague sense that they both mean "messing with data in a way that makes conclusions look more meaningful than they are." And ok, I looked them up, and have a clearer sense now, but I don't actually need to know more than what I just read in the first two sentences of the wikipedia article.

The basic concept is just from intro statistics-any particular series of results could just be from chance-and if you're selecting variables after the fact based on them correlating, you can end up with things that look convincing but are actually meaningless, like "every president elected in years that are multiples of 20 from 1840-1960 onwards died in office, so there must be have been a curse which was broken when Reagan survived being shot in an assassination attempt." Outside of certain disciplines, nobody else needs to spend two weeks going through the ins and outs of this. Majors in those disciplines will benefit more from methods courses in their own discipline.

I'd be all for core courses that take an interdisciplinary approach to studying some sort of topic or issue that allows students to explore some of these ideas within a broader context.

Huh--I actually thought Unit 5 would be the most important to non-specialists, since it's Unit 5 that tells you about how to be cautious of what is actually published and hence picked up by reporters and disseminated publicly. It's only through the unit 5 filter than any scientific results become known and available for discussion for non-specialists.

I'm reminded of a bit of congressional testimony by Fauci, in which some congressman or other was talking about hydroxychloroquine, and Fauci said that it was bunkum, and the congressperson said 'but it was a peer-reviewed study!' and Fauci said that it didn't matter whether or not it was peer-reviewed, because it wasn't a double-blind controlled study. That is exactly the kind of exchange that illustrates the need for this kind of class: Unit 2 would say why you shouldn't trust that kind of study design, and Unit 5 would explain how that study still got published.

(Although, on re-reading your comment, perhaps you meant Unit 4 might be too esoteric for generalists? That's certainly possible.)

[Parasaurolophus]

How many of the gen ed requirements in science are met by a philosophy of science course?  I know the one I took decades ago counted as a philosophy elective, despite being taught by a psychologist with relevant expertise.

I don't know. Obviously, it would depend on the institution. I'm also not sure that gen ed electives are the way to saturate the university with that kind of programming.

What I'm observing is that for many of us, this course already exists (in a perfectly respectable, competent fashion!), even if we don't know about it. In those cases, it seems silly to insist on duplicating it simply because we're not aware of it, rather than, say, actively promoting it, building it into degree requirements, or cross-listing it.

If we scale up to requiring a philosophy of science course as the required science gen ed, how will those sections be staffed?  There aren't that many philosophers of science, unless we go to straight up enormous remote lectures with self-paced online work.  That's content delivery, but misses all the important translation from academic knowledge to daily use in reading the news and evaluating information for one's own life.

There are probably more of them than you think: their professional association has around 700 members, and dozens of new scholars graduate in the subject every year (it's quite popular at the grad level these days, even despite the credentials arms race).

Obviously, that's still not enough for every university to get in on the action. But I'm not making that kind of prescriptive claim!

[polly_mer]

There are probably more of them than you think: their professional association has around 700 members, and dozens of new scholars graduate in the subject every year (it's quite popular at the grad level these days, even despite the credentials arms race).

Obviously, that's still not enough for every university to get in on the action. But I'm not making that kind of prescriptive claim!

700 is fewer than I expected based on annual PhD conferral rates for other small fields.  Dozens of new scholars every year is very, very small.

I also have to wonder how many of those folks at the universities where they teach 2/2 or lighter, mostly in major, with substantial research expected would really want to be carrying the necessary service load.

One section of 20 enthusiastic students is a fabulous experience.  I greatly enjoyed the undergrad philosophy of science course I took as a grad student who begged my advisor.

Force marching people through the gen ed requirements is a special skill set.  How many of the only 700 people who are still working in academia (I tend to encounter these folks outside of academia and never at places like Super Dinky) really will step up to take on this service in the way that would be necessary?

[marshwiggle]

Seems like its quickly going in to areas that are of concern to scientists and are certainly important to teach to majors, but aren't things that everyone else needs to learn in a particularly exhaustive way.

The proposed Unit 5 is a good example. I have to admit harking and p hacking are both the sort of terms  that as a non specialist who reads statistical stuff sometimes, I see all the time and have a vague sense that they both mean "messing with data in a way that makes conclusions look more meaningful than they are." And ok, I looked them up, and have a clearer sense now, but I don't actually need to know more than what I just read in the first two sentences of the wikipedia article.

The basic concept is just from intro statistics-any particular series of results could just be from chance-and if you're selecting variables after the fact based on them correlating, you can end up with things that look convincing but are actually meaningless, like "every president elected in years that are multiples of 20 from 1840-1960 onwards died in office, so there must be have been a curse which was broken when Reagan survived being shot in an assassination attempt." Outside of certain disciplines, nobody else needs to spend two weeks going through the ins and outs of this. Majors in those disciplines will benefit more from methods courses in their own discipline.

I'd be all for core courses that take an interdisciplinary approach to studying some sort of topic or issue that allows students to explore some of these ideas within a broader context.

Huh--I actually thought Unit 5 would be the most important to non-specialists, since it's Unit 5 that tells you about how to be cautious of what is actually published and hence picked up by reporters and disseminated publicly. It's only through the unit 5 filter than any scientific results become known and available for discussion for non-specialists.

I'm reminded of a bit of congressional testimony by Fauci, in which some congressman or other was talking about hydroxychloroquine, and Fauci said that it was bunkum, and the congressperson said 'but it was a peer-reviewed study!' and Fauci said that it didn't matter whether or not it was peer-reviewed, because it wasn't a double-blind controlled study. That is exactly the kind of exchange that illustrates the need for this kind of class: Unit 2 would say why you shouldn't trust that kind of study design, and Unit 5 would explain how that study still got published.

Yes! That was exactly my reason for such a course. People are exposed to all kinds of "scientific" results in the media, (not to mention all kinds of clickbait), and most people (including the journalists talking about them) have any idea what kind of automatic skepticism should be applied.

I remember some years ago a journalist noting that organizations will sometimes choose someone to cover "science" who has no background in science so that they can *communicate to the general public. They would never dream of doing such a thing in other areas, such as sports. (What good would a sports writer be who had no interest in or knowledge of sports????)

*Because anyone who knows any science will be, you know, too geeky to relate to real human beings.

[jimbogumbo]

Great list and suggestions! I such a course would work really well as a (don't hurt me!) MOOC. I'm serious, advertised to older people who especially lack this info. When I taught a class that also aired on local cable I was stunned at how many people of various ages and classes talked about sessions that had watched.

I would make sure in one of the sub-units on data and conclusion that there is a discussion of Simpson's paradox.

[jimbogumbo]

Sorry for the double post, but Kiana's comment about Math Appreciation on the get ed thread reminded me of Eric Gaze and this source, which I regularly use in our Quantitative Reasoning course: https://thinkingquantitatively.wordpress.com

[polly_mer]

Great list and suggestions! I such a course would work really well as a (don't hurt me!) MOOC. I'm serious, advertised to older people who especially lack this info. When I taught a class that also aired on local cable I was stunned at how many people of various ages and classes talked about sessions that had watched.

A MOOC is probably better than a gen ed requirement if the goal is better scientific literacy among adults.

[Wahoo Redux]

I had in mind a course which introduced non-scientists to a variety of hands-on basic science, say a 3-week unit on astronomy, a 3-week unit on chemistry, a 3-week unit on geology, a 3-week unit on biology (or something to this effect) or statistics etc. and how each of these uses the scientific method.

One of my best experiences as an undergrad was a fieldtrip to look at rock formations.  It was great to see the knowledge in action.

[marshwiggle]

I had in mind a course which introduced non-scientists to a variety of hands-on basic science, say a 3-week unit on astronomy, a 3-week unit on chemistry, a 3-week unit on geology, a 3-week unit on biology (or something to this effect) or statistics etc. and how each of these uses the scientific method.

One of my best experiences as an undergrad was a fieldtrip to look at rock formations.  It was great to see the knowledge in action.

Part of the danger in this sort of approach is the implication that the "scientific method" is kind of a machine that takes in observations and spits out valid laws. (Not neccessarily what you were implying, but how it is sometimes presented.) What I'm after is a course highlighting how complex the process is, and how many ways it can go wrong, even by people who are well-meaning. There are so many "scientific" claims people are exposed to that they need to understand how those claims could be misleading or outright wrong.

[spork]

I had in mind a course which introduced non-scientists to a variety of hands-on basic science, say a 3-week unit on astronomy, a 3-week unit on chemistry, a 3-week unit on geology, a 3-week unit on biology (or something to this effect) or statistics etc. and how each of these uses the scientific method.

One of my best experiences as an undergrad was a fieldtrip to look at rock formations.  It was great to see the knowledge in action.

This is even less than what is supposed to be happening in elementary school. Why do college students need it?

The scientific method as presented by Feynman in the video and by the Pinterest diagram have an implicit assumption that the individual is curious enough to make an observation and wonder why the phenomenon they have observed happens. Most of the students where I work instead want to be told what to memorize so that they can get one step closer to a bachelor's degree credential. Asking the question "Why does X happen?" is not why they are in college.

It looks to me that Feynman is at Cornell in the video, so probably this presentation was filmed in the late 1940s. I find it interesting that the audience is both interested in the subject of his lecture and, given their laughter, capable of understanding it.

[polly_mer]

I had in mind a course which introduced non-scientists to a variety of hands-on basic science, say a 3-week unit on astronomy, a 3-week unit on chemistry, a 3-week unit on geology, a 3-week unit on biology (or something to this effect) or statistics etc. and how each of these uses the scientific method.

One of my best experiences as an undergrad was a fieldtrip to look at rock formations.  It was great to see the knowledge in action.

This is even less than what is supposed to be happening in elementary school. Why do college students need it?

You beat me to it, Spork.  K-12 students should have had multiple months multiple times with all these areas with increasing sophistication based on grade level.

Perhaps people need to know the philosophy behind the Next Gen Science Standards for k-12: https://ngss.nsta.org/PracticesFull.aspx

or see some of the standards at grade level: https://ngss.nsta.org/AccessStandardsByTopic.aspx

Second grade is processes that shape the Earth, which is geology rock formations, and then again in fourth grade, middle school, and high school.  Blocky's school went on geology field trips multiple times.

The fix is still at k-12 based on decades of research on science education.