Scientific literacy course for non-scientists

[marshwiggle]

I brought this up in another thread, so I'd be interested in hearing what other people think.

Rather than forcing non-science majors to take "Intro <whatever>" from some discipline, I think it would be far more useful to have citizens who understand how science works. To that end, here are some topic suggestions. (We have a 12 week term here, so about a week a topic seems to make sense.)
In no  particular order:

• Correlation vs. causation (statistics)
• Introduction to probability and statistics; in particular, Type I and Type II error (statistics)
• Uncertainty in measurement and error propagation (physical science)
• Data modeling and extrapolation; specifically how widely model extrapolations can diverge (physical science)
• Cognitive biases and experimental design (social sciences)
• Double blind and observational studies (medical sciences)
• Polling, surveys, selection bias, etc. (social sciences)
• Replication, peer review, and publication bias (how reliable are published results?)
• Populations versus individuals (statistics)
• Factor analysis, reliability and validity (statistics)
• Randomness, and how hard it is to achieve

In many cases I've identified areas where it typically occurs.
Additions? Deletions? Replacements?

[dismalist]

That's very fine for the correlation, Marsh, but there's not nearly enough for the causation. I would aspire to 50/50.

Here's Feynman on Method, very entertaining, too:

https://www.youtube.com/watch?v=EYPapE-3FRw

[marshwiggle]

That's very fine for the correlation, Marsh, but there's not nearly enough for the causation. I would aspire to 50/50.

Here's Feynman on Method, very entertaining, too:

https://www.youtube.com/watch?v=EYPapE-3FRw

Wow! That was amazing.

A topic to add to my list, inspired by the video:

• Occam's razor

[polly_mer]

Fixing k-12 education is better than yet another one-off in college.  12 years of projects, case studies, and other practical, every day  applications for people who are not professional scientists should be plenty.

Remediating at the college level is expensive and ineffective.  Stop fixing the wrong problem!

[Hibush]

While it is redundant to some of the proposed sections, I'd devote a whole section to HARKing. So many people collect a bunch of data that seem related to the problem and hope it self-assembles into an answer. More pernicious is when a designed experiment shows no main effect, but some unanticipated variation shows up. Then they assemble a hypothesis post hoc and try to apply ex ante statistics. Common scientific misconduct that many scientists in training don't realize is a problem.

[Puget]

While it is redundant to some of the proposed sections, I'd devote a whole section to HARKing. So many people collect a bunch of data that seem related to the problem and hope it self-assembles into an answer. More pernicious is when a designed experiment shows no main effect, but some unanticipated variation shows up. Then they assemble a hypothesis post hoc and try to apply ex ante statistics. Common scientific misconduct that many scientists in training don't realize is a problem.

I agree this is important to teach, but the bolded part depends on the field. In psychology we now cover this pretty early and often (certainly at the beginning grad methods level, but filtering down to undergrad research methods courses now), and preregistration is becoming the norm to prevent it.

[Parasaurolophus]

That's very fine for the correlation, Marsh, but there's not nearly enough for the causation. I would aspire to 50/50.

Here's Feynman on Method, very entertaining, too:

https://www.youtube.com/watch?v=EYPapE-3FRw

Wow! That was amazing.

A topic to add to my list, inspired by the video:

• Occam's razor

Sadly, most people who talk about Ockham's Razor get it seriously wrong.

But yes, I would have added a section on the scientific method. And, in particular, on different accounts of what that method actually is.

Frankly, I'd also have to add a unit on evolutionary explanations, because there's an awful lot of really bad appeals to them out there, including and especially!) by scientists who ought to know better.

But here's the thing: courses like this already exist in a lot of universities. But they're taught in the philosophy or HPS departments. No one course hits all the topics mentioned, but many come close.

[ergative]

This sounds like an outstanding idea. I second the recommendation to discuss HARKing, and also what Gelman calls the Garden of Forking Paths.

I wonder whether it might be useful to organize these topics not in terms of sub-disciplines (social sciences people use statistics and like to extrapolate data too) but in terms of something more structural about the process of quantitative reasoning. For example:

Unit 1: What can we learn from numbers
-causation vs. correlation
-probability (Type I vs. Type II error)
-randomness

Unit 2: How can we learn it?
-double blind and observational studies
-from individuals to populations
-polling, surveys

Unit 3: What can go wrong when operating in good faith?
-measurement uncertainty and error propagation
-cognitive biases and experimental design
-Data modeling and extrapolation; specifically how widely model extrapolations can diverge
-selection bias
-underpowered experiments and Type M error
-garden of forking paths

Unit 4: What can go wrong when operating in bad faith?
-p-hacking
-HARKing

Unit 5: The publication problem
-file-drawer effects and publication bias
-preregistration and open science

[marshwiggle]

This sounds like an outstanding idea. I second the recommendation to discuss HARKing, and also what Gelman calls the Garden of Forking Paths.

I wonder whether it might be useful to organize these topics not in terms of sub-disciplines (social sciences people use statistics and like to extrapolate data too) but in terms of something more structural about the process of quantitative reasoning.

I like your sub-units. My only point in identifying sub-disciplines is to illustrate how even people with a science education will have seen some of these much more than others. To address Polly's point about k-12 improvements, that's great as well, but I think an overview bringing all of these things together has value beyond seeing individual concepts in isolation. (Even after graduate school in STEM, many of these I've only come across in my own interests as an adult.)

Adding HARKing is a great idea, as is evolutionary explanations; I realized a couple I missed were confidence intervals, and ethical considerations, which would be part of observational studies for obvious reasons.

I'd love to work in some behavioural economics, but there probably wouldn't be room because it would sort of need its own course.

[marshwiggle]

(Sorry for the double post.)

Sadly, most people who talk about Ockham's Razor get it seriously wrong.

But yes, I would have added a section on the scientific method. And, in particular, on different accounts of what that method actually is.

Frankly, I'd also have to add a unit on evolutionary explanations, because there's an awful lot of really bad appeals to them out there, including and especially!) by scientists who ought to know better.

But here's the thing: courses like this already exist in a lot of universities. But they're taught in the philosophy or HPS departments. No one course hits all the topics mentioned, but many come close.

The problem with this is that it gives the impression that these issues are somehow esoteric or arcane, rather than fundamental considerations that scientists and others thinking about science need to do consistently and deliberately.

Would a course on science fiction be better taught in some STEM department than in an English department since it's "science" fiction? Or would that understate the importance of the literary considerations that apply to all kinds of literature, and are automatically and explicitly examined by English scholars?

[polly_mer]

Would a course on science fiction be better taught in some STEM department than in an English department since it's "science" fiction? Or would that understate the importance of the literary considerations that apply to all kinds of literature, and are automatically and explicitly examined by English scholars?

Ideally, this would be team taught as a two-course sequence first as how science and technology affect human society using literary analysis and then how expectations from literature affect science and technology developments by humans.

[Ruralguy]

Close to how I have done it the times I have taught this course, though not exactly.

[Puget]

This sounds like an outstanding idea. I second the recommendation to discuss HARKing, and also what Gelman calls the Garden of Forking Paths.

I wonder whether it might be useful to organize these topics not in terms of sub-disciplines (social sciences people use statistics and like to extrapolate data too) but in terms of something more structural about the process of quantitative reasoning. For example:

Unit 1: What can we learn from numbers
-causation vs. correlation
-probability (Type I vs. Type II error)
-randomness

Unit 2: How can we learn it?
-double blind and observational studies
-from individuals to populations
-polling, surveys

Unit 3: What can go wrong when operating in good faith?
-measurement uncertainty and error propagation
-cognitive biases and experimental design
-Data modeling and extrapolation; specifically how widely model extrapolations can diverge
-selection bias
-underpowered experiments and Type M error
-garden of forking paths

Unit 4: What can go wrong when operating in bad faith?
-p-hacking
-HARKing

Unit 5: The publication problem
-file-drawer effects and publication bias
-preregistration and open science

This is a great syllabus! I'd be in favor of adding a course like this to our stats and research methods sequence for majors. I'm not sure it would work for the originally stated purpose of being a course for non-scientists, unless you kept the stats pieces at a very conceptual level.

One thing I'd amend slightly is that I'm not sure there's a clear distinction between good faith and bad faith -- that is, although some people who p-hack and HARK are operating in bad faith, I think a lot honestly trick themselves into believing they aren't doing anything wrong--

When I teach about HARKing and the importance of preregistration, I talk about how the human brain is a story-telling machine-- we find patterns where maybe there aren't any and create persuasive stories about why those patterns make sense, and once something makes sense we sometimes come to believe that we always expected that result. I call this the "Golem effect" (It was my birthday, so it aught to have been my birthday present, so it was my birthday present, etc.). So preregistration protects ourselves from ourselves, no bad faith assumed.

I think this framing is important because no one wants to think of themselves as possibly acting in bad faith, so to make these practices universal you need to emphasize that they are good for everyone, not just those tempted to commit data fraud. In fact, these practices are unlikely to prevent actual, intentional data fraud, because someone who would do that is also perfectly capable of post-registering a study and lying about it. Other practices, like requiring open data, may be more helpful but even then careful outright fabrication would probably go undetected. Luckily these cases are probably fairly rare.

[marshwiggle]

This is a great syllabus! I'd be in favor of adding a course like this to our stats and research methods sequence for majors. I'm not sure it would work for the originally stated purpose of being a course for non-scientists, unless you kept the stats pieces at a very conceptual level.

I think all of these concepts could be introduced in  a way that was worthwhile without being overly technical. I've seen websites that have clever illustrations of statistical concepts that don't involve any math but work for a general audience.

For example:
https://4.bp.blogspot.com/-wmZzvsY_Tec/Vws0f4MJn9I/AAAAAAAAORs/gipKxA7aDboP0gx2vSmyQS_ZoVBPzqaWA/s1600/Type%2BI%2Band%2BII%2Berror.jpg

One thing I'd amend slightly is that I'm not sure there's a clear distinction between good faith and bad faith -- that is, although some people who p-hack and HARK are operating in bad faith, I think a lot honestly trick themselves into believing they aren't doing anything wrong--

The "Garden of Forking Paths" article referenced above actually does a great job of pointing this out; it goes through examples of how this can happen without any bad faith.

[Puget]

The "Garden of Forking Paths" article referenced above actually does a great job of pointing this out; it goes through examples of how this can happen without any bad faith.

Yes, that's a classic, and emphasizes why it is so important to preregister a detailed data analysis plan rather than just hypotheses. I make students write out their data analysis plan as pseudo-code (specifying the exact analysis with all independent and dependent variables and all covariates), as well as specifying the criteria for outliers, and what will be done if data are non-normal. When the data themselves might force an alternative plan (e.g., a confirmatory factor analysis shows poor model fit, precluding further analyses with that model), we specify what the contingency plan is in that case. We talk a lot about eliminating researcher degrees of freedom.