How Not to Design an Experiment: CHE Article Fail

[polly_mer]

  A history professor suggests that the rapid, middle-of-the-term conversion to all online instruction at many institutions is a "natural experiment" that will tell us how students learn online.

Yes, the author has some reasonable caveats, but asserts that good scientists can just solve those problems.  A tweet I saw from a professor likened the current situation to giving the swimming test in the middle of a flood to everyone who ended up in the rushing waters, regardless of previous experience swimming or even preparation before ending up in the river.

It's like some people don't know how scientific experiments work and are pretty iffy on what kind of transferable conclusions can be made from convenience observations of possibly related situations.

[Cheerful]

Author says:  That brings us back to the coronavirus, which has created a set of unprecedented natural experiments. For the first time, entire student bodies have been compelled to take all of their classes online. So we can examine how they perform in these courses compared to the face-to-face kind, without worrying about the bias of self-selection.

On what planet is this professor living?  He has no concept of bias or crisis or life/death or trauma or.....

[mahagonny]

Author says:  That brings us back to the coronavirus, which has created a set of unprecedented natural experiments. For the first time, entire student bodies have been compelled to take all of their classes online. So we can examine how they perform in these courses compared to the face-to-face kind, without worrying about the bias of self-selection.

On what planet is this professor living?  He has no concept of bias or crisis or life/death or trauma or.....

I took him to mean the self selection factor, that is choosing to take a course online rather than live and in person, is gone, since there is no choice at this time. Controlled experiment or not, we will be observing and forming our theories.
I don't know how much slippage there will be but in at least one course that I teach the online experience would be getting compared to a lab setting that was already sadly neglected and compromised in terms of physical facility, unregulated eligibility to enroll and class size. So the lesson there might be if you want to kill something off through neglect, best to do it in increments.

[marshwiggle]

A tweet I saw from a professor likened the current situation to giving the swimming test in the middle of a flood to everyone who ended up in the rushing waters, regardless of previous experience swimming or even preparation before ending up in the river.

This illustrates the strange idea that some have expressed, even here of having to "do a bad job" of adapting to online to avoid encouraging the move to online as normal after this is past. Consider the following:

• If you had to change textbooks partway through a term, how would that be?
• If an instructor was ill, and had to be replaced partway through the term, how would that be?
• If some sort of flood or problem made your normal lab unavailable partway through the term, how would that be?

In other words, is there any kind of disruptive event that could happen during a term that would be so easy to adapt to as to possibly make the course better?

Even more so, how many instructors are so amazing at teaching to begin with that such a result is possible? All of my success has come from incremental improvements over time. Any big change I've made has been rough at first, and required a lot of tweaking to make it work well.

TL;DR You don\t have to try to adapt badly; if you manage to adapt remotely reasonably you're doing about as well as can be expected.

[mahagonny]

I can't see any reason to rewrite the course syllabus for the last half of the term, as some are apparently doing. For one thing, there's the issue of changing the rules in the middle of the baseball game. For another, the syllabus I am required to use looks nothing like any syllabus I ever wrote or could have written. Too silly to get involved with.

[Caracal]

A tweet I saw from a professor likened the current situation to giving the swimming test in the middle of a flood to everyone who ended up in the rushing waters, regardless of previous experience swimming or even preparation before ending up in the river.

This illustrates the strange idea that some have expressed, even here of having to "do a bad job" of adapting to online to avoid encouraging the move to online as normal after this is past. Consider the following:

• If you had to change textbooks partway through a term, how would that be?
• If an instructor was ill, and had to be replaced partway through the term, how would that be?
• If some sort of flood or problem made your normal lab unavailable partway through the term, how would that be?

In other words, is there any kind of disruptive event that could happen during a term that would be so easy to adapt to as to possibly make the course better?

Even more so, how many instructors are so amazing at teaching to begin with that such a result is possible? All of my success has come from incremental improvements over time. Any big change I've made has been rough at first, and required a lot of tweaking to make it work well.

TL;DR You don\t have to try to adapt badly; if you manage to adapt remotely reasonably you're doing about as well as can be expected.

Yeah, I agree. I think the caveat is that you also aren't required to drive yourself into the ground trying to make it perfect.