Question on NSF CAREER comment

[Vid]

All,

My NSF CAREER is declined with multiple rating VG-G, VG-G, and VG-G. Here is the rational for recommendation.

Rationale for Recommendation:
-The proposal is overall well organized and easy to follow. The proposal would be stronger if it were more focused and better integrated. The education plan is plausible, but needs to be assessed. The PI is encouraged to improve the proposal and resubmit.

My question is do the panel members recommend revising and resubmitting? It was my first submission.


Thank you good people!

[fizzycist]

You should always revise and resubmit until you are no longer eligible. With 3 VG/G on first try it is probably unnecessary to switch directorates or overall topic.

"More focused and better integrated" sounds like a stock critique that reviewers use when, for whatever reason, they just weren't *that* excited about your proposal compared to the others they read. You should not take it personally and the outcome can be different with the same proposal but different reviewers (or even the same reviewers who are in a better mood that day or whatever). Nevertheless, it can't hurt to include in your revision a short section highlighting how your research and education plans are well integrated (which for some reason is an important review criterion for CAREER).

I take "Education plan needs to be assessed" to mean the panel did not see how you would assess whether your educational goals were met? If so, it sounds like a more specific critique that you should probably address in the revision.

But that was also unlikely to be score-driving either, so you may want to try and read between the lines of the individual panelist reviews to see what they really thought. Or not--I've personally found that obsessing over reviews is not that helpful for preparing a revised proposal. What I've found is more helpful is to publish on the topic in the interim and to keep improving how you communicate the main ideas to a broad audience.

[Vid]

Thank you for your suggestions. Much appreciated.

Would you think my proposal will  be reviewed by the same reviewers next year? The reviewers seemed to be excited about the research. They liked the proposed methodologies, and BI. Anyway, this was my first submission.

I am going to contact the PO and meet with him over zoom!

Thank you again for your suggestion.

Vi

[aspiring.academic]

Very good tips.

I'd add to meet with your PD. She or he was in the panel session and would be better positioned to help you read through the lines. Overall for the first attempt, to receive not a single Fair, you did very well and should be commended.

[polly_mer]

The proposal would be stronger if it were more focused and better integrated.

What exactly are you going to do?

Why do all the parts sum up to be a complete project instead of being a bunch of disconnected pieces that are individually interesting, but are not focused on one specific question?

Compare:

A: I'm going to investigate carrots for the first time period, pie for the next two time periods, sauerkraut for the following time period, and tablecloths for the remaining time.  I'm going to use the X technique while studying carrots, the Y technique while studying pie, the Z technique for studying sauerkraut, and hope I figure out something along the way for the tablecloths.

B: I have a theory related to how tablecloths at picnics affect the consumption of non-main-dishes (e.g., vegetables, condiments, and desserts).  My previous work on salads and cakes indicates carrots are prototypical vegetables, sauerkraut is a prototypical condiment, and pie is a prototypical dessert. The order of investigation is carrots, sauerkraut, and pie to build up the consolidated technique necessary to study the tablecloths in conjunction with the consumables.

The education plan is plausible, but needs to be assessed.

When I do a quick internet search, I turn up:

https://www.research.colostate.edu/rao/wp-content/uploads/sites/12/2020/02/2020-NSF-CAREER-AWARD-GUIDANCE-FOR-FACULTY.pdf

A competitive proposal will include plans for assessing or evaluating your educational activities, tools, or materials. You are encouraged to make connections with appropriate education experts, and to include the necessary letters of commitment in your application. NSF recommends that applicants leverage existing NSF-supported activities or other educational projects ongoing on campus.

https://research.usu.edu/rd/wp-content/uploads/sites/17/2015/10/Developing-Education-Components_NSF-CAREER.pdf

Q: How should a PI go about developing an assessment plan?

A: First, think in terms of your goals for your educational component, and describe to yourself evidence that would help you to determine if they were met. For example, if you want to improve student learning in a course, how do you recognize when they have mastered the material? You would say that they need to "understand" the material. That's a good start, but understanding is an internal mental state and we cannot observe understanding yet. So, ask yourself what you might observe to convince yourself that a student has understood the material. What things would you look at?

There has been a lot of work done on developing instruments to assess things such as critical thinking and cognition, and there are efforts funded by NSF and the Lilly Endowment to collect assessment information and tools and to make them generally more available. For example, the Wabash National Study focuses on assessing Liberal Arts Education, but much of this information can also be applied to STEM. You can find information on their outcomes and assessment instruments for different outcomes on their website [http://www.wabashnationalstudy.org/wns/instruments.html].

In cases in which you want to compare influences of your intervention, one place to start is where you have multiple sections of an undergraduate course. Then, you might collect data with assessment instruments that would allow you to compare how students perform in one or more sections that experienced the educational innovation, with students from other sections that did not experience the educational innovation. This is pretty easy if the sections already have common exams. If they don't, you might collaborate with a faculty member teaching another section to give common exams. Another option is to compare student performance in a version of a course with student performances in previous years.

Often, you may need to involve others with expertise in assessment who can advise you on the use of a particular assessment instrument. In most universities, you'll find faculty members with that expertise in the College of Education and Human Development. Exactly who you recruit will depend on what you're trying to assess. For example, if you want to look at critical thinking, you would reach out to one group of people; if you need a survey instrument, you would reach out to people with expertise in surveys. If the assessment requires more qualitative research, you'll want to reach out to folks who are experts in qualitative research methods.

[lee2002hu]

You should talk to your PO and figure out what is between the lines of these comments.
There are many things not reflected through the panel summary during the review.

All,

My NSF CAREER is declined with multiple rating VG-G, VG-G, and VG-G. Here is the rational for recommendation.

Rationale for Recommendation:
-The proposal is overall well organized and easy to follow. The proposal would be stronger if it were more focused and better integrated. The education plan is plausible, but needs to be assessed. The PI is encouraged to improve the proposal and resubmit.

My question is do the panel members recommend revising and resubmitting? It was my first submission.


Thank you good people!

[Vid]

All; thank you very much.

Polly_mer; those links to further CAREER proposal writing info are great resources. Thank you for sharing.

I am going to email the PD tomorrow.

You guys are the best colleagues EVER!

--Vi

You should talk to your PO and figure out what is between the lines of these comments.
There are many things not reflected through the panel summary during the review.

All,

My NSF CAREER is declined with multiple rating VG-G, VG-G, and VG-G. Here is the rational for recommendation.

Rationale for Recommendation:
-The proposal is overall well organized and easy to follow. The proposal would be stronger if it were more focused and better integrated. The education plan is plausible, but needs to be assessed. The PI is encouraged to improve the proposal and resubmit.

My question is do the panel members recommend revising and resubmitting? It was my first submission.


Thank you good people!

[mleok]

It could very well be that there's nothing fundamentally wrong with your proposal, but that other proposals were more compelling. Since only a small fraction of CAREER proposals are funded, and the rest are declined, the panel summary is usually tweaked so that it is consistent with the recommendation. A good suggestion is to always think of a CAREER proposal as the first 5 years of a 10 year plan, the goal is to set you up for bigger things.

[pgher]

Thank you for your suggestions. Much appreciated.

Would you think my proposal will  be reviewed by the same reviewers next year? The reviewers seemed to be excited about the research. They liked the proposed methodologies, and BI. Anyway, this was my first submission.

I am going to contact the PO and meet with him over zoom!

Thank you again for your suggestion.

Vi

I'll just caution you here that the answer is "probably not." For a journal article, a re-submission is evaluated by the same reviewers, so you can just satisfy their desires. NSF operates differently. Sometimes you'll get some of the same reviewers; more often, the panel has substantial turnover. I have never served as a panelist for the same program twice in a row.

In addition, sometimes panelists, especially for CAREER, are trying to be nice. They don't want you to be disheartened, so they'll say kind things. In the end, none of them rated it excellent, so you need to do significant work to improve it. Not to remedy the specific deficiencies they identified, but to make it stronger in every way.

A colleague and I chased a particular NSF program for about five years before we broke through. One year, they would tell us our topic was too broad. So we would rework the idea to focus on a particular application. The next year, they would tell us our topic was too narrow and should be generalized. We swung back and forth for a while before finding the pitch that hit the mark.

[Vid]

Thank you very much. I am going to do zoom meeting with the PD and ask some details (reading between lines) about the comments as well as will ask whether the panel members were enthusiastic about my proposal or not.

What other questions would you suggest to ask the PD? 

P.S. I am a reviewer for this PD' program (not for CAREER) and he knows my work well. I did share my recent publications with him before submitting my CAREER proposal (he thanked me for acknowledging NSF bc I have an ongoing grant with his program). I develop research software in my field and he likes it.

Thank you for your suggestion.

-Vi

[Beebee]

NSF panels are different each year, so checking every box is not the way to a better score necessarily. It is well-informed feedback, and should broadly drive you towards a better proposal. I doubt these were the only feedback you received - I assume some of the concerns about focus and integration, for instance, is detailed in other comments. Talking with the program manager is also really good advice. "More focus" can imply your scope is too broad, and difficult to achieve within the time frame and with the resources you are asking for. It may also imply you are not justifying your choice of path (e.g. parameters you are seeking to test) clearly. "Better integration" may mean your tasks appear disparate, not part of a coherent theme. Or it may refer to the integration of educational and research activities, which is a criterion for CAREER awards. One approach for this is to involve your group in outreach events, incorporate some of it into your teaching, etc. Finally, "assessment" is an explicit criterion for the outreach activities. Your school may have an office that can help you with that (e.g. by helping you prepare before-and-after surveys, etc.). Good luck!

[mleok]

NSF panels are different each year, so checking every box is not the way to a better score necessarily. It is well-informed feedback, and should broadly drive you towards a better proposal. I doubt these were the only feedback you received - I assume some of the concerns about focus and integration, for instance, is detailed in other comments. Talking with the program manager is also really good advice. "More focus" can imply your scope is too broad, and difficult to achieve within the time frame and with the resources you are asking for. It may also imply you are not justifying your choice of path (e.g. parameters you are seeking to test) clearly. "Better integration" may mean your tasks appear disparate, not part of a coherent theme. Or it may refer to the integration of educational and research activities, which is a criterion for CAREER awards. One approach for this is to involve your group in outreach events, incorporate some of it into your teaching, etc. Finally, "assessment" is an explicit criterion for the outreach activities. Your school may have an office that can help you with that (e.g. by helping you prepare before-and-after surveys, etc.). Good luck!

This is good advice.