ableistein
ACUF Moderator
Registered: 04/19/07
Posts: 50
| |
dulongt
Registered: 01/10/08
Posts: 10
|
|
|
Reply with quote | #2 |
Regarding:
3. Vision and Strategic Goals
Line# 556, "[Consider using something from a service assessment here]"
I could not find anything regarding the use of ensembles from NWS service assessments. However, I did find a good supporting statement regarding the use of the SREF by Storm Prediction Center forecasts for making fire weather forecasts.
Suggest adding "SPC forecasters have begun using output
from the National Centers for Environmental Prediction
(NCEP) Short-Range Ensemble Forecast (SREF) system
as guidance for the national Fire Weather Outlooks. The
use of the SREF enhances the forecast process by
quantifying the likelihood that key fire weather
parameters will reach or exceed critical thresholds."
Citation found on World Wide Web at
http://www.spc.noaa.gov/publications/taylor/AMS_SREF.pdf
__________________
/s/Thomas W. Dulong/s/ |
| |
Pheppner3si
Registered: 09/17/07
Posts: 2
|
|
|
Reply with quote | #3 |
My first impression was that this document was great read for meteorologists, but would not be palatable to some we are trying to reach (i.e., educators). I am not suggesting that we 'dumb down' content, but I think we need to consider writing some additional verbage that is geared toward a more general audience, who inevitably will see this work.
A couple of examples:
The Executive Summary should state at the very top what is "Forecast Uncertainty" -- which sets the stage for the rest of summary. Instead, we jump right into "This plan defines ..." We don't need to junk that, but maybe a preceding motherhood sentence that says "Although meteorological accuracy has greatly improved during the evolution of the science, weather forecasts in many ways remain imperfect and contain elements of uncertainty."
I think we need some words at the top that set things up.
Ditto that remark for the next section "Purpose and Introduction". It would be FAR BETTER to take the 2nd and 3rd paragraphs and put them right up top. These paragraphs do a great job of saying where we've come from. I'd take the first paragraph and drop it into position #3. Once the introduction is made, then it is great to follow it in a third paragraph by "The purpose of this plan is to define". |
| |
BMorrow
Registered: 09/27/07
Posts: 3
|
|
|
Reply with quote | #4 |
I have read the document and as a lay person find it well written and relatively easy reading. I understand most of the examples. I think the social science parts are well-covered. The emphasis on public education is a key factor in my opinion. I do hope the document does not get lengthened significantly. In order to be clear and concise I think there is a tendency for those with technical expertise to "over-explain."
__________________
Dr. Betty Hearn Morrow
SocResearch Miami
|
| |
dchilderbrand
Registered: 03/11/08
Posts: 14
|
|
|
Reply with quote | #5 |
I have one caveat before I list my comments below...I read the 7/9/09 version and due to time constraints will focus on that version for now. Some of these comments are specific enough that they will be found in other parts of this forum.
(1) not sure where to locate or how to present (the first reference is in the Preface)...but we need to explicitly define "reliable" in the context of probabilistic forecasts. Most readers will interpret reliable to mean always being there...as it is used most commonly in operations. Maybe we have a box...maybe just a footnote...but regardless...more clarification is needed.
(2) Box 1...from my rough calculations...the % of ensemble members that indicate a potential strong storm looks more like 30% instead of 50% that is given in the text. Maybe someone like Tom Hamill can determine the exact percentage.
(3) I see in the newest version that Scenario for Enhancing Public Well Being was removed. I think a strong example would really help communicate that uncertainty can really affect your daily life...examples might include (a) outdoor wedding with or without a tent which can be determined based on probability of rain or cool temps. (b) scheduling travel around potential problems...could be deciding to take the train instead of driving...or could simply be pushing departure time up a few hours to avoid possible bad road conditions.
(4) bigger picture...social science...to my knowledge isn't mentioned until page 19 of the 7/9/09 version...under the academic sector roles and responsibilities. I think it needs to mentioned earlier on as a critical component of this effort...as uncertainty is inherently challenging to comprehend and can easily be misinterpreted. Further...I think it needs a big role in strategic goal #3.
|
| |
dnovak
Registered: 10/01/07
Posts: 5
|
|
|
Reply with quote | #6 |
-Overall, I thought it was a comprehensive document. However there was a missing emphasis on observational quantity, quality, and data assimilation. A large multi agency investment in observations - their quantity, quality, and assimilation in models - is a critical need not addressed. Initial NCEP tests as to why the ECMWF performs well on some cases when the GFS does not point to a variety of observational data issues (QC, types, and their assimilation) (Recent WAF/NWP AMS conference talks by Alpert et al. and Kumar et al.). Also, there is mounting evidence that adaptive observation programs (THORPEX TPARC and the NCEP Winter Storms Reconnaissance Program lead to forecast improvement for minimal investment. Improved estimates of the current state will help narrow the ensemble distributions. I think this should be it's own sub-objective under objective 2 (or 4). |
| |
ableistein
ACUF Moderator
Registered: 04/19/07
Posts: 50
|
|
|
Reply with quote | #7 |
I did read through the entire draft and think its pretty good. But for the record, fundamentally I'm not clear how this document will be used. The concern I have (and there is nothing that really can be done about this) is that it will not be used to drive progress for the "Enterprise". To that end this document might be received more of a wish-list than a road map.
The goals as stated are proper however a lot of the objectives seem to be those that will occur anyway outside the framework ACUF. For example - model improvement, better initializations, etc.
Another concern - are the "leads" as listed aware they are "leads" in ensuring progress is made? The leads as defined seem too generic - example "government" - who in the government is taking the lead on improving ensemble model performance? In this sense the goals and objectives stated in the document have no practical way of being carried out.
These are just thoughts and not show stoppers - but there has been a lot of work to get to this point and I don't want this document die in someone's "temp" directory.
Pete Manousos |
| |
anomaly
Registered: 10/15/05
Posts: 6
|
|
|
Reply with quote | #8 |
Here are comments from Chuck Doswell:
Having looked over the roadmap in Paul's presentation, it looks to me as if the dominant theme of the goals remains what I see as scientific and technical issues regarding the development of probabilistic forecast products. Yes, some attention is paid to the general topic that concerns me the most: the research needed to ensure that any new products introduced will be EFFECTIVE (Strategic Goal #3), but the fact that it's goal #3 seems to put it on a back burner. Developing a new product suite before we have much information about how to ensure its effectiveness in serving user needs looks like a classic case of putting the cart before the horse. I understand why a community consisting mostly of meteorologists would see priorities this way, but it's not a good "roadmap" in my opinion.
Chuck Doswell |
| |
remorss
Registered: 09/21/07
Posts: 7
|
|
|
Reply with quote | #9 |
Overall, I agree with previous comments that the draft is fairly comprehensive and pretty good. I'm also not sure how it will be used and agree that being concise (to the extent possible) will help with usability.
Two general comments:
1) Some of the objectives seem very government-focused, as does the overall feel of the document in places. Are the private and academic sectors' interests adequately reflected in the "vision" and road map?
2) In a number of places, the document is focused on one-way education and "outreach" from the enterprise to users. The overall impression is that approach is for the enterprise to spend a lot of time, money, and effort developing uncertainty information, and then to push the results out to users, educating them along the way about how useful the information will be to them and how they can use it. This paradigm is unlikely to be successful unless there is a strong component of feedback from users to the enterprise, starting at the beginning and maintained throughout. There is some language discussing this two-way education but it gets lost in a number of places. |
| |
Wxman007
Registered: 08/18/09
Posts: 1
|
|
|
Reply with quote | #10 |
This paper is well-done. This was my first opportunity to review it and I’m very impressed. I do have two general comments.
1. Assuming that this report will have readership beyond meteorologists, the term “hydrometeorology” (and its other forms) should be defined up front.
2. The report states that there are really only two main groups dealing with the issue of communicating forecast uncertainty, the Meteorological community and the User community. I believe that there is a third group which must be added, and that is the Interface group, which resides between the other two. I view this group as the ones who will do three things: understand what the Meteorologist community is trying to state, understand how the User community (multiple) “hears” the weather information and the form it must take to be useful, and develop tools (decision support tools or systems) for the translation from “weather information” into “user information” . The Interface group is made up of several sub-groups, such as social scientists (already mentioned in the report), risk management experts, economists and others.
I realize that social scientists are mentioned in the report, as are decisions support tools, but I believe it’s important to elevate the importance of social scientists and the others I highlighted above. While it is certainly important that meteorologists understand the users of their products and services (and for NWP models to provide output in User-specified terminology), they will never be able to convert their forecast (which is really grey-scale) into the binary answer that most Users need. Conversely, the Interface groups do have the ability to do that. This is covered to a limited degree in Appendices C and D, while Scenario 3 in Appendix E mentions a study done involving American Airlines which confirms the point I made just above.
If the Team feels that this is logical, then there are a number of areas which must be edited. In fact there are too many for me to list or create. If the Team doesn’t feel that the report should go this far or if it goes beyond the bounds established for the group, I would lobby that there should be a mention of the importance of engaging Social Science in the Executive Summary. |
| |
jdemuth
Registered: 09/26/07
Posts: 3
|
|
|
Reply with quote | #11 |
I’ll caveat my comments by saying that I went through the document pretty quickly.
A couple overarching thoughts:
-- Overall, I’d like to see more discussion about the roles of the social sciences, especially earlier on in the document. This could include an overview describing what the social sciences are and why they’re relevant. More specifically, I think we should better frame the roles of the social sciences beyond simply suggesting that we “educate”, “assist”, or in some other way help users of forecast uncertainty information better “interpret and use forecast information to improve their decision-making”. We need to more strongly weave in the need to assess how users (all users –- not just members of the general public) access, interpret, prefer, and use forecast uncertainty information. Moreover, this needs to be explored in multiple contexts – different users, regions, weather phenomena, etc. It’s also essential to realize this is a constantly changing field of research as we improve our predictions, as technology changes, etc. In other words, we can’t do one research project and generalize across all populations, all events, and all times. But that’s what makes this an exciting field of study!
-- I’m concerned that we make several statements in the report (primarily from the social science point-of-view) that we assume to be true, but that we don’t know empirically. I can provide some examples...
A few specific comments (these are NOT exhaustive):
-- Re: The success criteria for strategic goal #3 … evidence that people are making better decisions can/should be both quantitative and qualitative
-- Re: Strategic goal #4 … “enabling forecast uncertainty research” should include efforts on both the physical science and social science fronts.
-- We should reframe how we present the roles of social science in the “Academic sector” piece. It’s not just about teaching meteorologists social science, but about developing sustainable partnerships between meteorologists and social scientists from a range of disciplines. Moreover, social science goes beyond simply understanding people’s response to information.
-- I didn’t read through the text of every objective, but I can see that there are several places we can work in the roles of the social sciences, especially throughout Strategic Goal #3. I can also see places to add in roles of the social sciences in Goals #1 and #4.
|
| |
lupoa
Registered: 08/24/09
Posts: 1
|
|
|
Reply with quote | #12 |
As a late comer, I would have to agree with the comments that the document is thorough, and it's well done. Under strategic goal 1, non-ensemble products should be included, but they can be assumed to be embedded in that goal.
It is good to see that research will be done to test the effectiveness of new products. This means doing some research regarding where the end-users are in their understanding of current and new forecast products. There may have to be some effort expended in educating the public. For example, as Hurricane Bill approached the coast, the "cone of probability" for tropical storm force winds touched Massachusetts and areas near there. While the probabilities were low (20%), some I spoke to who had seen the product took it as a "done deal" that Boston was going to get a hurricane.
__________________
Anthony R. Lupo |
| |
PSchultz
Registered: 09/24/07
Posts: 3
|
|
|
Reply with quote | #13 |
The ESRL Global Systems Division conducted two workshops here in Boulder with NWS forecasters on the subject of communicating uncertainty. The first one was last October and concentrated on the logistics of producing probabilistic forecasts on AWIPS. The second was the first week of August and was broadened to include uncertainty more generally. I feel fairly confident saying from the perspective of the NWS/WFOs, communicating uncertainty will take three forms: probabilities, alternative scenarios, and decision support via human interaction. Although WFOs would not be involved in the production of alternative scenarios (that would be between MDL and NCEP), that kind of uncertainty information would obviously be useful for the purposes of decision support.
I thought some of the vagueness in the current draft is related to a lack of clarity about exactly what is meant by "communicating uncertainty". Maybe this simple concept can help with that. I have not yet made specific recommendations regarding where changes should be made, but I'll do that if people think it's useful. |
| |
