Symbology Evaluation Results
(PDF version of the evaluation results)

Purpose

This document serves as a response to the comments gathered from the Public Evaluation of the Homeland Security Emergency Management Symbology. The evaluation was available on the website from December 8, 2003 - January 31, 2004. Several emergency management organizations were contacted and invited to participate:

• International Association of Emergency Managers (IAEM)
• National Association of Emergency Managers
• International Association of Fire Chiefs
• OASIS Emergency Management Technical Committee list
• EM forum
• Disaster Management Information System (DMIS)

The evaluation responses were provided to Kent State's Department of Geography for analysis. The compiled results and some demographic information as provided by the respondents are contained herein.

Next Steps

• Improve definitions of features
• ANSI preparation and submittal
• Examine extension of set to include other homeland security disciplines
• Examine extension of symbols to include lines and polygons

Survey Participants

Location

394 valid evaluations were received. In the evaluation, a respondent information section was used to capture some basic demographic information about the evaluation participants. Twenty-one percent of the participants provided information about their geographic location. The majority of responders were located in California, Washington State, Kansas, Florida and Mississippi. However, sixteen states were not reported as being part of the sample at all. These states were New Mexico, South Dakota, Nebraska, Oklahoma, Missouri, Louisiana, Indiana, Virginia, Georgia, South Carolina, Connecticut, Maine, New Hampshire, Vermont, Alaska and Hawaii.

The second part of the respondent information section attempted to identify if participants worked in the public or private sectors, and in which areas. Respondents were able to select more than one choice, and most respondents selected at least one area. As the following graph indicates, the majority of participants worked in the public sector.

Professional Background

An important aspect of communicating symbology preferences is identifying the occupation of evaluation participants. Only 13% of participants chose not to respond. Out of the 343 participants that did respond, 45% indicated that they work in emergency management, while 16% chose fire fighting and 39% chose GIS. Participants also indicated their roles and primary responsibilities. 55% indicated that they were managers, while 45% were technical specialists. These figures show that, in general, managers or GIS staff responded to the survey. The data suggests that the majority of survey participants were First Responders, the target community for this project.

Survey Sections

The symbology in this evaluation was broken down into the following five categories:

The evaluation contained a total of 214 questions. For each feature and accompanying symbol, the evaluation participant was asked to either "Accept" or "Reject" the symbol and the accompanying definition of the feature. Participants were also permitted to skip questions where they had no opinion; these "No preference" results were calculated into the survey totals.

Additionally, participants were given space to comment on every symbol and definition. In general, the majority of comments received were critical of specific symbol functionality or design. Participants often provided detailed explanations of how a symbol might be changed in order to make it more useful. These comments were taken into consideration along with a statistical analysis of results in order to determine which symbols should be considered for redesign.

Survey Results and Analysis

By applying a Mann-Whitney statistical comparison of the survey results to those of a pilot study, it was determined that those symbols that received lower than a 75% approval rating should be evaluated for review and redesign. After further review of the comments, sixteen symbols were revised. A summary of changes can be found at - http://www.fgdc.gov/HSWG/ref_pages/PrintableChanges.htm

The symbols that received a low approval rating are included in the sections below:

  Incidents Section

Of the five evaluation sections, the Incidents section received the highest number of rejections (compared with responses of "Approve" or "No Preference"). Comments and recommendations were particularly detailed and emphatic in this section, in comparison to overall evaluation results. In some cases, participants made detailed suggestions despite having given the symbol an overall "Approval" rating; in these cases the comments were still taken into consideration and the symbol reviewed.

  Civil Disturbance Incident (66% accepted, 16% rejected, 17% no preference)

  Civil Demonstrations (74% accepted, 11% rejected, 15% no comment)

  Civil Displaced Population (73% accepted, 12% rejected, 15% no preference)

  Civil Rioting (70% accepted, 14% rejected, 16% no preference)

  Bomb Threat (70% accepted, 14% rejected, 16% no comment)

  Bomb (69% accepted, 14% rejected, 17% no preference)

  Bomb Explosion (70% accepted, 13% rejected, 17% no preference)

  Industrial Facility Fire (70% accepted, 4% rejected, 26% no preference)

  Origin (71% accepted, 3% rejected, 26% no preference)

  Residential Fire (72% accepted, 2% rejected, 26% no preference)

  Organic Peroxides (74% accepted, 9% rejected, 17% no preference)

  Natural Events Section

Seven symbols within Natural Events had less than a 75% acceptance rate. These were Avalanche, Drizzle, Fog, Hail, Rain, Snow and Thunderstorm.

Particularly notable for its low acceptance rate was the symbol for Rain, which was accepted by only 62% of the respondents. Over 20% of total the evaluation participants commented on the Natural Events symbology. The remainder of the symbols in the Natural Events section was generally well received, with acceptance rates above 75%. Please note that changes were not made to Drizzle, Fog, Hail, Rain, Snow and Thunderstorm because these symbols are already part of an International Standard for weather symbology.

  Avalanche (74% accepted, 6% rejected, 20% no preference)

  Drizzle (68% accepted, 13% rejected, 19% no preference)

  Fog (73% accepted, 9% rejected, 18% no preference)

  Hail (68% accepted, 12% rejected, 20% no preference)

  Rain (62% accepted, 18% rejected, 21% no preference)

  Snow (74% accepted, 7% rejected, 19% no preference)

  Thunderstorm (71% accepted, 10% rejected, 19% no preference)

  Operations Section

While the majority of Operation symbols received favorable ratings, this section contains the four symbols with the lowest acceptance ratings of the entire evaluation. The lowest one, Emergency Collection Evacuation Point, garnered only a 59% acceptance rate and much criticism in the way of comments.

  Emergency Collection Evacuation Point(59% accepted, 20.5% rejected, 20.5% no preference)

  Emergency Incident Command Center(74% accepted, 9% rejected, 17% no preference)

  Emergency Public Service Center(74% accepted, 8% rejected, 18% no preference)

  Emergency Water Distribution Center(72% accepted, 9% rejected, 18% no preference)

  Infrastructure Section

This section, comprising 86 symbols (40% of the total symbols within the evaluation), had a high overall acceptance rate, and a low overall rejection rate. All symbols received an acceptance rating above 75%, with the majority around 80-83%, except for Religious Institution, which had an acceptance rate of exactly 75%.

  Religious Institution (75% accepted, 8% rejected, 17% no preference)

  Damage/Operational Section

This section presented an overall index of symbol categories by showing the outline of symbols in each section of the evaluation. Respondents rated this organizational scheme of levels very highly. The acceptance rate was between 80-83%, and the rejection rate between 2-5%.