3.10. Validation Results

This section discusses damage and loss results from three comparative and validation studies with different emphases:

1. The Influence of Year Built (Zillow vs. NSI and Parametric Study) first explores the implications of utilizing year built information based on two data resources (NSI and Zillow). As discussed in Asset Description, the Zillow-based model tends to predict higher year predictions (more recent constructions) than the NSI-based model, especially for the southern edge of the city. This is reflected by the fact that Zillow-based damage and loss in that region is lower than the NSI-based results. Following the parametric study in Verification of Rulesets, the observed trend in damage and loss with respect to year built is discussed.

2. The Validation: StEER contrasts the testbed’s damage estimates versus the visually-assigned damage states from StEER members and explores the influence of uncertainty in building attributes and peak wind speed. Several notable trends between building damage and building attribute type will be discussed and reasoned.

3. The Validation: FEMA Historical Geospatial Damage Assessment focuses on validating the testbed damage estimates against the FEMA Historical Damage Assessment Database (535 records associated with this testbed). The testbed results are found to slightly overestimate the damage for the undamaged state (DS-0), which suggests that future investigations are still needed given uncertainties around the methodology used to generate the FEMA data.

3.10.1. Influence of Year Built (Zillow vs. NSI and Parametric Study)

The year built is a critical attribute as it triggers attribute assignments by various rulesets. As discussed in Asset Description, different sources of data were used to predict the YearBuilt, engaging both the National Structural Inventory (NSI) (with SURF correcting some misalignments and gaps) and Zillow, which also provides the year built information for many of the residual buildings in the studied region. The implications of using different sources of year built information to project damage are explored in Fig. 3.10.1.1, noting the contrast map that depicts the difference between the damage state projected by SURF-NSI and that projected by Zillow. Note that the difference is not prominent in downtown Lake Charles, although divergence is noted southwest of the city where the Zillow set predicts relatively lower damage states. Compared in Fig. 3.10.1.2 are the CDF distributions of expected damage states (NSI vs. Zillow) as well as the probability of exceeding DS-2 and DS-3. Fig. 3.10.1.3 plots the cumulative distributions of expected damage states for four different year built bins: Pre 1960, 1960-1979, 1980-1999, and Post 2000. In general, we see improved building performance (low damage states) as code revisions by years as recalling the parametric study in Verification of Rulesets, the dominant building type is WSF1 (Fig. 3.1.4.3) whose wind-induced loss is particularly sensitive to nail spacing requirements varied by years.

Fig. 3.10.1.1 Comparison of Expected Damage States (NSI vs. Zillow)

Fig. 3.10.1.2 CDFs of expected DS and probability of exceeding DS-2 and DS-3.

Fig. 3.10.1.3 CDFs of expected DS by year bins (NSI results).

3.10.2. Validation: StEER

For Hurricane Laura, the Structural Extreme Event Reconnaissance (StEER) Network released a dataset of visual inspections and damage estimations for sample buildings in Lake Charles ([Roueche21], referred to as StEER buildings). This testbed includes 99 StEER buildings, providing an opportunity to validate the workflow. The relatively small sample size also allows for the investigation of the influence of different modeling parameters on the estimated damage states and their correlation with corresponding field observations. This helps in understanding the dominant trends in the data. As will be illustrated, some of these trends validate the effectiveness of the methodology, while others indicate potential measures to improve wind resistance.

Before presenting the results, it is important to note, as discussed in StEER’s report on Hurricane Laura ([Roueche21]), that StEER’s assessments for Laura were conducted virtually from street view imagery due to COVID-19 travel restrictions. Thus, this “ground truth” is less reliable in discerning some aspects of the damage than a traditional StEER mission with in-person, up-close forensic assessments. Also, since rulesets from New Jersey are applied to assign attributes and are not consistent with the historical regulatory environment in Louisiana, the predicted damage states are likely to be lower, particularly for construction older than 2006 (the year Louisiana first implemented its statewide building codes linked to IBC/IRC). Finally, as also discussed in StEER’s report ([Roueche21]), there were low rates of compliance with shuttering requirements in the affected area (based on observations immediately after landfall). Thus, it is likely that the shutter attributes are assigned in this workflow at a higher rate than actually observed in the event, leading to lower rates of predicted damage than those observed in the field.

Fig. 3.10.2.1 plots the estimated damage states against key building attributes that are found to influence the resulting damage state under the hurricane wind hazard. Major observed trends are as follows:

1. In general, the damage state is negatively correlated with the year built. Buildings constructed after 2000 are found to have a significant improvement in wind resistance and have lower damage states. This is reasoned by the authors to be led by the assumption in the ruleset that post-2000 buildings would have shutter measures and stronger roof deck attachments. It is important to note, however, that this effect predicted based on New Jersey rulesets will not be observed in Louisiana until at least 2006 when statewide codes were enacted. Furthermore, it is likely to not be pronounced given the earlier observation about low rates of compliance with shuttering requirements.

2. The surface roughness, i.e., the terrain feature, is found to be a key variable when considering the potential damage from a hurricane to a building. Even when moving from open terrain to light suburban, following the damage functions in HAZUS, we observe about a 50% reduction in the average damage state. This reduction is associated with the decrease in mean wind speed as surface roughness increases.

3. Shutters could significantly help reduce the potential wind-induced damage. The average damage state is reduced from 2.2 to 1.2. This observation makes the low compliance noted by StEER particularly noteworthy.

4. The type of garage also influences the building’s performance against wind hazards. Garages per SFBC 1994 are found to exhibit much better behavior than standard and weak garages. This effect is not expected to be pronounced in Lake Charles given the low incidence of attached garages noted in the samples discussed in Asset Description.

5. As expected, a roof deck attachment using a tighter nail spacing (6 in vs. 12 in) is found to perform much better.

6. As also expected, a gable roof is found to have slightly worse performance than hip and flat roofs.

Fig. 3.10.2.1 Influential building attributes on the wind-induced damage state.

It is noted that the trends observed in Fig. 3.10.2.1 are based on a so-called “median” model representing our best estimates of building attributes. However, in reality, due to the lack of high-resolution or multiple data resources, we may not have 100% confidence in certain building attributes or hazard fields, e.g., the year built or peak wind speed at a specific site. Hence, the estimated “median” damage state from the “median” model may not fully represent the real damage potential. As illustrated in Fig. 3.10.2.2, if we consider the uncertainty in the year built (instead of a deterministic value, assuming it follows a normal distribution with a standard deviation of 10 years), we could sample the year built from the distribution and repeat the damage assessment a sufficient number of times to estimate the distribution of the damage state. This, however, assumes differences are attributable to uncertainties in the year built and neglects the fact that the predictors are biased by attribute assignments using rulesets that are derived from a different regulatory environment (New Jersey) with a longer history of adopting and enforcing model building codes statewide. This fact may explain why the predicted damage states bias toward lower levels of damage (assuming less vulnerable structures than was the case in practice).

Fig. 3.10.2.2 Uncertainty in the estimated damage state.

Following this idea, for each of the 99 StEER buildings, we sampled 100 “year built” values from a distribution with the mean of the given “year built” value and a standard deviation of 10 years, and ran the damage assessment for each sampled case. Fig. 3.10.2.3 summarizes the comparison between the mean and 95-percentile of the estimated damage state and the StEER damage state. Similar exercises (Fig. 3.10.2.4) are conducted for the peak wind speed where we sampled 100 PWS from a distribution with a standard deviation of 20 mph.

Fig. 3.10.2.3 Influence of uncertainty in “year built” on the estimated damage state.

Fig. 3.10.2.4 Influence of uncertainty in PWS on the estimated damage state.

3.10.3. Validation: FEMA Historical Geospatial Damage Assessment

FEMA and the Department of Interior lead the development of the Geospatial Platform, an internet-based service environment that provides a suite of well-managed geospatial data, services, applications, and tools. The FEMA Historical Damage Assessment Database is a repository of geospatial damage assessments from past National Disaster events where damage assessments were conducted either using high-resolution imagery or by means of geospatial modeling.

For Hurricane Laura, the damage categories of 112,571 buildings in Louisiana were modeled from flood depths at the structure as characterized based on modeled wind, flood, or surge data. Five damage categories are used: No Damage (NOD), Affected (AFF), Minor (MIN), Major (MAJ), and Destroyed (DES). Out of the 112,571 buildings, 535 buildings, located in the westbound of Lake Charles city at the waterfront of Prien Lake, are also investigated in this testbed. Table 3.10.3.1 is a subset of data obtained from FEMA Public Data. Table 3.10.3.2 is the building inventory for this subset. The full inventory and FEMA damage assessment results can be accessed by FEMA Damage Assessment of the 535 buildings and BIM inventory of the 535 buildings.

Fig. 3.10.3.1 FEMA Historical Geospatial Damage Assessment Database (2020 Hurricane Laura).

Table 3.10.3.1 FEMA Damage Assessment example buildings in Lake Charles.

ID	DMG_LEVEL	COUNTY	STATE	LONGITUDE	LATITUDE
1	DES	Calcasieu	LA	-93.29	30.17
2	NOD	Calcasieu	LA	-93.30	30.17
3	NOD	Calcasieu	LA	-93.29	30.17
4	NOD	Calcasieu	LA	-93.29	30.17
5	NOD	Calcasieu	LA	-93.28	30.16
6	NOD	Calcasieu	LA	-93.29	30.17
7	NOD	Calcasieu	LA	-93.29	30.17
8	NOD	Calcasieu	LA	-93.29	30.17
9	NOD	Calcasieu	LA	-93.30	30.17
10	AFF	Calcasieu	LA	-93.28	30.17
11	NOD	Calcasieu	LA	-93.29	30.16
12	NOD	Calcasieu	LA	-93.30	30.17
13	NOD	Calcasieu	LA	-93.29	30.17
14	NOD	Calcasieu	LA	-93.28	30.17
15	NOD	Calcasieu	LA	-93.28	30.17
16	NOD	Calcasieu	LA	-93.29	30.16
17	NOD	Calcasieu	LA	-93.29	30.17
18	NOD	Calcasieu	LA	-93.29	30.17
19	NOD	Calcasieu	LA	-93.29	30.17
20	AFF	Calcasieu	LA	-93.28	30.16

Table 3.10.3.2 BIM inventory of example buildings in this testbed overlapped with the FEMA Damage Assessment Database.

ID	RoofShape	FloodZone	Area	Longitude	Latitude	Terrain	DSWII	BuildingType	AvgJanTemp	Garage	NumberOfStories	YearBuilt	RoofSlope	MeanRoofHt	OccupancyClass
1	Gable	X	112	-93.29	30.18	51	132	Wood	Above	0	1	1994	0	12	RES3
2	Hip	X	148	-93.30	30.17	42	132	Wood	Above	0	1	1998	0.06	13.1	RES3
3	Gable	X	65	-93.29	30.17	11	132	Wood	Above	1	1	1994	0.08	13.5	RES3
4	Gable	X	110	-93.29	30.18	52	132	Wood	Above	0	1	1995	0.05	12.9	RES3
5	Hip	X	581	-93.29	30.17	11	132	Wood	Above	0	1	1996	0.05	17.3	RES1
6	Gable	X	83	-93.29	30.17	11	132	Wood	Above	0	1	1999	0.17	15.1	RES1
7	Hip	X	148	-93.30	30.17	42	132	Wood	Above	0	1	1998	0.06	13.1	RES3
8	Gable	X	55	-93.28	30.17	11	132	Wood	Above	0	1	1997	0.16	14.9	RES3
9	Hip	X	487	-93.29	30.16	11	132	Wood	Above	1	1	1996	0.07	13.2	RES1
10	Hip	X	107	-93.30	30.17	42	132	Wood	Above	0	1	1998	0	12	RES3
11	Gable	X	423	-93.29	30.17	11	132	Wood	Above	0	1	1998	0.13	11.3	RES1
12	Flat	X	144	-93.28	30.17	52	132	Wood	Above	1	1	1995	0.2	12	RES3
13	Gable	X	101	-93.29	30.17	42	132	Wood	Above	1	2	1997	0.24	16.4	RES1
14	Gable	X	86	-93.29	30.17	11	132	Wood	Above	0	1	1998	0.51	17.2	RES1
15	Hip	X	797	-93.29	30.17	51	132	Wood	Above	1	1	1995	0.04	12.8	RES3
16	Flat	X	90	-93.30	30.17	11	132	Wood	Above	0	2	1999	0.65	33.3	RES1
17	Gable	X	129	-93.29	30.17	11	132	Wood	Above	1	1	1997	0.24	16.4	RES3
18	Gable	X	704	-93.29	30.17	42	132	Wood	Above	1	2	1998	0.07	13.3	RES1
19	Gable	X	112	-93.29	30.17	11	132	Wood	Above	0	1	1998	0.11	16.3	RES3
20	Gable	X	137	-93.28	30.17	11	132	Wood	Above	0	1	1997	0	12	RES3

Fig. 3.10.3.2 shows the damage categories collected from the FEMA Damage Assessment Database, where most buildings are assessed as having No Damage or being Affected. It is also noted from Table 3.10.3.1 that all these buildings are subject to CAT-4 gust wind speed, which is consistent with our input wind field where the 3-s gust peak wind speed is about 130 \(mph\) (please see Hazard Characterization).

To compare these damage categories with the estimated damage states by the workflow in this testbed, we assume that the five damage categories are mapped to the five damage states in the HAZUS wind damage functions, assuming the inundation does not cause significant flood-induced damage to the buildings. While this assumption will be the subject of future validation, Fig. 3.10.3.3 summarizes (1) the expected damage states of the 535 buildings, (2) the spatial distribution of the difference between the FEMA Damage Categories and estimated damage states, and (3) the statistical distribution of the difference between the two. Given that the FEMA set is based on a discrete (category-based) description while the testbed set is based on probabilistic estimates, the two results have fairly good agreement except that the FEMA set has buildings predicted as no damage (DS-0). This over-prediction of DS-0 is also seen in StEER set results (Fig. 3.10.2.3 and Fig. 3.10.2.4). The uncertainty in the year built and wind speed, to some extent, can help explain this over-prediction, as can the fact that the rulesets applied are translated directly from New Jersey. It is further important to note that FEMA assessments also consider the effects of storm surge. Future investigations are still needed to consider other possible factors causing this mismatch in predicted DS-0.

Fig. 3.10.3.2 FEMA Damage Category in the Geospatial Damage Assessment Database (for the 535 buildings).

Fig. 3.10.3.3 Comparison: FEMA Damage Category vs. expected damage state by the workflow (for the 535 buildings).

Roueche21(1,2,3)

Roueche, D. Kameshwar, S. Vorce, M. Kijewski-Correa, T. Marshall, J. Mashrur, N. Ambrose, K. Brown, C. Childress, O. Fox, D. Morris, K. Rawajfih, H. Rodriguez, L. (2021) “Field Assessment Structural Teams: FAST-1, FAST-2, FAST-3”, in StEER - Hurricane Laura. DesignSafe-CI. https://doi.org/10.17603/ds2-dha4-g845