EVALUATING HYDROLOGIC MODEL PERFORMANCE OF GLOBAL AND LOCAL WEATHER DATA INPUTS

Runoffs from hydrologic models are often used in flood models, among other applications. These runoffs are converted from rainfall, signifying the importance of weather data accuracy. A common challenge for modelers is local weather data sparsity in most watersheds. Global weather datasets are often used as alternative. This study investigates the statistical significance and accuracy between using local weather data for hydrologic models and using the Climate Forecast System Reanalysis (CFSR), a global weather dataset. The Soil and Water Assessment Tool (SWAT) was used to compare the two weather data inputs in terms of generated discharges. Both long-term and event-based results were investigated to compare the models against absolute discharge values. The basin’s average total annual rainfall from the CFSR-based model (4062 mm) was around 1.5 times the local weather-based model (2683 mm). These basin precipitations yielded annual average flows of 53.4 cms and 26.7 cms for CFSR-based and local weatherbased models, respectively. For the event-based scenario, the dates Typhoon Ketsana passed through the Philippine Area of Responsibility were checked. CFSR only read a spatially averaged maximum daily rainfall of 18.8 mm while the local gauges recorded 157.2 mm. Calibration and validation of the models were done using the observed discharges in Sto. Niño Station. The calibration of local weather-based model yielded satisfactory results for the Nash-Sutcliffe Efficiency (NSE), percent of bias (PBIAS), and ratio of the RMSE to the standard deviation of measured data (RSR). Meanwhile, the calibration of CFSR model yielded unsatisfactory values for all three parameters.


INTRODUCTION
Hydrologic models, in the form of numerical modelling, are useful in a number of applications. For example, results of such models can subsequently be used as input to hydrodynamic, hydraulic, and even water quality models. In contrast to physical models, numerical hydrologic models are much more economical and flexible to use. Meteorological data is an important component in numerical hydrologic models. Accurate rainfall and other weather data are expected to yield more accurate simulated discharges. In the Philippines, however, there is a lack of ground-based local weather stations that have continuous data over long periods of time. For example, in the 24 subbasins that drain to Laguna Lake which have a total land area of 2891.4 sq. km, there are only 14 weather stations that contain data of more than 10 years (i.e. 13 -44 years of weather data, inclusive of gaps or missing data). Almost half of these stations are located in Marikina River Basin, leaving some of the 24 subbasins to have no weather station at all. This sparsity and the huge number of missing data in the available local weather datasets render hydrologic models using them as data input to be less accurate. Often, there is also a reactive instead of a proactive approach to monitoring these weather stations.
This proves to be a challenge in creating long-term continuous hydrologic models. An alternative is to use available global weather data like the China Meteorological Assimilation Driving Datasets for the SWAT model or CMADS and the Climate Forecast System Reanalysis or CFSR. This study focuses on the global weather dataset of CFSR, which has recently been an alternative input to hydrologic models for data-sparse regions (Bui et al., 2021). It was created to generate a uniform, continuous, best-estimate data of the interaction between the Earth's land, oceans, and atmosphere on a global scale. It offers readily usable input of rainfall, temperature, relative humidity, solar radiation and wind speed for the Soil and Water Assessment Tool (SWAT) model.
However, based on a study done for the Cau River Basin in Vietnam, CFSR produced overestimated rainfall and "unsatisfactory" simulated river discharges (Dao et al., 2021). A study reviewing the alternative climate products for SWAT modelling also shows that CFSR performs poorly for Asia and Africa, but performs well in the United States and South America (Tan et al., 2021). These observations are yet to be confirmed for watersheds in the Philippines due to the lack of studies as of the moment. For example, a study in Palico Watershed in Batangas utilized CFSR but only for weather parameters other than precipitation (Briones et al., 2016). A review of SWAT studies in Southeast Asia also showed that as of 2019, there had only been a total of 5 transnational studies published in the Philippines that utilized the SWAT model itself (Tan et al., 2019).
The lack of studies investigating the applicability of CFSR to watersheds in the Philippines, as well as the lack of gauges in some basins, prompts the objective of this study to see whether the CFSR weather data is a good substitute for ground-based local weather stations in terms of being used as an input to hydrologic models. The study is to focus on the Marikina River Basin due to 2 main reasons: 1) it has enough local weather stations to be put in contrast to the few grid points of the CFSR, and 2) the hydrologic model for the basin is relevant to the flooding in Metro Manila since Marikina River is the biggest tributary of Pasig River which inundates the country's capital during storm events, signifying the importance of accuracy of hydrologic models created for the basin.
The initial hypothesis is that the hydrologic model utilizing local weather stations would yield nearer discharges to the observed discharges than the model using CFSR weather data. The hydrologic model performance of the 2 weather datasets will be tested using the SWAT model.

Study Site
The Marikina River Basin has a catchment area of around 546 sq.km. It is located at the southern part of Luzon, about less than 5 kilometers northwest of Laguna Lake, and around 9 kilometers east of Manila Bay. Its basin is generally rounded in shape before its rivers converge into the Upper Marikina River. Its downstream areas are highly urbanized as it is already part of the National Capital Region (NCR).
As previously mentioned, the Marikina River Basin contributes the highest discharges to Pasig River which increases Metro Manila's risk of flooding during heavy storm events.

Modeling Platform
ArcSWAT, a SWAT model integrated in the GIS platform ArcMap, was utilized to create the hydrologic models. One of the advantages of ArcSWAT is its ability to maximize the spatial variability of the hydrological response units (HRUs) within the watershed upon overlaying the land cover, soil cover, and slope map which was derived from the terrain data used.

Digital Elevation Model:
The terrain data used in the hydrologic models was an Interferometric Synthetic Aperture Radar Digital Elevation Model (IFSAR DEM) resampled to a resolution of 10m. The IFSAR DEM, originally with a resolution of 5 m and a 1-meter root-mean-square-error vertical accuracy, is a Digital Terrain Model (DTM) and was obtained from the National Mapping and Resource Information Authority (NAMRIA). DTMs exclude the elevations of built-up structures and trees, thus allowing for a more natural flow of water over the topography. DTMs are often the type of DEM used in hydrologic models for this reason.

2.3.2
Land Cover Map: The land cover map used for all the hydrologic models created for this study was the NAMRIA Land Cover Map of 2015. The land cover classifications were simply grouped into 9 classes: forests, agricultural lands, urban lands (medium residential), range grasses, range brushes, inland water, barren, non-forested-and forestedwetlands. A limitation of this study is the use of a constant land-cover throughout the continuous hydrologic simulation.
The dominant land cover within the basin is brush/shrub with 44.5% surface area cover. Built-up areas take up about 21.4% mainly in the downstream areas while forests cover about 13.7% of the basin.

Soil Map:
A global and digital soil map from the Food and Agriculture Organization (FAO) was used for the soil cover. The FAO Soil Map also has a database that readily offers the inputs needed by SWAT which are a wide range of soil characteristics for multiple layers of soil.  Dystric nitosols are classified under hydrologic soil group D which is characterized by clayey soils and has the highest runoff potential among the four hydrologic soil groups. Meanwhile, both Gleyic Cambisols and Eutric Nitosols are under hydrologic soil group C which is characterized as sandy clay loam and has low infiltration rates.

Local Weather Data:
The local weather data were obtained from 2 agencies: Philippine Atmospheric, Geophysical and Astronomical Services Administration (PAGASA) and Effective Flood Control Operation System (EFCOS). Historical data from 10 PAGASA stations (Ambulong, NAIA, Port Area, Science Garden, Tanay, Tayabas, Boso-Boso, Mt. Oro, Sta. Cruz, and IRRI) were used to create a weather generator (WGEN) to fill in the gaps or missing data of all the weather gauges to be used for the local weather-based model. Some of these ten stations are already outside the Marikina River Basin's vicinity, but were still used either for the WGEN and/or for other weather parameters. IRRI Station in Los Baños, for example, is the only nearest gauge to the basin with long-term continuous solar radiation data. Pasig Station is also managed by PAGASA but wasn't included in the gauges used for the creation of WGEN due to its length of available data. The WGEN in ArcSWAT requires ideally at least 20 years of continuous data to create long enough statistics of the weather parameters to be used for interpolation in filling in the gaps. Since even the PAGASA gauges had large gaps in some of the gauges, manual filling in of data were done. Data of Port Area and NAIA weather stations, for example, were used to reinforce the data of Science Garden and Pasig stations. Four EFCOS stations (Mt. Campana, Aries, Nangka, and Napindan) were used to have a more representative weather data input during the latter years of the simulation. All in all, there were 11 ground-based local weather stations in the Marikina River Basin's vicinity that were used for the local weather-based model.

Global Weather Data:
The CFSR data has a grid resolution of 0.5° or approximately 56 km. It has available continuous weather data from January 1, 1979 to July, 31, 2014.
As can be seen in the figure, there are only 4 CFSR weather grid points within the basin's vicinity. However, since the method of weather data inputting in ArcSWAT is selecting the nearest gauge per subbasin, only three of these actually contributed to the global weather-based model (with the northwestern grid point being excluded). Figure 5 below shows all the local weather stations as well as the CFSR weather grid points.

Figure 5. Local weather stations and CFSR weather grid
points in the basin's vicinity.

Discharge Data:
The hydrologic models need to be calibrated and validated to ensure that the model is representative of the real-world system. The Sto. Niño Water Level Monitoring Station (WLMS) in Marikina was used as the point of calibration and validation. Daily water levels and corresponding discharges recorded from this station were obtained from the Department of Public Works and Highways (DPWH). The acquired daily flows were from year 2000 to 2010.

2.4.1
Weather Data Preparation: As stated for the local weather-based model, a WGEN was created from the PAGASA weather stations. This fills the gaps in any gauge for each timestep that that gauge has no data. Those stations which are already far from the Marikina River Basin were solely used for the WGEN. Each subbasin adopts the weather data of the nearest weather station during the simulation.
For the CFSR-based model, the CFSR database was directly inputted to the ArcSWAT model.

Watershed Delineation:
The Marikina River Basin was delineated using the auto-delineation process in ArcSWAT. This basically detects the topographic highs and lows from the DEM input to digitize the streams, determine the flow direction and flow accumulation, and finally, to delineate the subbasins. The process resulted to a total of 30 subbasins. The downstream-most outlet set was set near the confluence of Marikina River with Manggahan Floodway. A different outlet was also set at the Sto. Niño WLMS so that simulated flows can be extracted exactly at that point.

HRU Definition:
After inputting all the topographic data inputs and the land cover, soil cover, and slope classes were already overlain, the hydrologic response units or HRUs were defined to be the dominant land cover, soil type, and slope class per subbasin. This means that each subbasin is composed of only one HRU.

SWAT Simulation:
Two simulations were done, each with a different weather input: 1) local weather data and 2) gridded weather data of CFSR. The simulation period was set to 19 years starting from 1995 to 2013. The starting year of simulation was limited by the availability of data in the local weather dataset while the last year was dictated by the end period of the CFSR. Two years (2003 and 2004) were used as the calibration period while another year (2008) was used as the validation period. The performance of the two models was also checked in terms of capturing storm events by investigating the simulation results during the time Typhoon Ketsana (a.k.a. Tropical Storm Ondoy or TS Ondoy) passed by the Philippine Area of Responsibility.
ArcSWAT utilizes the SCS Curve Number Method to transform rainfall into runoff.
From here on, stand-alone words "Local" and "Global", as is indicated in table headers and plot legends, refer to the local weather-based and CFSR-based models respectively.

2.4.5
Statistical Tests: To see how the models performed, some statistical parameters were obtained to measure the nearness of the simulated to the observed flows. The Nash-Sutcliffe Efficiency (NSE) which ranges from -∞ to 1 measures the overall fit of the graph. A value of 1.0 indicates a perfect match of the simulated to the observed, while a negative value indicates that the mean observed value is even better than the simulated value (Moriasi et al., 2007). The percent of bias (PBIAS) shows how the simulated values can either be generally larger or smaller than their counterparts in the observed values (Moriasi et al., 2007). On the other hand, RSR is the ratio of the root-mean-square-error or RMSE and the standard deviation of measured data. This ranges from 0, which signifies optimal value, to a large positive value, which signifies a lesser fit of the simulated to the observed (Golmohammadi et al., 2014).

Precipitation
With the weather data input being the sole independent variable in the study, the major difference between the two models would be seen in the precipitation. The graph below shows the basin's annual average precipitation for both models. The average total annual rainfall produced by the CFSR is 4,062.66 mm. This is about 1.5 times the average total annual rainfall generated by the local weather stations which is only 2683.27 mm. The annual total rainfall for local weather data ranged from 2273.32 mm to 3346.81 mm while that of global weather data ranged from 3647.22 mm to 5002.63 mm. The average daily rainfall for wet season was found to be around 11.76 mm for the local weather and 19.07 mm for the global weather. For the dry season, it was 2.10 mm for the local weather and 4.24 mm for the global weather.

Flow Statistics
The long-term averages of flows were obtained for the whole period wherein observed data is available (i.e. 2000 -2010). The annual average flows at the location of Sto. Niño WLMS were 26.7 cms, 53.4 cms, and 44.1 cms for the local weatherbased model, CFSR-based model, and observed discharge data respectively. This makes the CFSR-based model fare better, yielding only about half as much deviation (20.9% higher) from the observed data as compared with its local weather counterpart (39.5% lower). The monthly and seasonal flow averages are presented in the table below.

Months
Average Flows (cms) While the average flows of the CFSR-based model were much nearer than the observed, the simulated flows of the local weather-based model generated higher R 2 values for each year when fitted with the observed. The average R 2 values generated were 0.52 for the local-weather based model and 0.22 for the CFSR-based model. The plot below shows the R 2 values per year. An R 2 value of greater than 0.5 is generally acceptable (Golmohammadi et al., 2014). The generally better fit of the local weather-based model's flows despite the lower resemblance in averages means that the trend (e.g. increase or decrease, presence and timing of peaks) of the actual discharges is more captured in this model.  The discharge hydrographs shown in Figure 9 show an almost consistent trend throughout the whole simulation. Peak discharges tend to be higher while baseflows are lower in the local weather-based model than in the CFSR-based model. Note, however, that these are already calibrated flows and the adjustment in some model parameters may have already effected this evident trend.

Water Balance
Along with the discharges, the resulting water balances of the two models were highly affected by the adjustments of parameters done in the calibration. The generally high flows from the CFSR-based model led to the lowering of runoff curve numbers, which subsequently leads to the lowering of surface runoff. The four components checked for the water balance are 1) surface runoff, 2) groundwater flow that returns to the streams as baseflow, 3) water percolating to aquifers, and 4) evapotranspiration. The table and chart below quantify the water balances of the two calibrated models. The groundwater flow, percolation, and evapotranspiration were all found to be greater than the surface runoff in both models. This can be attributed to Marikina River Basin's land cover distribution which is about 58% covered in either forests or brushes. It can also be attributed to the increasing of the soil's available water capacity during calibration: 70% and 10% for local weather-based and CFSR-based models respectively. These countered the low infiltration rates of the soil types dominant within the basin.

Sedimentation
The sediments flushing out of the Marikina River Basin through the downstream most reach (Upper Marikina River) was also checked. The annual average sediment outflow (241.2 metric tons) in the CFSR-based model was found to be about 1.7 times its counterpart in the local weather-based model (146.1 metric tons). The sediment yield from the reach is proportional to the river's discharge. As discussed previously, the annual average flow yielded by the CFSRbased model was about twice that of the local weather-based model. Unfortunately, comparison with observed data cannot be done due to the lack of sedimentation data.

Event-Based Scenario
To compare how the two datasets captured real storm events, rainfall values for the period (Sept. 24 -28, 2009) when Tropical Storm Ketsana (aka TS Ondoy) passed through the Philippine Area of Responsibility (PAR) were obtained. The CFSR data had an average daily rainfall of around 15.5 mm while the local gauges had an average of around 56.1 mm. It was also noticeable that the CFSR wasn't really able to capture Ondoy by looking at the peak daily rainfall values, in which the CFSR only had a maximum rainfall of 18.8 mm on Sept. 27 while the local gauges had a maximum rainfall of 157.2 mm on Sept. 25. The local gauges also read 91.5 mm of rainfall on Sept. 26, which was the landfall date of the storm. The corresponding yielded average discharges for this 5-day period were 105.0 cms and 280.1 cms for the CFSR-based and local weather-based models respectively. These average values along with the peak discharges fell short in comparison with the observed discharges which had an average of 631.0 cms for this period. The observed peak discharge was around 1,838 cms on Sept. 26, 2009.

Calibration and Validation
The collected daily discharge information at the Sto. Niño WLMS were used for calibrating and validating the simulated discharges. These absolute discharges were derived from water levels and a rating curve at the said station. The discharge hydrographs yielded by the two models had a remarkable difference, with the results from the local weatherbased model having evident resemblance to that of the observed. R-squared values of the initial, uncalibrated flows for the selected calibration period were 0.05 and 0.36 for the CFSR-based and the local weather-based models respectively.
Calibration was done by adjusting three parameters: 1) initial SCS runoff curve number for soil moisture condition II, 2) available water capacity of the soil, and 3) water depth threshold in shallow aquifer for occurrence of return flow. For both models, these same parameters were adjusted but different initial ranges and different number of simulations per iteration were selected in the auto-calibration procedure. The statistical parameters improved for both models, although only the local weather-based model yielded results that fell within the satisfactory ranges. Validation of the calibrated parameters for the local weather-based model yielded values of NSE and RSR that fell short by only 0.04 from the respective satisfactory ranges. The CFSR-based model yielded unsatisfactory statistical parameters after both the calibration and validation procedures.
The statistical parameters upon calibration and validation, as well as the discharge hydrographs for the selected validation period are presented, showing a much better fit of the local weather-based model with the observed.    While the CFSR was able to capture the general trend of increased flows during the wet season (May -Oct) in a basin located in a Type I Climate region such as the Marikina River Basin, it wasn't able to capture peaks arising from storm events unlike the local weather dataset.
The results of the calibration and validation, as well as the investigation of an event-based scenario in the simulations, show that the local weather-based model yielded discharges that are closer to the actual observed discharges than the CFSR-based model.
The CFSR weather data, however, can still be used if the main objective is to conduct long-term simulations to get an estimate of the average flows. The global dataset is good for this since the CFSR-based model yielded average flows nearer to the average of the observed (whether annual, monthly, or even seasonal). It can also be used in circumstances in which the local weather dataset is not long enough to be able to create its own weather generator for temporal interpolation for missing data, or in the event that there is really no groundbased local weather station in the area.
It is worth noting that the results of the study are assumed to be applicable only for the study area, and perhaps for basins with similar topography and in regions with similar climate or weather. In a study evaluating the CFSR weather data in the arctic watershed Malselv, 4 out of 5 subbasins yielded a higher NSE and R 2 value for the model using the global weather data as compared to the one using local weather data (Bui, 2021). Studies in tropical and sub-tropical regions might result to a more similar result as with this study. More studies in different types of topographies and climates can shed more light to the applicability of the CFSR, and its performance compared to local weather stations.

CONCLUSION
The hydrologic model utilizing spatially distributed local weather data fared much better than the model which used CFSR weather data. The resulting discharge hydrograph generated by the local weather-based model when compared with the observed discharges, yielded satisfactory values of NSE, PBIAS, and RSR after calibration, and a nearsatisfactory value of NSE and RSR after validation. Unfortunately, the model using CFSR weather data failed to reach satisfactory ranges for these three statistical criteria. This stresses the importance of having ground-based weather monitoring stations in currently ungauged basins wherein the hydrologic models are to be used for simulating event-based scenarios, at least for regions near the Marikina River Basin where the same results may also hold true. The CFSR-based model, however, is still deemed to be good for long-term simulations due to its good estimate of annual, monthly, and seasonal average flows.

RECOMMENDATION
The study presented the higher accuracy of the simulated discharges obtained by using local weather data inputs to the SWAT model as compared to using CFSR. However, it was beyond the scope of the study to see the effect of spatial distribution of local weather stations to the resulting discharges. It might be of practical use to also investigate the proper locations which would yield the optimal simulated discharges. This can be done by running the model with different combinations of weather stations.