Insitu wind and precipitation completeness for climate monitoring

logo

Please note that this repository is used for development and review, so quality assessments should be considered work in progress until they are merged into the main branch

Insitu wind and precipitation completeness for climate monitoring#

Production date: 05/09/2024

Produced by: Ana Oliveira (CoLAB +ATLANTIC)

Use Case: Impact of climate change on precipitation over the Iberian Peninsula.#

Quality Statement question#

  • How consistent are the estimates of precipitation and wind speed trends from E-OBS compared to those from reanalysis products?

In this Use Case we will access the E-OBS daily gridded meteorological data for Europe from 1950 to present derived from in-situ observations (henceforth, E-OBS) data from the Climate Data Store (CDS) of the Copernicus Climate Change Service (C3S) and analyse the spatial consistency of the E-OBS air temperature climatology over a given Area of Interest (AoI).

The Quality Statements of this Use Case are:#

  • Accordingly, with Bandhauer, M. et al (2021), both E-OBS and ERA5 demonstrate accuracy in capturing precipitation patterns, with E-OBS showing good agreement in regions with dense station networks while ERA5 exhibits consistent mesoscale pattern representation. The authors also concluded that E-OBS can capture extreme values of precipitation in areas with high data density but may smooth out extreme events in data-sparse regions, however ERA5 shows improved detection of extreme precipitation events due to advanced physical modeling.

  • Rivoire P., Martius O. and Naveau P. (2021) studied the consistency of precipitation and wind speed trends in ERA-5 compared to 2 observational datasets (E-OBS and CMORPH) analysis; since ERA-5 offers regular spatial and temporal coverage, the precipitation data are derived from short-term forecasts (not observations), the authors wanted to fill the gap in large-scale assessments of ERA-5 comparing it with E-OBS over Europe and globally with CMORPH, emphasizing intensity distribution and co-occurrence of events. The authors showed that ERA-5 demonstrated a high level of accuracy in capturing precipitation trends compared to E-OBS, in Europe where discrepancies are smaller. However, they found some differences in extreme precipitation events, indicating potential precision limitations in ERA-5 data, especially in tropical regions.

Methodology#

1. Define the AoI, search and download E-OBS
1.1. Download and prepared E-OBS data
2. Merge E-OBS with ERA5
3. Inspect and view the merged datasets
4. Calculate the Probability density function (PDF)
5. Calculate the the annual count of days
6. Map of the annual count of days
7. Main takeaways

1. Define the AoI, search and download E-OBS#

Import all the libraries/packages#

We will be working with data in NetCDF format. To best handle this data we will use libraries for working with multidimensional arrays, in particular Xarray. We will also need libraries for plotting and viewing data, in this case we will use Matplotlib and Cartopy.

Hide code cell source 
import cartopy.crs as ccrs
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import xarray as xr
import pymannkendall as mk
from c3s_eqc_automatic_quality_control import download, utils, diagnostics, plot

# # import libraries
# from cdo import cdo

# # check cdo
# cdo = Cdo()
# help(cdo.sinfov)

plt.rcParams["figure.figsize"] = [15, 5]
plt.style.use("seaborn-v0_8-notebook")

import warnings
import os
warnings.filterwarnings("ignore")

Data Overview#

To search for data, visit the CDS website: http://cds.climate.copernicus.eu. Here you can search for ‘in-situ observations’ using the search bar. The data we need for this tutorial is the E-OBS daily gridded meteorological data for Europe from 1950 to present derived from in-situ observations. This catalogue entry provides a daily gridded dataset of historical meteorological observations, covering Europe (land-only), from 1950 to the present. This data is derived from in-situ meteorological stations, made available through the European Climate Assessment & Dataset (ECA&D) project, as provided by National Meteorological and Hydrological Services (NMHSs) and other data-holding institutes.

E-OBS comprises a set of spatially continuous Essential Climate Variables (ECVs) from the Surface Atmosphere, following the Global Climate Observing System (GCOS) convention, provided as the mean and spread of the spatial prediction ensemble algorithm, at regular latitude-longitude grid intervals (at a 0.1° and 0.25° spatial resolution), and covering a long time-period, from 1 January 1950 to present-day. In addition to the land surface elevation, E-OBS includes daily air temperature (mean, maximum and minimum), precipitation amount, wind speed, sea-level pressure and shortwave downwelling radiation.

The E-OBS version used for this Use Case, E-OBSv28.0e, was released in October 2023 and its main difference from the previous E-OBSv27.0e is the inclusion of new series and some corrections for precipitation stations.

Having selected the correct dataset, we now need to specify what product type, variables, temporal and geographic coverage we are interested in. In this Use Case, the ensemble mean of precipitation (RR) and daily mean wind speed (FG) will be used, considering the last version available (v28.0e). These can all be selected in the “Download data” tab from the CDS. In this tab a form appears in which we will select the following parameters to download, for example:

  • Product Type: Ensemble mean

  • Variable: daily precipitation sum and daily mean wind speed

  • Grid resolution: 0.25

  • Period: Full period

  • Version: 28.0e

  • Format: Zip file (.zip)

At the end of the download form, select Show API request. This will reveal a block of code, which you can simply copy and paste into a cell of your Jupyter Notebook …

Download data … having copied the API request to a Jupyter Notebook cell, running it will retrieve and download the data you requested into your local directory. However, before you run it, the terms and conditions of this particular dataset need to have been accepted directly at the CDS website. The option to view and accept these conditions is given at the end of the download form, just above the Show API request option. In addition, it is also useful to define the time period and AoI parameters and edit the request accordingly, as exemplified in the cells below.

1.1. Download and prepared E-OBS data#

Hide code cell source 
# Define request
request = (
    "insitu-gridded-observations-europe",
    {
        "format": "zip",
        "product_type": "ensemble_mean",
        "variable": ["precipitation_amount", "wind_speed"],
        "grid_resolution": "0.25deg",
        "period": "full_period",
        "version": "28.0e",
        "area": [42.15, -9.50, 36.95, -6.19],
    },
)

# Process the request
ds_EOBS = download.download_and_transform(*request)
Hide code cell source 
# Subset data for the year range 1950 to 2020
ds_EOBS = ds_EOBS.sel(time=slice('1950-01-01', '2020-12-31'))

# Inspect data
ds_EOBS
Hide code cell source 
# Rename and update attributes for 'rr'
ds_EOBS = ds_EOBS.rename({'rr': 'RR EOBS'})
ds_EOBS['RR EOBS'].attrs['long_name'] = 'Total Precipitation'
ds_EOBS['RR EOBS'].attrs['units'] = 'mm'

# Rename and update attributes for 'var2'
ds_EOBS = ds_EOBS.rename({'fg': 'WS EOBS'})
ds_EOBS['WS EOBS'].attrs['long_name'] = 'Wind Speed'
ds_EOBS['WS EOBS'].attrs['units'] = 'm/s'

ds_EOBS

Download ERA5 and prepared data#

Hide code cell source 
# Define time interval
start = "1950-01"
stop = "2020-12"

# Define request
collection_id = "reanalysis-era5-single-levels"
request = {
    "product_type": "ensemble_mean",
    "format": 'netcdf',
    'variable': [
            '10m_u_component_of_wind', '10m_v_component_of_wind', 'total_precipitation',
        ],
    "time": [f"{hour:02d}:00" for hour in range(0, 24, 3)],
    "area": [44, -10, 36, 1],

} 

requests = download.update_request_date(request, start, stop)

# Process the request
ds_ERA5 = download.download_and_transform(collection_id, requests, chunks={"year": 1})
Hide code cell source 
# Subset data for the year range 1950 to 2020
ds_ERA5 = ds_ERA5.sel(time=slice('1950-01-01', '2020-12-31'))

# Inspect data
ds_ERA5
Hide code cell source 
# Convert units for precipitation from meters to millimeters
ds_ERA5['tp'] = ds_ERA5['tp'] * 1000
Hide code cell source 
# Resample precipitation to daily frequency and sum up
ds_ERA5_precip = ds_ERA5['tp'].resample(time='1D').sum(dim='time')
ds_ERA5_precip = ds_ERA5_precip.rename('RR ERA5')
ds_ERA5_precip.attrs['long_name'] = 'Total Precipitation'
ds_ERA5_precip.attrs['units'] = 'mm'
Hide code cell source 
ds_ERA5_precip
Hide code cell source 
# Calculate wind speed from U and V components
u10 = ds_ERA5['u10']
v10 = ds_ERA5['v10']
wind_speed_ERA5 = np.sqrt(u10**2 + v10**2)

# Resample wind speed to daily frequency
wind_speed_daily = wind_speed_ERA5.resample(time='1D').mean(dim='time')
wind_speed_daily = wind_speed_daily.rename('WS ERA5')
wind_speed_daily.attrs['long_name'] = '10m Wind Speed'
wind_speed_daily.attrs['units'] = 'm/s'
Hide code cell source 
# Combine ERA5 variables into one dataset
ds_ERA5_daily = xr.Dataset({'RR ERA5': ds_ERA5_precip, 'WS ERA5': wind_speed_daily})

2. Merge E-OBS with ERA5#

Hide code cell source 
# Interpolate ERA5 data to match E-OBS resolution
ds_ERA5_interp = ds_ERA5_daily.interp_like(ds_EOBS)

# Merge the datasets
merged_ds = xr.merge([ds_EOBS, ds_ERA5_interp], join='left')
Hide code cell source 
print(merged_ds)

3. Inspect and view the merged datasets#

Precipitation amount and Wind Speed plot representation, with an overview of a selected subset (year of 2000)

Hide code cell source 
# Calculate mean values over latitude and longitude
mean_ds = merged_ds.mean(dim=('longitude', 'latitude'))
Hide code cell source 
# Plot the Precipitation data
plt.figure(figsize=(10, 6))
plt.title('Precipitation Comparison: E-OBS vs ERA5')
plt.xlabel('Date')
plt.ylabel('Precipitation (mm)')

plt.plot(mean_ds.time, mean_ds['RR EOBS'], label='E-OBS Precipitation', color='red')
plt.plot(mean_ds.time, mean_ds['RR ERA5'], label='ERA5 Precipitation', color='blue')

# Add legend
plt.legend(loc='upper left')

# Show the plot
plt.grid(True)
plt.show()
Hide code cell source 
# Specify the year you want to subset (e.g., 2000)
year_to_subset = '2000'

# Subset the dataset for the specified year
subset_rr = mean_ds.sel(time=year_to_subset)

subset_rr
Hide code cell source 
# Plot the Precipitation data considering a subset
plt.figure(figsize=(10, 6))
plt.title('Precipitation Comparison - year 2000')
plt.xlabel('Date')
plt.ylabel('Precipitation (mm)')

plt.plot(subset_rr.time, subset_rr['RR ERA5'], label='ERA5 (RR)', color='blue')
plt.plot(subset_rr.time, subset_rr['RR EOBS'], label='E-OBS (RR)', color='red')

# Add a legend
plt.legend()

# Show the plot
plt.grid(True)
plt.show()
Hide code cell source 
# Plot the Wind Speed (WS) data
plt.figure(figsize=(10, 6))
plt.title('Wind Speed Comparison: E-OBS vs ERA5')
plt.xlabel('Date')
plt.ylabel('Wind Speed (m/s)')

plt.plot(mean_ds.time, mean_ds['WS ERA5'], label='ERA 5 (WS)', color='blue')
plt.plot(mean_ds.time, mean_ds['WS EOBS'], label='E-OBS (WS)', color='red')

# Add a legend
plt.legend()

# Show the plot
plt.grid(True)
plt.show()
Hide code cell source 
# Plot the Wind Speed (WS) data considering a subset
plt.figure(figsize=(10, 6))
plt.title('Wind Speed Comparison - year 2000')
plt.xlabel('Date')
plt.ylabel('Wind Speed (m/s)')

# Plot tg data
plt.plot(subset_rr.time, subset_rr['WS ERA5'], label='ERA5 (WS)', color='blue')
plt.plot(subset_rr.time, subset_rr['WS EOBS'], label='E-OBS (WS)', color='red')

# Add a legend
plt.legend()

# Show the plot
plt.grid(True)
plt.show()

Save the data into .csv and .nc formats#

Hide code cell source 
# Save as NetCDF for inspecting spatially averaged data
merged_ds.to_netcdf('RR_WS_EOBS_ERA5_merged.nc')
# Convert xarray dataset to pandas DataFrame
df = merged_ds.to_dataframe()
df.to_csv('RR_WS_EOBS_ERA5_merged.csv')

4. Calculate the Probability density function (PDF)#

Of each alternative 30-year period

Work in progress#

5. Calculate the the annual count of days#

When RR>=10mm; as defined by WMO Expert Team on Climate Change Detection Indices, ETCCDI

Work in progress#

6. Map of the annual count of days#

These maps will be compared to disclose if RR extremes are shifting in terms of mean and the distribution’s tails. The results from these maps shall be developed and compared considering the two datasets.

Work in progress#

7. Main takeaways#

Work in progress#

If you want to know more#

Key resources#

Some key resources and further reading were linked throughout this assessment.

The CDS catalogue entry for the data used were:

Code libraries used:

References#

[1] World Meteorological Organization (WMO) Guidelines on the Calculation of Climate Normals:

[2] Cornes, R., G. van der Schrier, E.J.M. van den Besselaar, and P.D. Jones. 2018: An Ensemble Version of the E-OBS Temperature and Precipitation Datasets, J. Geophys. Res. (Atmospheres), 123.

[3] Bandhauer, Moritz, Francesco Isotta, Mónika Lakatos, Cristian Lussana, Line Båserud, Beatrix Izsák, Olivér Szentes, Ole Einar Tveito, and Christoph Frei. 2022. “Evaluation of Daily Precipitation Analyses in E-OBS (V19.0e) and ERA5 by Comparison to Regional High-Resolution Datasets in European Regions.” International Journal of Climatology 42 (2): 727–47.

[4] Hofstra, Nynke, Malcolm Haylock, Mark New, and Phil D. Jones. 2009. “Testing E-OBS European High-Resolution Gridded Data Set of Daily Precipitation and Surface Temperature.” Journal of Geophysical Research Atmospheres.

[5] Rivoire, Pauline, Olivia Martius, and Philippe Naveau. 2021. “A Comparison of Moderate and Extreme ERA-5 Daily Precipitation With Two Observational Data Sets.” Earth and Space Science 8 (4): e2020EA001633.

Contents