{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![logo](./img/LogoLine_horizon_CAMS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Atmosphere Data Store (ADS) Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook provides you an introduction to the [Copernicus Atmosphere Data Store (ADS)](https://ads.atmosphere.copernicus.eu/) and gives you a practical example how to access and retrieve data from the ADS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outline\n",
    "* 1 - About\n",
    "* 2 - CAMS data retrieval\n",
    "  * 2.1 - Access data interactively via the ADS web interface\n",
    "  * 2.2 - Access data programmatically with the CDS API\n",
    "  * 2.3 - Further resources\n",
    "* 3 - Example: CAMS data request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<style>\n",
    "td, th {\n",
    "   border: 1px solid white;\n",
    "   border-collapse: collapse;\n",
    "}\n",
    "</style>\n",
    "<table align=\"left\">\n",
    "  <tr>\n",
    "    <th>Run the tutorial via free cloud platforms: </th>\n",
    "    <th><a href=\"https://mybinder.org/v2/gh/ecmwf-projects/copernicus-training-cams/master?labpath=da-ads-tutorial.ipynb\">\n",
    "        <img src = \"https://mybinder.org/badge.svg\" alt = \"Binder\"></th>\n",
    "    <th><a href=\"https://kaggle.com/kernels/welcome?src=https://github.com/ecmwf-projects/copernicus-training-cams/blob/master/da-ads-tutorial.ipynb\">\n",
    "        <img src = \"https://kaggle.com/static/images/open-in-kaggle.svg\" alt = \"Kaggle\"></th>\n",
    "    <th><a href=\"https://colab.research.google.com/github/ecmwf-projects/copernicus-training-cams/blob/master/da-ads-tutorial.ipynb\">\n",
    "        <img src = \"https://colab.research.google.com/assets/colab-badge.svg\" alt = \"Colab\"></th>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. About"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Copernicus Atmosphere Data Store (ADS)](https://ads.atmosphere.copernicus.eu/) is the data access portal of the [Copernicus Atmosphere Monitoring Service (CAMS)](https://atmosphere.copernicus.eu/) and offers access to data about the `Earth's past, present and future atmosphere`. Find more information in the [Atmosphere Data Store (ADS) documentation](https://confluence.ecmwf.int/display/CKB/Atmosphere+Data+Store+%28ADS%29+documentation)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![logo](./img/cams_data_store.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two ways to access data from the Copernicus Atmosphere Data Store (ADS):\n",
    "* [interactively](#access_manual) via the ADS web interface, or\n",
    "* [programmatically](#access_programmatic) with the CDS API\n",
    "\n",
    "In order to be able to access data from the Atmosphere Data Store (ADS), you first have to [self-register](https://ads.atmosphere.copernicus.eu/#!/home) at the ADS registration page (if you do not have an account yet) and second, you have to [login](https://ads.atmosphere.copernicus.eu/user/login) to the ADS portal. You will be able to browse the ADS web interface without beging logged in, but you have to login for being able to download data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Access data interactively via the ADS web interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ADS web interface` allows you to interactively `browse`, `select` and `download` data products offered by the ADS. First, under [Datasets](https://ads.atmosphere.copernicus.eu/cdsapp#!/search?type=dataset), you can browse and select the data you are interested in. In a second step, you can specify details of the data download form you wish to submit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filter and select a dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a first step, you can `browse` and `filter` the datasets you are interested in. The [Datasets](https://ads.atmosphere.copernicus.eu/cdsapp#!/search?type=dataset) interface allows you either to select data based on different categories, e.g. `Product type`, `Variable domain`, `Spatial / Temporal coverage`, but also offers a free text search. The list of datasets in the main window allows you to select the dataset you are interested in. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![logo](./img/ads_data_overview.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you selected a dataset, you then get redirected to a `data description` section, which provides you an overview of the chosen dataset. Under `Download data`, you have the option to specify the dataset you would like to download and to submit the download form."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submit the *Download data* form"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Data description` section (see 1 in the image below) provides you an overview of the dataset, including a list of variables that are available. Under the tab `Download data`, the `\"Download data\" form` opens (see 2), which allows you to interactively filter the dataset based on specific keywords, such as:\n",
    "* `Product type`\n",
    "* `Variable`\n",
    "* `Year / Month / Time`\n",
    "* `Geographical area`\n",
    "* `Format`\n",
    "\n",
    "At the end of the `\"Download data\" form`, you get two options to select: `Show API request` and `Submit Form`. If you want to download the data interactively, the data requests will be executed as soon as you click on the `Submit Form` button.\n",
    "\n",
    "**Note:** You need to be logged into the ADS portal and you have to accept the Terms and Conditions of the dataset before being able to download data.\n",
    "\n",
    "You will need the `Show API request`, if you want to request data programmatically. See [Section 2.2](#access_programmatic) for further information.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>NOTE</b>: <br>\n",
    "    Under the tab <code>Your requests</code> in the main menu, you can monitor the status of your data requests.</div>\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![logo](./img/ads_data_description_download_form.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Access data programmatically with the CDS API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `CDS Application Program Interface (CDS API)` is a Python library which allows you to access data from the ADS `programmatically`. The library is available for both Python versions, Python 2.7.x and Python 3, but we recommend to use the library under Python 3. In order to use the CDS API, follow the steps below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install the CDS API key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Self-register](https://ads.atmosphere.copernicus.eu/#!/home) at the ADS registration page (if you do not have an account yet)\n",
    "* [Login](https://ads.atmosphere.copernicus.eu/user/login) to the ADS portal and go to the [api-how-to page](https://ads.atmosphere.copernicus.eu/api-how-to)\n",
    "* Copy the CDS API key displayed in the black terminal window in a file under `$HOME/.cdsapirc`\n",
    "\n",
    "**Note:** You find your CDS API key displayed in the black terminal box under the section `Install the CDS API key`. If you do not see a URL or key appear in the black terminal box, please refresh your browser tab. \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![logo](./img/ads_api_key_terminal.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below creates the file under your current working directory. Make sure to replace the `################` with your personal `CDS API key`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile ./.adsapirc\n",
    "\n",
    "url: https://ads.atmosphere.copernicus.eu/api/v2\n",
    "key: ################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>NOTE</b>: <br>\n",
    "   The Copernicus Climate Change Service (C3S) uses the same CDS API software, but provides access to different datasets. If you are accessing data also from the Copernicus Climate Data Store and you already have a <code>~/.cdsapirc</code> file created, then you have to store your ADS key elsewhere. <br>\n",
    "    You can e.g. store your ADS key under <code>./.adsapirc</code> and you can pass the key information to the client library explicitly. See the code cell below how to do this. </div>\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cdsapi\n",
    "import yaml\n",
    "\n",
    "with open ('./.adsapirc', 'r') as f:\n",
    "    credentials= yaml.safe_load(f)\n",
    "    \n",
    "\n",
    "c = cdsapi.Client(url=credentials['url'], key=credentials['key'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Alternative: Set CDS API credentials manually"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can also define variables for `url` and `key`. These variables can then be set when you define the `cdsapi.Client(url=URL, key=KEY)` (see next section for the different components of a data retrieval request).\n",
    "\n",
    "**Please note:** in the workflow notebooks, we will use this modality and set manually the CDS API key information for the data retrievals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = 'https://ads.atmosphere.copernicus.eu/api/v2'\n",
    "KEY = '############################'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install the CDS API client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to install the `CDS API client`. You can do this with the package management system `pip`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install cdsapi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the CDS API client for data access"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the `CDS API` is installed, it can be used to request data from the Atmosphere Data Store."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, you see the principle of a `data retrieval` request. You always have to make sure to first **import the cdsapi** and define a `cdsapi.Client()` before you are able to execute an `API request`. You can use the [web interface](https://ads.atmosphere.copernicus.eu/cdsapp#!/search?type=dataset) to browse through the datasets. At the end of the `Download form`, there is the option to choose `Show API request`. If you click this button, the `API request` appears (see example below), which you can copy paste into your coding workflow.\n",
    "\n",
    "**Please note:** you have to accept the Terms and Conditions of the dataset through the ADS web interface before you are able to request data via the CDS API. \n",
    "\n",
    "<br>\n",
    "\n",
    "![logo](./img/ads_api_request.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>NOTE</b>: <br>\n",
    "    For data originating from ECMWF's Meteorological and Archival System (MARS), data is stored on a grid with longitudes from 0 to 360 degrees. It can be reprojected to a regular geographic latitude-longitude grid, by setting the keyword argument <code>area</code> and <code>grid</code>. Per default, data is retrieved in <code>GRIB</code>. If you wish to retrieve the data in <code>netCDF</code>, you have to specify it by using the keyword argument <code>format</code>.</div>\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Further resources\n",
    "\n",
    "* [Atmosphere Data Store (ADS) documentation](https://confluence.ecmwf.int/x/fMQMD)\n",
    "* [Common Error Messages for CDS Requests](https://confluence.ecmwf.int/x/RKOpD)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Example: CAMS data request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: CAMS global atmospheric composition forecasts - Organic matter aerosol optical depth at 550 nm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Data used in [*Fire Monitoring* case study](./05%Fire%Monitoring.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CDS API name: `cams-global-atmospheric-composition-forecasts`\n",
    "\n",
    "> - Variable: `organic_matter_aerosol_optical_depth_550nm`\n",
    "> - Date: `2021-08-01/2021-08-08`\n",
    "> - Time: `00:00`\n",
    "> - Leadtime hour: `[0, 6, 12, 18, 24, 30, 36, 42, 48]`\n",
    "> - Type: `forecast`\n",
    "> - Area: `[80, -150, 25, -50]`\n",
    "> - Format: `netcdf_zip`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cdsapi\n",
    "c = cdsapi.Client(url=URL, key=KEY)\n",
    "c.retrieve(\n",
    "    'cams-global-atmospheric-composition-forecasts',\n",
    "    {\n",
    "        'date': '2021-08-01/2021-08-08',\n",
    "        'type': 'forecast',\n",
    "        'format': 'netcdf_zip',\n",
    "        'variable': 'organic_matter_aerosol_optical_depth_550nm',\n",
    "        'time': '00:00',\n",
    "        'leadtime_hour': [\n",
    "            '0', '12', '18',\n",
    "            '6',\n",
    "        ],\n",
    "        'area': [\n",
    "            80, -150, 25,\n",
    "            -50,\n",
    "        ],\n",
    "    },\n",
    "    './2021-08_NAmerica_organicMatterAOD.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p></p>\n",
    "<span style='float:right'><p style=\\\"text-align:right;\\\">This project is licensed under <a href=\"./LICENSE\">APACHE License 2.0</a>. | <a href=\\\"https://github.com/ecmwf-projects/copernicus-training\">View on GitHub</a></span>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}