Skip to content

A Python library that acts as a client to download, pre-process and post-process weather data. Friendly for users on VPN/PROXY connections.

License

Notifications You must be signed in to change notification settings

edrewitz/WxData

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

484 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

WxData

Alt text Alt text

Conda Recipe Conda Version Conda Platforms PyPI Anaconda-Server Badge Anaconda-Server Badge

DOI

Anaconda Downloads

Conda Downloads

PIP Downloads:

PyPI - Downloads

(C) Eric J. Drewitz 2025-2026

An open-source package that helps meteorologists and weather enthusiats download, pre-process and post-process various types of weather data.

This package only retrieves open-source publicly available weather data.

This package provides the following extra functionality compared to existing packages for downloading weather data:

How To Install

Copy and paste either command into your terminal or anaconda prompt:

Install via Anaconda

conda install wxdata

Install via pip

pip install wxdata

How To Update To The Latest Version

Copy and paste either command into your terminal or anaconda prompt:

Update via Anaconda

This is for users who initially installed WxData through Anaconda

conda update wxdata

Update via pip

This is for users who initially installed WxData through pip

pip install --upgrade wxdata

Important Compatibility Information

Python 3.14 is not compatible with the pip version of the eccodes library

Methods to fix eccodes compatibility with Python 3.14:

  1. Uninstall the pip version of WxData and install WxData via Anaconda

    Steps For Method 1

    1. pip uninstall wxdata
    2. conda install wxdata
  2. If the user is unable to use Anaconda as a package manager, the user must set up a new Python environment with the following specifications:

Specifications

Python >= 3.10 and Python <= 3.13

Python 3.10 is compatible.

Python 3.11 is compatible.

Python 3.12 is compatible.

Python 3.13 is compatible

Then pip install wxdata after the new Python environment is set up.

  1. Friendly for users working on VPN/PROXY connections.

    • Users input their PROXY IP address as a dictionary and pass it into the function to avoid SSL errors
      • If the user is on a VPN/PROXY Connection the following is needed:

                      proxies=None ---> proxies={
                                        'http':'http://url',
                                        'https':'https://url'
                                        }
        
                     [e.g. get_observed_sounding_data('nkx', proxies=proxies)]
        
        Alt text

    For more information on configuring proxies: https://requests.readthedocs.io/en/latest/user/advanced/#proxies

    • Some data access functions work on VPN/PROXY connections without needing to define VPN/PROXY settings:

      • METARs
      • NOAA Storm Prediction Center/National Weather Service Products
      • FEMS
    • Data access methods that users need to define VPN/PROXY IP addresses if using a VPN/PROXY connection:

      • Various Forecast Models
      • Observed Sounding Data from University of Wyoming
      • Real-Time Mesoscale Analysis
  2. Converts GRIB variable keys into variable keys that are in plain language.

    • (e.g. 'r2' ---> '2m_relative_humidity')
  3. Has a scanner that checks if the data files on your PC are up to date with those on the data server.

    • This is a safeguard to protect newer developers from getting temporary IP address bans from the various data servers.
    • Improves performance by preventing the potential of repetative downloading the same dataset.
  4. Preserves system memory via the following methods:

    • Clears out old data files before each new data download.
    • Optional setting clear_recycle_bin=True in all functions.
      • When clear_recycle_bin=True the computer's recycle/trash bin is cleared with each run of the script using any WxData function.
      • If a user wishes to not clear out their recycle bin set clear_recycle_bin=False.

WxData Examples

Regular Users

  1. Downloading METAR Data
  2. Downloading Observed Sounding Data
  3. Downloading the first 72 hours of the ECMWF IFS and ECMWF AIFS
  4. Downloading the GEFS members p01 and p02 for only Temperature
  5. Downloading the Real-Time Mesoscale Analysis (RTMA)
  6. Downloading the RAWS SIG Group Fuels Data and the NFDRS Forecast for the RAWS SIG Groups for the South Ops Geographic Area Coordination Center
  7. Downloading the SPC Convective Outlook for CONUS
  8. Downloading NWS Maximum Temperature Forecast for Hawaii
  9. Downloading the GFS0P25 then performing pixel and line queries on the data
  10. Downloading various datasets from the Applied Climate Information System (ACIS)
  11. Downloading AIGFS Data
  12. Downloading AIGEFS Data
  13. Downloading and plotting the Climate Prediction Center 6-10 Day Precipitation Outlook

Advanced Users

  1. Using the client module to download the latest HadCRUT5 Analysis netCDF file and open this dataset in xarray
  2. Downloading the GFS0P25 for temperature fields and using run_external_scripts() to post-process this GFS0P25 dataset in an external Python script

WxData Documentation

Global Forecast System (GFS)

  1. GFS0P25
  2. GFS0P25 SECONDARY PARAMETERS
  3. GFS0P50

AI Global Forecast System (AIGFS)

  1. AIGFS

Hybrid Global Ensemble Forecast System (HGEFS)

  1. HGEFS

Global Ensemble Forecast System (GEFS)

  1. GEFS0P50
  2. GEFS0P50 SECONDARY PARAMETERS
  3. GEFS0P25

AI Global Ensemble Forecast System (AIGEFS)

  1. AIGEFS Members (Pressure Parameters
  2. AIGEFS Members (Surface Parameters
  3. AIGEFS Ensemble Mean & Ensemble Spread

ECMWF Open Data

  1. ECMWF IFS
  2. ECMWF AIFS
  3. ECMWF High Resolution IFS
  4. ECMWF IFS Wave

Real-Time Mesoscale Analysis (RTMA)

  1. RTMA
  2. RTMA Comparison

NOAA Storm Prediction Center Outlooks/Climate Prediction Center Outlooks/National Weather Service Forecasts

  1. Get NDFD Grids
  2. Climate Prediction Center Outlooks

METAR Observations

  1. METAR Observations

FEMS RAWS Network

  1. Get Single Station RAWS Data
  2. Get Each SIG of RAWS Data for a Geographic Area Coordination Center
  3. Get NFDRS Forecast Data for Each SIG for a Geographic Area Coordination Center

Observed Atmospheric Soundings

  1. University Of Wyoming Soundings

GFS Post-Processing

  1. Primary GFS Post-Processing
  2. Secondary GFS Post-Processing

AIGFS Post-Processing

  1. AIGFS Post-Processing

GEFS Post-Processing

  1. Primary GEFS Post-Processing
  2. Secondary GEFS Post-Processing

AIGEFS Post-Processing

  1. AIGEFS Members Post-Processing
  2. AIGEFS Single Post-Processing

HGEFS Post-Processing

  1. HGEFS Post-Processing

ECMWF Post-Processing

  1. ECMWF IFS and ECMWF High Resolution IFS
  2. ECMWF AIFS
  3. ECMWF IFS Wave

Real-Time Mesoscale Analysis Post-Processing

  1. RTMA

xmACIS2 Climate Data

  1. xmACIS2 Client

Custom Gridded Data

  1. Gridded Data Client

Custom CSV Data

  1. CSV Data Client

Cyclic Points For Hemispheric Plots

  1. Cyclic Points

Shifting Longitude From 0 to 360 --> -180 to 180

  1. shift_longitude

Pixel Query

  1. pixel_query

Line Query

  1. line_query

Running External Python Scripts In Your Workflow

1 run_external_scripts

Importing Functions from WxData

     """
     This file hosts all of the functions in the WxData Python library that directly interact with the user. 
     
     (C) Eric J. Drewitz 2025-2026
     """
     
     
     """
     This section of functions are for users who want full wxdata functionality.
     
     These functions do the following:
     
     1) Scan for the latest available data. 
         - If the data on your local machine is not up to date, new data will download automatically.
         - If the data on your local machine is up to date, new data download is bypassed.
         - This is a safeguard that prevents excessive requests on the data servers.
         
     2) Builds the wxdata directory to store the weather data files. 
         - Scans for the directory branch and builds the branch if it does not exist. 
     
     3) Downloads the data.
         - Users can define their VPN/PROXY IP Address as a (dict) in their script and pass their
           VPN/PROXY IP address into the function to avoid SSL Certificate errors when requesting data.
         - Algorithm allows for up to 5 retries with a 30 second break between each retry to resolve connection
           interruptions while not overburdening the data servers. 
     
     4) Pre-processes the data via filename formatting and correctly filing in the wxdata directory. 
     
     5) Post-processing by doing the following tasks:
          - Remapping GRIB2 variable keys into plain language variable keys.
          - Fixing dataset build errors and grouping all variables together.
          - Transforms longitude from 0 to 360 degrees into -180 to 180 degrees.
          - Subsetting the data to the latitude/longitude boundaries specified by the user. 
          - Converting temperature from kelvin to units the user wants (default is Celsius).
          - Returning a post-processed xarray.array to the user. 
          
     6) Preserves system memory by doing the following:
          - Deleting old data files before each new download.
          - When clear_recycle_bin=True, the user's recycle bin is also cleared. 
     """
     
     # Global Forecast System (GFS)
     from wxdata.gfs.gfs import(
         gfs_0p25,
         gfs_0p25_secondary_parameters,
         gfs_0p50
     )
     
     # AI Global Forecast System (AIGFS)
     from wxdata.aigfs.aigfs import aigfs
     
     # Hybrid Global Ensemble Forecast System (HGEFS)
     from wxdata.hgefs.hgefs import hgefs_mean_spread
     
     # Global Ensemble Forecast System (GEFS)
     from wxdata.gefs.gefs import(
         gefs_0p50,
         gefs_0p50_secondary_parameters,
         gefs_0p25
     )
     
     # AI Global Ensemble Forecast System (AIGEFS)
     from wxdata.aigefs.aigefs import(
         aigefs_pressure_members,
         aigefs_surface_members,
         aigefs_single
     )
     
     # European Centre for Medium-Range Weather Forecasts (ECMWF)
     from wxdata.ecmwf.ecmwf import(
         ecmwf_ifs,
         ecmwf_aifs,
         ecmwf_ifs_high_res,
         ecmwf_ifs_wave
     )
     
     # FEMS RAWS Network
     from wxdata.fems.fems import(
         get_single_station_data,
         get_raws_sig_data,
         get_nfdrs_forecast_data
     )
     
     # Real-Time Mesoscale Analysis (RTMA)
     from wxdata.rtma.rtma import(
         rtma, 
         rtma_comparison
     )
     
     # NOAA 
     # Storm Prediction Center Outlooks
     # Climate Prediction Center Outlooks
     # National Weather Service Forecasts
     from wxdata.noaa.nws import(
         get_ndfd_grids,
         get_cpc_outlook
     )
     
     # Observed Upper-Air Soundings
     # (University of Wyoming Database)
     from wxdata.soundings.wyoming_soundings import get_observed_sounding_data
     
     # METAR Observational Data (From NOAA)
     from wxdata.metars.metar_obs import download_metar_data
     
     """
     This section hosts all the functions and modules that involve post-processing the data.
     These are the functions and modules that:
     
     1) Re-map the GRIB2 Variable Keys into Plain Language Keys
     2) Build the xarray.array of the various datasets. 
     
     """
     
     
     # Global Forecast System (GFS)
     import wxdata.post_processors.gfs_post_processing as gfs_post_processing
     
     # AI Global Forecast System (AIGFS)
     import wxdata.post_processors.aigfs_post_processing as aigfs_post_processing
     
     # Hybrid Global Ensemble Forecast System (HGEFS)
     import wxdata.post_processors.hgefs_post_processing as hgefs_post_processing
     
     # Global Ensemble Forecast System (GEFS)
     import wxdata.post_processors.gefs_post_processing as gefs_post_processing
     
     # AI Global Ensemble Forecast System (AIGEFS)
     import wxdata.post_processors.aigefs_post_processing as aigefs_post_processing
     
     # European Centre for Medium-Range Weather Forecasts (ECMWF)
     import wxdata.post_processors.ecmwf_post_processing as ecmwf_post_processing
     
     # Real-Time Mesoscale Analysis (RTMA)
     from wxdata.post_processors.rtma_post_processing import process_rtma_data
     
     
     """
     This section hosts the utility functions accessable to the user. 
     
     These functions provide helpful utilities when analyzing weather data. 
     
     Utility functions are geared towards the following types of users:
     
     1) Users who want to use their own scripts to download the data however, they
        would like to use the wxdata post-processing capabilities. 
        
     2) Users who want to make hemispheric graphics or any graphics where cyclic points
        resolve missing data along the prime meridian or international dateline. 
     """
     # WxData function using cartopy to make cyclic points
     # This is for users who wish to make graphics that cross the -180/180 degree longitude line
     # This is commonly used for Hemispheric graphics
     # Function that converts the longitude dimension in an xarray.array 
     # From 0 to 360 to -180 to 180
     from wxdata.utils.coords import(
         cyclic_point,
         shift_longitude
     )
     
     # Functions to pixel query and query pixels along a line between points A and B
     # Function to interpolate to n amount of points in between x and y values respectively
     from wxdata.utils.tools import(
         pixel_query,
         line_query,
         linear_anti_aliasing
     )
     
     """
     This section hosts the various data clients that retrieve various types of data.
     
     These clients can be easily configured to work on VPN/PROXY connections.
     """
     
     # These are the wxdata HTTPS Clients with full VPN/PROXY Support
     # Client List:
     #  - get_gridded_data()
     #  - get_csv_data()
     #  - get_xmacis_data()
     import wxdata.client.client as client
     
     # This function executes a list of Python scripts in the order the user lists them
     from wxdata.utils.scripts import run_external_scripts

Citations

MetPy: May, R. M., Goebbert, K. H., Thielen, J. E., Leeman, J. R., Camron, M. D., Bruick, Z., Bruning, E. C., Manser, R. P., Arms, S. C., and Marsh, P. T., 2022: MetPy: A Meteorological Python Library for Data Analysis and Visualization. Bull. Amer. Meteor. Soc., 103, E2273-E2284, https://doi.org/10.1175/BAMS-D-21-0125.1.

xarray: Hoyer, S., Hamman, J. (In revision). Xarray: N-D labeled arrays and datasets in Python. Journal of Open Research Software.

cartopy: Phil Elson, Elliott Sales de Andrade, Greg Lucas, Ryan May, Richard Hattersley, Ed Campbell, Andrew Dawson, Bill Little, Stephane Raynaud, scmc72, Alan D. Snow, Ruth Comer, Kevin Donkers, Byron Blay, Peter Killick, Nat Wilson, Patrick Peglar, lgolston, lbdreyer, … Chris Havlin. (2023). SciTools/cartopy: v0.22.0 (v0.22.0). Zenodo. https://doi.org/10.5281/zenodo.8216315

NumPy: Harris, C.R., Millman, K.J., van der Walt, S.J. et al. Array programming with NumPy. Nature 585, 357–362 (2020). DOI: 10.1038/s41586-020-2649-2. (Publisher link).

Pandas: Pandas: McKinney, W., & others. (2010). Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference (Vol. 445, pp. 51–56).

dask: Dask Development Team (2016). Dask: Library for dynamic task scheduling. URL http://dask.pydata.org

cfgrib: Author: ECMWF, Year: (2025), Title: cfgrib: A Python interface to map GRIB files to xarray, Source: https://github.com/ecmwf/cfgrib

requests: K. Reitz, "Requests: HTTP for Humans". Available: https://requests.readthedocs.io/.

shapeography: Eric J. Drewitz. (2026). edrewitz/shapeography: shapeography 1.0 Released (shapeography1.0). Zenodo. https://doi.org/10.5281/zenodo.18676845

geopandas: Kelsey Jordahl, Joris Van den Bossche, Martin Fleischmann, Jacob Wasserman, James McBride, Jeffrey Gerard, … François Leblanc. (2020, July 15). geopandas/geopandas: v0.8.1 (Version v0.8.1). Zenodo. http://doi.org/10.5281/zenodo.3946761

Data Sources

  1. National Oceanic and Atmospheric Administration/National Center for Environmental Prediction
  2. European Centre for Medium-Range Weather Forecasts
  3. University of Wyoming
  4. National Oceanic and Atmospheric Administration/National Weather Service
  5. National Oceanic and Atmospheric Administration/Aviation Weather Center
  6. National Oceanic and Atmospheric Administration/Climate Prediction Center
  7. Applied Climate Information System (ACIS)

About

A Python library that acts as a client to download, pre-process and post-process weather data. Friendly for users on VPN/PROXY connections.

Topics

Resources

License

Contributing

Stars

Watchers

Forks

Packages

No packages published

Languages