API

pyncei.bot

Tools to access data from NOAA’s Climate Data Online Web Services v2 API

class pyncei.bot.NCEIBot(token, wait=0.2, cache_name=None, **cache_kwargs)[source]

Bases: object

Contains functions to request data from the NCEI web services

wait

time in seconds between requests. NCEI allows a maximum of five queries per second.

Type

float

validate_params

whether to validate query parameters before making a GET request. Defaults to False.

Type

bool

max_retries

number of times to retry requests that fail because of temporary connectivity or server lapses. Retries use an exponential backoff. Defaults to 12.

Type

int

The get functions described below use a common set of keyword arguments. The sortorder, limit, offset, and max arguments can be used in any get function; other keywords vary by endpoint. Most values appear to be case-sensitive. Query validation, if enabled, should capture most but not all case errors.

Parameters

  • datasetid (str or list) – the id or name of a NCEI dataset. Multiple values allowed for most functions. Examples: GHCND; PRECIP_HLY; Weather Radar (Level III).

  • datacategoryid (str or list) – the id or name of a NCEI data category. Data categories are broader than data types. Multiple values allowed. Examples: TEMP, WXTYPE, Degree Days.

  • datatypeid (str or list) – the id or name of a data type. Multiple values allowed. Examples: TMIN; SNOW; Long-term averages of fall growing degree days with base 70F.

  • locationid (str or list) – the id or name of a location. Multiple values allowed. If a name is given, the script will try to map it to an id. Examples: Maryland; FIPS:24; ZIP:20003; London, UK.

  • stationid (str or list) – the id of name of a station in the NCEI database. Multiple values allowed. Examples: COOP:010957.

  • startdate (str or datetime) – the earliest date available

  • enddate (str or datetime) – the latest date available

  • sortfield (str) – field by which to sort the query results. Available sort fields vary by endpoint.

  • sortorder (str) – specifies whether sort is ascending or descending. Must be ‘asc’ or ‘desc’.

  • limit (int) – number of records to return per query

  • offset (int) – index of the first record to return

  • max (int) – maximum number of records to return. Not part of the API.

__init__(token, wait=0.2, cache_name=None, **cache_kwargs)[source]

Initializes NCEIBot object

Parameters

  • token (str) – NCEI token

  • wait (float or int) – time in seconds to wait between requests

  • cache_name (str) – path to cache

  • cache_kwargs – any keyword argument accepted by requests_cache.CachedSession

get_data(**kwargs)[source]

Retrieves historical climate data matching the given parameters

See NCEIBot for more details about each keyword argument.

Parameters

  • datasetid (str) – Required. Only one value allowed.

  • startdate (str or datetime) – Required. Returned stations will have data for the specified dataset/type from on or after this date.

  • enddate (str or datetime) – Required. Returned stations will have data for the specified dataset/type from on or before this date.

  • datatypeid (str or list) – Optional

  • locationid (str or list) – Optional

  • stationid (str or list) – Optional

  • units (str) – Optional. One of ‘standard’ or ‘metric’.

  • sortfield (str) – Optional. If provided, must be one of ‘datatype’, ‘date’, or ‘station’.

  • sortorder (str) – Optional

  • limit (int) – Optional

  • offset (int) – Optional

  • max (int) – Optional

Returns

List of dicts containing historical weather data

get_datasets(datasetid=None, **kwargs)[source]

Returns data from the NCEI dataset endpoint

See NCEIBot for more details about each keyword argument.

Parameters

  • datasetid (str) – a single dataset to return information about. Optional. The kwargs are ignored if this is provided.

  • datatypeid (str or list) – Optional

  • locationid (str or list) – Optional

  • stationid (str or list) – Optional

  • sortfield (str) – Optional. If provided, must be one of ‘id’, ‘name’, ‘mindate’, ‘maxdate’, or ‘datacoverage’.

  • sortorder (str) – Optional

  • limit (int) – Optional

  • offset (int) – Optional

  • max (int) – Optional

Returns

List of dicts containing metadata for all matching datasets

get_data_categories(datacategoryid=None, **kwargs)[source]

Returns codes and labels for NCDI data categories

See NCEIBot for more details about each keyword argument.

Parameters

  • datacategoryid (str) – a single data category to return information about. Optional. The kwargs are ignored if this is provided.

  • datasetid (str or list) – Optional

  • locationid (str or list) – Optional

  • stationid (str or list) – Optional

  • startdate (str or datetime) – Optional

  • enddate (str or datetime) – Optional

  • sortfield (str) – Optional. If provided, must be one of ‘id’, ‘name’, ‘mindate’, ‘maxdate’, or ‘datacoverage’.

  • sortorder (str) – Optional

  • limit (int) – Optional

  • offset (int) – Optional

  • max (int) – Optional

Returns

List of dicts containing metadata for all matching data categories

get_data_types(datatypeid=None, **kwargs)[source]

Returns information about NCEI data categories

See NCEIBot for more details about each keyword argument.

Parameters

  • datatypeid (str) – a single data type to return information about. Optional. The kwargs are ignored if this is provided.

  • datasetid (str or list) – Optional

  • locationid (str or list) – Optional

  • stationid (str or list) – Optional

  • datacategoryid (str or list) – Optional

  • startdate (str or datetime) – Optional

  • enddate (str or datetime) – Optional

  • sortfield (str) – Optional. If provided, must be one of ‘id’, ‘name’, ‘mindate’, ‘maxdate’, or ‘datacoverage’.

  • sortorder (str) – Optional

  • limit (int) – Optional

  • offset (int) – Optional

  • max (int) – Optional

Returns

List of dicts containing metadata for all matching data types

get_location_categories(locationcategoryid=None, **kwargs)[source]

Returns information about NCEI location categories

See NCEIBot for more details about each keyword argument.

Parameters

  • locationcategoryid (str) – a single location category to return information about. Optional. The kwargs are ignored if this is provided.

  • datasetid (str or list) – Optional

  • sortfield (str) – Optional. If provided, must be one of ‘id’ or ‘name’.

  • sortorder (str) – Optional

  • limit (int) – Optional

  • offset (int) – Optional

  • max (int) – Optional

Returns

List of dicts containing metadata about location categories

get_locations(locationid=None, **kwargs)[source]

Returns metadata for locations matching the given parameters

See NCEIBot for more details about each keyword argument.

Parameters

  • locationid (str) – a single location to return information about. Optional. The kwargs are ignored if this is provided.

  • datasetid (str or list) – Optional

  • locationcategoryid (str or list) – Optional

  • datacategoryid (str or list) – Optional

  • sortfield (str) – Optional. If provided, must be one of ‘id’, ‘name’, ‘mindate’, ‘maxdate’, or ‘datacoverage’.

  • sortorder (str) – Optional

  • limit (int) – Optional

  • offset (int) – Optional

  • max (int) – Optional

Returns

List of dicts containing metadata for all matching locations

get_stations(stationid=None, **kwargs)[source]

Returns metadata for stations matching the given parameters

See NCEIBot for more details about each keyword argument.

Parameters

  • stationid (str) – a single station to return information about. Optional. The kwargs are ignored if this is provided.

  • datasetid (str or list) – Optional

  • locationid (str or list) – Optional

  • datacategoryid (str or list) – Optional

  • datatypeid (str or list) – Optional

  • extent (str or iterable) – comma-delimited bounding box of form ‘min_lat, min_lng, max_lat, max_lng’ or equivalent iterable. Optional.

  • sortfield (str) – Optional. If provided, must be one of ‘id’, ‘name’, ‘mindate’, ‘maxdate’, or ‘datacoverage’.

  • sortorder (str) – Optional

  • limit (int) – Optional

  • offset (int) – Optional

  • max (int) – Optional

Returns

List of dicts containing metadata for all matching stations

find_ids(term=None, endpoints=None)[source]

Find key terms that match the search string for the given endpoints

Parameters

  • term (str) – the term to search for. If None, returns a list of all available terms for the specified endpoint(s).

  • endpoints (str or list) – name of one or more NCEI endpoints

Returns

List of (endpoint, id, name) for matching key terms from the specified endpoint

refresh_lookups(keys=None)[source]

Update the csv files used to populate the endpoint lookups

Parameters

keys (list) – list of endpoints to populate. If empty, everything but stations will be populated.

Returns

None

class pyncei.bot.NCEIResponse(iterable=(), /)[source]

Bases: list

Wraps results of one or more calls to the NCEI API

Extends list. Each response is stored as an entry in the list.

key_order = ['id', 'uid', 'name', 'station', 'latitude', 'longitude', 'elevation', 'elevationUnit', 'datacoverage', 'date', 'mindate', 'maxdate', 'datatype', 'attributes', 'value', 'url', 'retrieved']

list used to order the keys in the NCEI data

date_formats = {'date': '%Y-%m-%dT%H:%M:%S', 'maxdate': '%Y-%m-%d', 'mindate': '%Y-%m-%d', 'retrieved': '%Y-%m-%dT%H:%M:%S'}

dict mapping NCEI fields to date formats

values()[source]

Gets the results from all responses

Returns

generator of dicts

first()[source]

Gets the first result from the compiled responses

Returns

dict

count()[source]

Counts the number of results that have been returned

Returns

number of records returned as int

total()[source]

Counts the total number of results available for all URLs

Returns

total number of records matching the responses as int

to_csv(path)[source]

Writes data to a CSV

Parameters

path (str) – path to csv

to_dataframe()[source]

Writes data to a dataframe

Returns

pandas.DataFrame or geopandas.GeoDataFrame if geopandas is installed and the responses include coordinates