Skip to content

Combine read_nsrdb_psm4 and parse_nsrdb_psm4 #2445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Combine read_nsrdb_psm4 and parse_nsrdb_psm4 #2445

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1835204
psm4: combine parse_ and read_ into one function
kandersolar Apr 23, 2025
e9c09d4
lint
kandersolar Apr 23, 2025
89e352b
move metadata parsing out of context manager
kandersolar Apr 23, 2025
1df8af3
fix see also
kandersolar Apr 23, 2025
37e71da
make and test reusable function in pvlib.tools
kandersolar Apr 23, 2025
f462d86
lint
kandersolar Apr 23, 2025
72313ea
Apply suggestions from code review
kandersolar Apr 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/sphinx/source/reference/iotools.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ of sources and file formats relevant to solar energy modeling.
iotools.get_nsrdb_psm4_conus
iotools.get_nsrdb_psm4_full_disc
iotools.read_nsrdb_psm4
iotools.parse_nsrdb_psm4
iotools.get_psm3
iotools.read_psm3
iotools.parse_psm3
Expand Down
6 changes: 3 additions & 3 deletions docs/sphinx/source/whatsnew/v0.12.1.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ Enhancements
:py:func:`~pvlib.iotools.get_nsrdb_psm4_aggregated`,
:py:func:`~pvlib.iotools.get_nsrdb_psm4_tmy`,
:py:func:`~pvlib.iotools.get_nsrdb_psm4_conus`,
:py:func:`~pvlib.iotools.get_nsrdb_psm4_full_disc`,
:py:func:`~pvlib.iotools.read_nsrdb_psm4`, and
:py:func:`~pvlib.iotools.parse_nsrdb_psm4`. (:issue:`2326`, :pull:`2378`)
:py:func:`~pvlib.iotools.get_nsrdb_psm4_full_disc`, and
:py:func:`~pvlib.iotools.read_nsrdb_psm4`. (:issue:`2326`, :pull:`2378`,
:pull:`2445`)

Documentation
~~~~~~~~~~~~~
Expand Down
1 change: 0 additions & 1 deletion pvlib/iotools/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@
from pvlib.iotools.psm4 import get_nsrdb_psm4_conus # noqa: F401
from pvlib.iotools.psm4 import get_nsrdb_psm4_full_disc # noqa: F401
from pvlib.iotools.psm4 import read_nsrdb_psm4 # noqa: F401
from pvlib.iotools.psm4 import parse_nsrdb_psm4 # noqa: F401
from pvlib.iotools.pvgis import get_pvgis_tmy, read_pvgis_tmy # noqa: F401
from pvlib.iotools.pvgis import read_pvgis_hourly # noqa: F401
from pvlib.iotools.pvgis import get_pvgis_hourly # noqa: F401
Expand Down
153 changes: 56 additions & 97 deletions pvlib/iotools/psm4.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import requests
import pandas as pd
from json import JSONDecodeError
from pvlib import tools

NSRDB_API_BASE = "https://developer.nrel.gov/api/nsrdb/v2/solar/"
PSM4_AGG_ENDPOINT = "nsrdb-GOES-aggregated-v4-0-0-download.csv"
Expand Down Expand Up @@ -82,7 +83,7 @@ def get_nsrdb_psm4_aggregated(latitude, longitude, api_key, email,
Aggregated v4 API.

The NSRDB is described in [1]_ and the PSM4 NSRDB GOES Aggregated v4 API is
described in [2]_,.
described in [2]_.

Parameters
----------
Expand Down Expand Up @@ -132,7 +133,7 @@ def get_nsrdb_psm4_aggregated(latitude, longitude, api_key, email,
timeseries data from NREL PSM4
metadata : dict
metadata from NREL PSM4 about the record, see
:func:`pvlib.iotools.parse_nsrdb_psm4` for fields
:func:`pvlib.iotools.read_nsrdb_psm4` for fields

Raises
------
Expand All @@ -151,19 +152,15 @@ def get_nsrdb_psm4_aggregated(latitude, longitude, api_key, email,
result in rejected requests.

.. warning:: PSM4 is limited to data found in the NSRDB, please consult
the references below for locations with available data. Additionally,
querying data with < 30-minute resolution uses a different API endpoint
with fewer available fields (see [4]_).
the references below for locations with available data.

See Also
--------
pvlib.iotools.get_nsrdb_psm4_tmy, pvlib.iotools.get_nsrdb_psm4_conus,
pvlib.iotools.get_nsrdb_psm4_full_disc, pvlib.iotools.read_nsrdb_psm4,
pvlib.iotools.parse_nsrdb_psm4
pvlib.iotools.get_nsrdb_psm4_full_disc, pvlib.iotools.read_nsrdb_psm4

References
----------

.. [1] `NREL National Solar Radiation Database (NSRDB)
<https://nsrdb.nrel.gov/>`_
.. [2] `NSRDB GOES Aggregated V4.0.0
Expand Down Expand Up @@ -213,7 +210,7 @@ def get_nsrdb_psm4_aggregated(latitude, longitude, api_key, email,
# the CSV is in the response content as a UTF-8 bytestring
# to use pandas we need to create a file buffer from the response
fbuf = io.StringIO(response.content.decode('utf-8'))
return parse_nsrdb_psm4(fbuf, map_variables)
return read_nsrdb_psm4(fbuf, map_variables)


def get_nsrdb_psm4_tmy(latitude, longitude, api_key, email, year='tmy',
Expand All @@ -225,7 +222,7 @@ def get_nsrdb_psm4_tmy(latitude, longitude, api_key, email, year='tmy',
TMY v4 API.

The NSRDB is described in [1]_ and the PSM4 NSRDB GOES TMY v4 API is
described in [2]_,.
described in [2]_.

Parameters
----------
Expand Down Expand Up @@ -276,7 +273,7 @@ def get_nsrdb_psm4_tmy(latitude, longitude, api_key, email, year='tmy',
timeseries data from NREL PSM4
metadata : dict
metadata from NREL PSM4 about the record, see
:func:`pvlib.iotools.parse_nsrdb_psm4` for fields
:func:`pvlib.iotools.read_nsrdb_psm4` for fields

Raises
------
Expand All @@ -295,19 +292,16 @@ def get_nsrdb_psm4_tmy(latitude, longitude, api_key, email, year='tmy',
result in rejected requests.

.. warning:: PSM4 is limited to data found in the NSRDB, please consult
the references below for locations with available data. Additionally,
querying data with < 30-minute resolution uses a different API endpoint
with fewer available fields (see [4]_).
the references below for locations with available data.

See Also
--------
pvlib.iotools.get_nsrdb_psm4_aggregated,
pvlib.iotools.get_nsrdb_psm4_conus, pvlib.iotools.get_nsrdb_psm4_full_disc,
pvlib.iotools.read_nsrdb_psm4,pvlib.iotools.parse_nsrdb_psm4
pvlib.iotools.read_nsrdb_psm4

References
----------

.. [1] `NREL National Solar Radiation Database (NSRDB)
<https://nsrdb.nrel.gov/>`_
.. [2] `NSRDB GOES Tmy V4.0.0
Expand Down Expand Up @@ -357,7 +351,7 @@ def get_nsrdb_psm4_tmy(latitude, longitude, api_key, email, year='tmy',
# the CSV is in the response content as a UTF-8 bytestring
# to use pandas we need to create a file buffer from the response
fbuf = io.StringIO(response.content.decode('utf-8'))
return parse_nsrdb_psm4(fbuf, map_variables)
return read_nsrdb_psm4(fbuf, map_variables)


def get_nsrdb_psm4_conus(latitude, longitude, api_key, email, year='2023',
Expand All @@ -369,7 +363,7 @@ def get_nsrdb_psm4_conus(latitude, longitude, api_key, email, year='2023',
v4 API.

The NSRDB is described in [1]_ and the PSM4 NSRDB GOES CONUS v4 API is
described in [2]_,.
described in [2]_.

Parameters
----------
Expand Down Expand Up @@ -418,7 +412,7 @@ def get_nsrdb_psm4_conus(latitude, longitude, api_key, email, year='2023',
timeseries data from NREL PSM4
metadata : dict
metadata from NREL PSM4 about the record, see
:func:`pvlib.iotools.parse_nsrdb_psm4` for fields
:func:`pvlib.iotools.read_nsrdb_psm4` for fields

Raises
------
Expand All @@ -437,19 +431,16 @@ def get_nsrdb_psm4_conus(latitude, longitude, api_key, email, year='2023',
result in rejected requests.

.. warning:: PSM4 is limited to data found in the NSRDB, please consult
the references below for locations with available data. Additionally,
querying data with < 30-minute resolution uses a different API endpoint
with fewer available fields (see [4]_).
the references below for locations with available data.

See Also
--------
pvlib.iotools.get_nsrdb_psm4_aggregated,
pvlib.iotools.get_nsrdb_psm4_tmy, pvlib.iotools.get_nsrdb_psm4_full_disc,
pvlib.iotools.read_nsrdb_psm4, pvlib.iotools.parse_nsrdb_psm4
pvlib.iotools.read_nsrdb_psm4

References
----------

.. [1] `NREL National Solar Radiation Database (NSRDB)
<https://nsrdb.nrel.gov/>`_
.. [2] `NSRDB GOES Conus V4.0.0
Expand Down Expand Up @@ -499,7 +490,7 @@ def get_nsrdb_psm4_conus(latitude, longitude, api_key, email, year='2023',
# the CSV is in the response content as a UTF-8 bytestring
# to use pandas we need to create a file buffer from the response
fbuf = io.StringIO(response.content.decode('utf-8'))
return parse_nsrdb_psm4(fbuf, map_variables)
return read_nsrdb_psm4(fbuf, map_variables)


def get_nsrdb_psm4_full_disc(latitude, longitude, api_key, email,
Expand All @@ -513,7 +504,7 @@ def get_nsrdb_psm4_full_disc(latitude, longitude, api_key, email,
Disc v4 API.

The NSRDB is described in [1]_ and the PSM4 NSRDB GOES Full Disc v4 API is
described in [2]_,.
described in [2]_.

Parameters
----------
Expand Down Expand Up @@ -563,7 +554,7 @@ def get_nsrdb_psm4_full_disc(latitude, longitude, api_key, email,
timeseries data from NREL PSM4
metadata : dict
metadata from NREL PSM4 about the record, see
:func:`pvlib.iotools.parse_nsrdb_psm4` for fields
:func:`pvlib.iotools.read_nsrdb_psm4` for fields

Raises
------
Expand All @@ -582,19 +573,16 @@ def get_nsrdb_psm4_full_disc(latitude, longitude, api_key, email,
result in rejected requests.

.. warning:: PSM4 is limited to data found in the NSRDB, please consult
the references below for locations with available data. Additionally,
querying data with < 30-minute resolution uses a different API endpoint
with fewer available fields (see [4]_).
the references below for locations with available data.

See Also
--------
pvlib.iotools.get_nsrdb_psm4_aggregated,
pvlib.iotools.get_nsrdb_psm4_tmy, pvlib.iotools.get_nsrdb_psm4_conus,
pvlib.iotools.read_nsrdb_psm4, pvlib.iotools.parse_nsrdb_psm4
pvlib.iotools.read_nsrdb_psm4

References
----------

.. [1] `NREL National Solar Radiation Database (NSRDB)
<https://nsrdb.nrel.gov/>`_
.. [2] `NSRDB GOES Full Disc V4.0.0
Expand Down Expand Up @@ -644,19 +632,19 @@ def get_nsrdb_psm4_full_disc(latitude, longitude, api_key, email,
# the CSV is in the response content as a UTF-8 bytestring
# to use pandas we need to create a file buffer from the response
fbuf = io.StringIO(response.content.decode('utf-8'))
return parse_nsrdb_psm4(fbuf, map_variables)
return read_nsrdb_psm4(fbuf, map_variables)


def parse_nsrdb_psm4(fbuf, map_variables=True):
def read_nsrdb_psm4(filename, map_variables=True):
"""
Parse an NSRDB PSM4 weather file (formatted as SAM CSV).
Read an NSRDB PSM4 weather file (formatted as SAM CSV).

The NSRDB is described in [1]_ and the SAM CSV format is described in [2]_.

Parameters
----------
fbuf: file-like object
File-like object containing data to read.
filename: str, path-like, or buffer
Filename or in-memory buffer of a file containing data to read.
map_variables: bool, default True
When true, renames columns of the Dataframe to pvlib variable names
where applicable. See variable :const:`VARIABLE_MAP`.
Expand Down Expand Up @@ -726,12 +714,19 @@ def parse_nsrdb_psm4(fbuf, map_variables=True):
Examples
--------
>>> # Read a local PSM4 file:
>>> df, metadata = iotools.read_nsrdb_psm4("data.csv") # doctest: +SKIP

>>> # Read a file object or an in-memory buffer:
>>> with open(filename, 'r') as f: # doctest: +SKIP
... df, metadata = iotools.parse_nsrdb_psm4(f) # doctest: +SKIP
... df, metadata = iotools.read_nsrdb_psm4(f) # doctest: +SKIP

See Also
--------
pvlib.iotools.read_nsrdb_psm4, pvlib.iotools.get_psm4
pvlib.iotools.get_nsrdb_psm4_aggregated
pvlib.iotools.get_nsrdb_psm4_tmy
pvlib.iotools.get_nsrdb_psm4_conus
pvlib.iotools.get_nsrdb_psm4_full_disc
pvlib.iotools.read_psm3

References
----------
Expand All @@ -740,34 +735,36 @@ def parse_nsrdb_psm4(fbuf, map_variables=True):
.. [2] `Standard Time Series Data File Format
<https://web.archive.org/web/20170207203107/https://sam.nrel.gov/sites/default/files/content/documents/pdf/wfcsv.pdf>`_
"""
# The first 2 lines of the response are headers with metadata
metadata_fields = fbuf.readline().split(',')
metadata_fields[-1] = metadata_fields[-1].strip() # strip trailing newline
metadata_values = fbuf.readline().split(',')
metadata_values[-1] = metadata_values[-1].strip() # strip trailing newline
with tools._file_context_manager(filename) as fbuf:
# The first 2 lines of the response are headers with metadata
metadata_fields = fbuf.readline().split(',')
metadata_values = fbuf.readline().split(',')
# get the column names so we can set the dtypes
columns = fbuf.readline().split(',')
columns[-1] = columns[-1].strip() # strip trailing newline
# Since the header has so many columns, excel saves blank cols in the
# data below the header lines.
columns = [col for col in columns if col != '']
dtypes = dict.fromkeys(columns, float)
dtypes.update({'Year': int, 'Month': int, 'Day': int, 'Hour': int,
'Minute': int, 'Cloud Type': int, 'Fill Flag': int})

data = pd.read_csv(
fbuf, header=None, names=columns, usecols=columns, dtype=dtypes,
delimiter=',', lineterminator='\n') # skip carriage returns \r

metadata_fields[-1] = metadata_fields[-1].strip() # trailing newline
metadata_values[-1] = metadata_values[-1].strip() # trailing newline
metadata = dict(zip(metadata_fields, metadata_values))
# the response is all strings, so set some metadata types to numbers
metadata['Local Time Zone'] = int(metadata['Local Time Zone'])
metadata['Time Zone'] = int(metadata['Time Zone'])
metadata['Latitude'] = float(metadata['Latitude'])
metadata['Longitude'] = float(metadata['Longitude'])
metadata['Elevation'] = int(metadata['Elevation'])
# get the column names so we can set the dtypes
columns = fbuf.readline().split(',')
columns[-1] = columns[-1].strip() # strip trailing newline
# Since the header has so many columns, excel saves blank cols in the
# data below the header lines.
columns = [col for col in columns if col != '']
dtypes = dict.fromkeys(columns, float) # all floats except datevec
dtypes.update(Year=int, Month=int, Day=int, Hour=int, Minute=int)
dtypes['Cloud Type'] = int
dtypes['Fill Flag'] = int
data = pd.read_csv(
fbuf, header=None, names=columns, usecols=columns, dtype=dtypes,
delimiter=',', lineterminator='\n') # skip carriage returns \r

# the response 1st 5 columns are a date vector, convert to datetime
dtidx = pd.to_datetime(
data[['Year', 'Month', 'Day', 'Hour', 'Minute']])
dtidx = pd.to_datetime(data[['Year', 'Month', 'Day', 'Hour', 'Minute']])
# in USA all timezones are integers
tz = 'Etc/GMT%+d' % -metadata['Time Zone']
data.index = pd.DatetimeIndex(dtidx).tz_localize(tz)
Expand All @@ -779,41 +776,3 @@ def parse_nsrdb_psm4(fbuf, map_variables=True):
metadata['altitude'] = metadata.pop('Elevation')

return data, metadata


def read_nsrdb_psm4(filename, map_variables=True):
"""
Read an NSRDB PSM4 weather file (formatted as SAM CSV).

The NSRDB is described in [1]_ and the SAM CSV format is described in [2]_.

Parameters
----------
filename: str or path-like
Filename of a file containing data to read.
map_variables: bool, default True
When true, renames columns of the Dataframe to pvlib variable names
where applicable. See variable :const:`VARIABLE_MAP`.

Returns
-------
data : pandas.DataFrame
timeseries data from NREL PSM4
metadata : dict
metadata from NREL PSM4 about the record, see
:func:`pvlib.iotools.parse_nsrdb_psm4` for fields

See Also
--------
pvlib.iotools.parse_nsrdb_psm4, pvlib.iotools.get_psm4

References
----------
.. [1] `NREL National Solar Radiation Database (NSRDB)
<https://nsrdb.nrel.gov/>`_
.. [2] `Standard Time Series Data File Format
<https://web.archive.org/web/20170207203107/https://sam.nrel.gov/sites/default/files/content/documents/pdf/wfcsv.pdf>`_
"""
with open(str(filename), 'r') as fbuf:
content = parse_nsrdb_psm4(fbuf, map_variables)
return content
Loading
Loading