How to calculate downwelling spectral radiances

[dlebauer]

Currently the spectral radiances are uncalibrated, and provided in the environmental logger as:

    "spectrometer": {
      "maxFixedIntensity": "16383",
      "integration time in ?s": "5000",
      "band": {
        "wavelength": 337.70483,
        "spectrum": 1500.0
      },
      "band": {
        "wavelength": 338.16013791719934,
        "spectrum": 1500.0
      },
      "band": {
        "wavelength": 338.61548740418232,
        "spectrum": 1503.0
      },
      "band": {
        "wavelength": 339.07087845402685,
        "spectrum": 1500.0
      }, ...

from
2016-04-13_00-38-15_environmentlogger.zip

Calibration files are in EnvironmentLogger/CalibrationData/
Calibrations.zip

The output of the spectrometer is 'raw' counts.

@TinoDornbusch in #26 you wrote

You need to use the attached calibration files to convert it to units of µW m-2 s-1. Careful you need to take the bandwidth of the chip into account (0.4nm) if you want to convert to µmol m-2 s-1.

Could you or @markus-radermacher-lemnatec please clarify, and provide an equation for converting the information in the calibration files to reflectances?

[TinoDornbusch]

The spectrometer output is irradiance. Which units do you want? W m-2 s-1?

[czender]

My understanding is we will multiply the calibration times the count to obtain the irradiance in uW m-2 s-1. We are interested in that, and in spectral irradiance, i.e., uW m-2 m-1 s-1 where the additional "m-1" is per unit wavelength. It's unclear to me how to obtain this. Do we divide by the bandwidth = 0.4e-9 m? That would imply that the reported irradiance only accounts for the energy in the 0.4 nm bandpass, which is fine. And with the spectral irradiance we can integrate to obtain broadband and PAR. Please confirm if this is the case and if not, precisely what to do...
[NB: noticed 20160509 that I inadvertently added an extra factor of s-1 to the units in this post, and it persists in the next few messages. It should be ignored as it is folded into the W, i.e., W = J s-1. Sorry for any confusion this causes.]

[TinoDornbusch]

Irradiance in µW m-2 s-1 nm-1 = count * Joule/count / area / integration time

count //spectrometer output
Joule/count //table in calibration file
area = pi/4 *(3900µm/10^6 µm/m)^2; //this is the area of the fibre optics the light
integration time = 5000µs/10^6 µs/s //spectrometer output

[TinoDornbusch]

Yes right......... µW m-2 s-1 nm-1 you need to divide by 0.4nm (average bandwidth).

[czender]

Do you have a table of the actual bandwidth per channel? Or should we effective bandwidth of each channel is 0.4 nm? We will treat these bandpasses as effective square-wave (boxcar) bandwidths unless you say otherwise.

[dlebauer]

Current plan is for data products to follow CF conventions, as described in #14. CF provides these variables. The units they have include steradian. Will this be required to interpret the downwelling flux?

• downwelling_radiative_flux_per_unit_wavelength_in_air
  • alias: downwelling_spectral_radiative_flux_in_air
  • canonical_units: W m-2 m-1
  • description: Downwelling radiation is radiation from above. It does not mean "net downward". When thought of as being incident on a surface, a radiative flux is sometimes called "irradiance". In addition, it is identical with the quantity measured by a cosine-collector light-meter and sometimes called "vector irradiance". In accordance with common usage in geophysical disciplines, "flux" implies per unit area, called "flux density" in physics. A coordinate variable for radiation wavelength should be given the standard name radiation_wavelength.

[dlebauer]

@czender if I understand correctly the boxcar being integrating over [&lambda - 0.2, &lambda + 0.2]. That seems a reasonable assumption to start with - would such an assumption about the shape of this function introduce a meaningful level of error? (i.e. I don't fully understand, but the maximum flux of a boxcar would lower than a normal distribution).

@serbinsh and @ashiklom have suggested that we need FWHM for each band - is it possible that this is what 0.4um means? Does the manufacturer provide further details on the interpretation?

[czender]

radiance, not irradiance, has sterradian in the units. irradiance is radiance integrated over angles and multiplied by a cosine factor. my understanding is that the sensor essentially measures irradiance (at least that is what it is calibrated to return). so the docs may need to change.

i asked, and @TinoDornbusch will tell us what the 0.4 nm really means. it sounds like it is an effective squarewave bandpass, not FWHM.

[TinoDornbusch]

@czender Thanks for explaining the difference between radiance and irradiance.

I ask Ocean Optics for some clarification of that 0.4nm bandwidth. Nothing written in their documentation.

Note that the spectrometer has a cosine-corrected integrator. Hence we measure irradiance.

[dlebauer]

@czender raises the issue that the hyperspectral sensor measuring downwelling spectral radiances does not cover the range of the SWIR database.

He suggested that the key source of variability would be column water vapor (O 10%) , we could use near IR absorbance model the solar spectral flux in the longer wavebands (> 800).

My suggestion was that we could use the white reflector to calibrate an empirical model that predicts the spectral flux for all bands based on what we can observe (but is this double dipping if we also use the white reflector for calibration?). @ashiklom and @serbinsh feedback appreciated.

[TinoDornbusch]

@czender I asked Ocean Optics support. The bandwidth of the dispersion is NOT constant 0.4nm. To compute the width you need subtract adjacent center wavelengths.

To compute W m-2 you need to divide by this value, which roughly 0.4 nm.

[dlebauer]

@TinoDornbusch this is not clear to me. Could you please write out a formula? For example, if we have wavelengths

id	wavelength	bandwidth
1	337.70483
2	338.16013791719934
3	338.61548740418232
4	339.07087845402685

What would be the bandwidth for each of these wavelengths?

From what you say, it would be bandwidth_2 = wavelength_3 - wavelength_1 ?
Or would it be (wavelength_3 - wavelength_1)/2 ? (which is closer to 0.4)?

Do you have technical documentation from Ocean Optics?

[czender]

@FlyingWithJerome please change the bandwith wvl_dlt from scalar constant 0.4 nm to variable bandwith as indicated above, i.e., as difference between midpoints of adjacent center wavelengths. Maybe this will solve the issue we noticed yesterday, which is that 0.4 nm gives unrealistic spectral flux peaking toward UV rather than visible wavelengths ([http://dust.ess.uci.edu/tmp/flx_spc_dwn.jpg]). I will describe this during telecon tomorrow.

[TinoDornbusch]

@dlebauer We have no technical documentation of Ocean Optics. I ask for it.

[TinoDornbusch]

@dlebauer @czender Simply making the difference between adjacent center wavelength. This came from Ocean Optics:

1 337.7048 0.45530792
2 338.1601 0.45534949
3 338.6155 0.45539105
4 339.0709 .
.
last pixel copy of second to last

Delta 1 = WL2-WL1
Delta 2 = WL3-WL2
.
.
.
Last delta = second to last delta

Rounding to 3 digits is more than sufficient to achieve good accuracy.

[czender]

@TinoDornbusch We are having some difficulty obtaining reasonable numbers. Above you say

Irradiance in µW m-2 s-1 nm-1 = count * Joule/count / area / integration time

Are the numbers in the calibration file "joules/count", or do they include the area and integration time, so that they are "joules/count/area/integration time"? We have been assuming the latter, and so have not applied area and integration time corrections ourselves.

[TinoDornbusch]

The computation is correct. We have got a problem to solve. The dark reference. We need to collect a dark reference with closed sensor cap at regular intervals.

THe formula is Irradiance in µW m-2 = (count-darkref) * Joule/count / area / integration time / bandwidth

That's why your values do not make sense. At the moment we have no other solution that someone going up that gantry and covering the sensor cap at regular intervals.

I am going to ask Ocean optics how often one needs to do that dark reference. I am afraid that the chip is quite temperature sensitive and we need to do that very often. A manual solution is not applicable.

[czender]

@TinoDornbusch I think you neglected a factor of m-1:
µW m-2 m-1 = (count-darkref) * µJoule/count / area / integration time / bandwidth
Without measured spectral surface insolation, how will we calibrate the hyperspectral imagery into reflectance? The only viable paths are to "fix" the environmental logger calibration (which still won't solve the issue for SWIR calibration) or use a model instead.

[czender]

The broadband flux computed as I understand your instructions, is O(10^12) too big. See flx_dwn variable here. While the darkref count will reduce this, it will not eliminate it. There must be a units error or missing factor somewhere.

[TinoDornbusch]

I made a test myself and got 15 kW m-2 at 0:40AM. Off scale by serveal orders of magnitude. The oceanoptics Support confirmed the proposed formula. I suggest to ask Ocean optics directly for clarification:

Oliver Lischtschenko oliver.lischtschenko@oceanoptics.com

[czender]

I will be on travel this week and unable to pursue this further until week of 5/31. In the meantime, I encourage others to prioritize flux calibration with Ocean Optics and/or gantry system integrators.

[dlebauer]

@TinoDornbusch would the Ocean Optics sea breeze software help in generating calibrated output from this sensor?

[dlebauer]

@czender I am still waiting on a reply from Oliver but found this page on how to calibrate the sensor http://oceanoptics.com/faq/order-operations-spectral-calculations/

[TinoDornbusch]

I put a black cap on the spectrometer today (7.6.201) around 4PM to measure dark reference over a period of min 24h to see if there are diurnal variations. Lets hope this is quite stable since the spectrometer is in a climatized box. Let me know whether you computed values make more sense now after subtraction of dark current.

[ghost]

@czender - can this issue be closed?

[czender]

Not yet. Last time @FlyingWithJerome and I checked the irradiance was 10^12 too big. We never looked at whether subtracting the dark current measured on 20160607 improved the irradiance retrieved from the EnvironmentalLogger. We will try to look again next week. @TinoDornbusch do you now get reasonable values for irradiance when correcting for dark current?

[TinoDornbusch]

@czender I did not have time to make the computation. In other words, I will look at it asap.

[solmazhajmohammadi]

To conduct an absolute irradiance measurement, it is necessary to have the following:

• S: Sample spectrum (counts per nanometer)
• D: Dark spectrum (counts per nanometer, with the same integration time, corrections, and smoothing as sample)
• C: Calibration (represented in micro-Joules per count)
• T: Integration time (represented here in seconds)
• A: Collection area (represented in square meters) unless the light source is entirely inside an integrating sphere
• dL: The wavelength spread (how many nanometers a given pixel represents)

Absolute irradiance (I) is computed as follows. Below, the subscript P will indicate a particular pixel for I, dL, S, D, and C. Thus, Sp refers to pixel P of the sample spectrum. Ip = (Sp - Dp) * Cp / (T * A * dLp)

Ip = (Sp - Dp) * Cp / (T * A * dLp)
dL = [L(p + 1) - L(p - 1)] / 2

Rewrite W as J/sec: W/m2/nm = J/m2/sec/nm

I checked the data with University of Oregon Solar radiation monitoring laboratory data, 0.5-2 W/m^2/nm seems to be a normal range for 400-800 nm (of course depends on when during the day we are looking at the data).
Dark measurement data is missing in globus, I upload them here:
https://drive.google.com/drive/folders/0Bz72iLVLzGKQQzdWcS1HY2xWT28?usp=sharing

(if the link is not opening, just copy and paste it in the url)

Please let me know if you are not getting similar results, I can check the code

[czender]

@FlyingWithJerome and @hmb1 pointing you to this thread. Jerome, please try to reconstruct these results with the environmental logger calculator.

[FlyingWithJerome]

@solmazhajmohammadi
Hi Solmaz, thank you for your data and algorithm. I just downloaded these JSONs from your Google drive.
Is there a variable called "dark spectrum?" Or it has been marked as something else? They seem to be the same as the older JSONs for me.

[TinoDornbusch]

Introduction to Environmental Biophysics says Iradiance is 500 W m-2 or 2100 µmol m-2 s-1 integrated from 400-700 nm

[solmazhajmohammadi]

@TinoDornbusch the graph shows the Spectral irradiance which is a function of photon wavelength.
And it's true what you mentioned. Total Irradiance can be computed by integrating the spectral irradiance over all wavelengths. For the sample that I checked was 450 W/m^2
@FlyingWithJerome Dark measurment has been done on June 6th from 4pm to June 7th 10am. You can find the data here:
https://drive.google.com/drive/folders/0Bz72iLVLzGKQQzdWcS1HY2xWT28?usp=sharing

[solmazhajmohammadi]

@FlyingWithJerome no specific name, You just dont see any changes in the spectrum (dark current is something between 1493-1507 for all the bands)

[dlebauer]

@solmazhajmohammadi @FlyingWithJerome is the dark spectrum something fairly constant that should be stored along with other more static calibration data (like RSR curves)? If so, could you send the file to @craig-willis so he can add it?

[czender]

@FlyingWithJerome since it requires major effort to perform a dark count calibration these data are likely to remain static for some time so please place directly in the environmental logger calibration routines.

[solmazhajmohammadi]

@dlebauer as long as we dont make any changes in the integration time, we can use the same dark measurement. I'll send the mean dark current measurement to @craig-willis today.

[solmazhajmohammadi]

Mean dark current values:

Spectrometer_DarkMeasurments.xlsx

[solmazhajmohammadi]

@czender @FlyingWithJerome

It was nice talking to you today and thank you for your feedbacks.
I have plotted the raw spectrum output (unit : count) in the following figure:

I chose some random days and plotted them, for all of them we have a pick in the visible range.

This is a noisy sample:

You can see a spike in the plot but this is just because the collected data and dark current is noisy for the wavelength below 400nm.
Here is the data from June 6 at 9:09 am:

Common pitfall: remember to divide out the result to the wavelength spread

[dlebauer]

@solmazhajmohammadi sorry I don't know all of the context, but are you saying that the time field under gantry_system_variable_metadata is provided in central time? I now realize that this is not provided according to ISO 8607 as discussed in #2 and #25; it is too late to change this field but perhaps adding a standard_time or at least a timezone field would be useful.

[solmazhajmohammadi]

@dlebauer Sorry, I double-checked... ,y bad
It is stored at local time.
I corrected the comment

[czender]

Thanks @solmazhajmohammadi Your plots look reasonable and have the expected spectral features. @FlyingWithJerome and I will make sure we get similar/identical results in the workflow.

[FlyingWithJerome]

@solmazhajmohammadi We double check the algorithms and data and yes, what we are facing now is just the plotting issue; the data themselves are OK.
BTW, Did you have the result on the integration from 400 - 700nm period?

[solmazhajmohammadi]

For the data on June 6th, I got 310.4671 W/m^2
I plotted some sample here with the integration result using the data from 2016-07-29

[ghost]

@czender can this be closed?

[czender]

Yes, indeedy.