Add extractor for data from EnvironmentLogger --> geostreams database #252

dlebauer · 2017-02-02T16:47:16Z

The data are transferred outside the core lemnatec pipeline -> JD, where are these?
Otherwise the conversion should be similar.

Zodiase · 2017-02-02T17:13:51Z

@dlebauer Who would be the person that knows where the data are and what is the schema/format of them?

jdmaloney · 2017-02-02T19:30:11Z

@Zodiase If you have access to Roger or the Terraref Globus Endpoint, the files are located at:
/gpfs/largeblockFS/projects/arpae/terraref/sites/ua-mac/raw_data/weather --> from Roger perspective
or
/gpfs/largeblockFS/projects/arpae/terraref/sites/ua-mac/raw_data/weather --> Globus perspective

I'm not sure of schema/format, but that is where the files are if you want to look at them.

Zodiase · 2017-02-09T21:25:41Z

@dlebauer @jdmaloney I roughly scanned the data files and noticed an inconsistency in file size/data point count and in some cases the naming conventions.

For example, 2017-01-26/WeatherStation_SecData_2017_01_26_1802.dat is more than 10 times the size of 2017-01-27/WeatherStation_SecData_2017_01_27_0915.dat. And while most folders contain files named WeatherStation_SecData_*_*_*_*.dat, some folders only contain one file WeatherStation_SecData.dat, but then there is file 2016-08-09/WeatherStation_SecData.dat sitting next to 2016-08-09/WeatherStation_SecData_2016_08_09_2234.dat and some others.

Is there a pattern to follow here? For the previous met extractor I wrote, it relies on the datasets having 24 partitioned .dat files containing non-overlapping data.

jdmaloney · 2017-02-09T21:31:53Z

@Zodiase I think there was a change in naming format there. 2016_08_09 was early on in the data collection and I think its naming scheme should be ignored, it's only for those first few days (new schema starts at 2016_08_10) and has been consistent since then. In terms of the size variance I am not sure. I'm just grabbing the files off the Network Storage device they have down at Maricopa, how the files are generated and what goes into them I do not know.

dlebauer · 2017-02-09T21:32:48Z

@Andrade-Pedro can you comment on this? I believe this is the second met station you set up. Also can you give approximate coordinates

dlebauer · 2017-02-09T21:33:44Z

@Andrade-Pedro I think JD answered but coordinates would be nice to have

Andrade-Pedro · 2017-02-12T14:56:47Z

Ground weather station coordinates: 33.074457 N, 111.975163 W

Zodiase · 2017-02-16T17:15:21Z

@jdmaloney Wait I just noticed the data files you pointed to me has already been handled by the first Met extractor in #156. The path you gave me is exactly the same as the one in this comment: #156 (comment)

max-zilla · 2017-02-16T17:26:53Z

@jdmaloney what data is David talking about "transferred outside core lemnatec pipeline"? We move the weather station data via pipeline. Does he mean the lightning data that we're doing the append-to-single-file stuff?

dlebauer · 2017-02-16T18:29:12Z

Max there is the Environmental Logger and the Weather Station. There was some confusion initially - Xingchen had already processed the weather station data and will now work on the Environmental Logger data.

…

On Thu, Feb 16, 2017 at 11:26 AM Max Burnette ***@***.***> wrote: @jdmaloney <https://github.com/jdmaloney> what data is David talking about "transferred outside core lemnatec pipeline"? We move the weather station data via pipeline. Does he mean the lightning data that we're doing the append-to-single-file stuff? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#252 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcX5xf1ubist3Trbm4MiFCuhb3tn2LPks5rdIbdgaJpZM4L1Wf9> .

jdmaloney · 2017-02-16T18:38:32Z

@Zodiase Here is the path to the other data:

/gpfs/largeblockFS/projects/arpae/terraref/sites/ua-mac/raw_data/EnvironmentLogger

Zodiase · 2017-02-22T21:07:24Z

Source File Schema

Document (with example values)

{
  "environment_sensor_fixed_infos": {
    "par_sensor": {
      "manufacturer": "www.apogeeinstruments.com",
      "model": "SQ214",
      "location in gantry system": "top of gantry"
    },
    "co2_sensor": {
      "sensor manufacturer": "Vaisala",
      "model": "Carbocap CO2 Probe GMP343 A1C1B0N0N0B",
      "sensor serial number": "L3420008",
      "additional info": "SO 5530060878",
      "calbration date": "2015.08.18",
      "location in gantry system": "camera box",
      "analog digital interface": "WAGO 750-478"
    },
    "weather_station": {
      "manufacturer": "www.thiesclima.com",
      "article nr": "4.9200.00.000",
      "analog digital interface": "wago 750-478",
      "location in gantry system": "top of gantry"
    },
    "spectrometer": {
      "manufacturer": "www.oceanoptics.com",
      "model": "STS-VIS",
      "location in gantry system": "top of gantry"
    }
  },
  "environment_sensor_readings": Array.<SensorReading>
}

SensorReading

{
  "timestamp": "2017.01.10-00:32:02",
  "weather_station": {
    "sunDirection": {
      "value": "357.5829340495",
      "unit": "degrees",
      "rawValue": "9.93285927915281"
    },
    "airPressure": {
      "value": "1017.8899502548",
      "unit": "hPa",
      "rawValue": "8.38038270210883"
    },
    "brightness": {
      "value": "1.0364391003",
      "unit": "kilo Lux",
      "rawValue": "0.00183111056855983"
    },
    "relHumidity": {
      "value": "75.0328073977",
      "unit": "relHumPerCent",
      "rawValue": "7.50328073976867"
    },
    "temperature": {
      "value": "11.542710654",
      "unit": "DegCelsius",
      "rawValue": "5.15427106540117"
    },
    "windDirection": {
      "value": "315.9984130375",
      "unit": "degrees",
      "rawValue": "8.77773369548631"
    },
    "precipitation": {
      "value": "0.0406736656",
      "unit": "mm/h",
      "rawValue": "0.00396740623187964"
    },
    "windVelocity": {
      "value": "1.5326395459",
      "unit": "m/s",
      "rawValue": "0.255439924314097"
    }
  },
  "sensor par": {
    "value": "0",
    "unit": "umol/(m^2*s)",
    "rawValue": "4"
  },
  "sensor co2": {
    "value": "480.039067269",
    "unit": "ppm",
    "rawValue": "11.6806250763033"
  },
  "spectrometer": {
    "maxFixedIntensity": "16383",
    "integration time in us": "5000",
    "wavelength": Array.<Number>,
    "spectrum": Array.<Number>
  }
}

Zodiase · 2017-02-22T21:20:25Z

Conversion

For each in `$.environment_sensor_readings`:

$.timestamp
> $.start_time

$.weather_station.airPressure(hPa)
> $.properties.air_pressure(Pa)

$.weather_station.relHumidity(relHumPerCent)
> $.properties.relative_humidity(%)

$.weather_station.temperature(DegCelsius)
> $.properties.air_temperature(K)

$.weather_station.windDirection(degrees) + $.weather_station.windVelocity(m/s)
> $.properties.eastward_wind(m/s) + $.properties.northward_wind(m/s)

$.weather_station.precipitation(mm/h)
> $.properties.precipitation_flux(kg m-2 s-1)

Zodiase · 2017-02-22T21:27:02Z

@dlebauer I've listed the conversion I'm sure about. There are a couple other fields I'm not sure about. Could you provide a mapping like you did in #156 (comment)?

dlebauer · 2017-02-22T22:16:04Z

@Zodiase

sunDirection --> zenith_angle (degree) @czender can you please confirm?
for wind velocity --> if you are converting to north and east vectors make sure to use appropriate trigonometry
sensor par --> surface_downwelling_photosynthetic_photon_flux_in_air (mol m-2 s-1)
sensor co2 --> mole_fraction_of_carbon_dioxide_in_air (umol/mol)
- the units of this can be 'umol / mol' or 'ppm'

For the spectrometer fields, these will be used in the hyperspectral workflow.

@FlyingWithJerome and @czender what / where is the conversion from the radiometer values to radiances as described in How to calculate downwelling spectral radiances reference-data#30?

dlebauer · 2017-02-22T22:20:45Z

@czender and @FlyingWithJerome where did we leave off with the creation of the Environmental logger files as netcdf? Should we leave those in netcdf files? If so, is there an issue (or should I create a new one) to make those as CF standard names?

FlyingWithJerome · 2017-02-22T23:10:40Z

@dlebauer I doubled checked the Environmental Logger Script in the computing pipeline, and the answer to the naming issue is yes. There is a TODO token in the line 252 for standard names.

@Zodiase @czender
If you would like to know how to convert the units, you may check the Environmental Logger Script. There is a large global dictionary "_UNIT_DICTIONARY" for you to refer to.

czender · 2017-02-23T16:44:05Z

@Zodiase sunDirection is computed (not measured) from time/location. it is not zenith angle. i'm not sure exactly what it is yet. the sensor documentation does not describe it. and the angle takes values that exceed 90 degrees during daytime, so it is neither a zenith angle nor an altitude. please leave it as sunDirection until we understand it. @solmazhajmohammadi or @TinoDornbusch do you know how thiesclima defines sunDirection?

czender · 2017-02-23T16:50:37Z

@FlyingWithJerome please note that the environmental logger script was moved to a new repository many months ago: https://github.com/terraref/extractors-environmental
It does not make sense to maintain the version in computing-pipeline. Please work with new repository and deprecate the old.

FlyingWithJerome · 2017-02-23T17:01:09Z

@czender Yes, I had already cloned the new repo

ghost · 2017-02-23T17:10:24Z

New functionality from PiClowder 2 will help with this. Xingchen will follow up with Max about this.

robkooper · 2017-03-09T05:16:12Z

I would prefer to use auth with username/password. That way it gets assigned to a user instead of the system user (which is what happens when you use the key).

Zodiase · 2017-03-09T05:58:09Z

I'm reading the existing environmental logger extractor code. Since I'm not that familiar with Python so I might have missed something, but so far from this portion I don't think that extractor is doing any unit conversion other than renaming the units to CF standards. For example the conversion from Celsius to Kelvin. When I was working on the meterological extractor I took it as a requirement to convert all values to CF variable names and units suggested in this table, using "preferred variables" as much as possible (which means breaking windVelocity and windDirection into eastward_wind and northward_wind. So far I don't see the environmental logger extractor code doing any of these. Did I miss anything?

max-zilla · 2017-03-09T14:20:06Z

@Zodiase I believe you are correct in both cases - the conversion is for the key only (not the value), and it needs to be both like the met extractor.

FlyingWithJerome · 2017-03-09T19:24:21Z

@max-zilla @Zodiase
You are correct. Numerical correction on these values will come with geostreaming in the next patch.

FlyingWithJerome · 2017-03-09T19:29:57Z

@max-zilla
But in this way, the API key will be hard coded in the driver (aka terra_environmentlogger.py)?

czender · 2017-03-09T20:02:01Z

@Zodiase thanks for reporting this. A 273 degree temperature bias is worth fixing. @FlyingWithJerome please make sure to convert value not just units. Double-check to make sure there are no missing factors of 10, 100, 1000 missing from the values of any converted quantity. Please note any discrepancies between our data names/standard_names and the table Xingchen points to so we can discuss these Monday.

FlyingWithJerome · 2017-03-10T18:43:55Z

@max-zilla
The Geostream API had worked fine yesterday, but today it raises HTTPError 500 again; is it under maintenance right now?

max-zilla · 2017-03-10T18:53:28Z

@FlyingWithJerome the postgres service was restarted, Clowder needed to be restarted as well to reconnect. Should work now.

FlyingWithJerome · 2017-03-13T17:04:42Z

@max-zilla

I double checked the API documentation in the clowder and cannot find the parts for Geostream. I would like to know the the host address I can POST to, and whether it supports JSON-Encoded data (seems that it does not support Python json.dumps() encoder).

max-zilla · 2017-03-13T17:53:53Z

@FlyingWithJerome here are two examples of extractors that post to Geostreams:
https://github.com/terraref/extractors-meterological/blob/master/datparser/terra_met_datparser.py#L244
https://github.com/terraref/extractors-metadata/blob/master/sensorposition/terra_sensorposition.py#L162

See create_sensor, get_sensor_id, create_stream, get_stream_id also.

Basically for geostreams...

"sensor" is top level - ignore the word sensor because it's confusing (from another project). For TERRA, each "sensor" is a plot in the field.
"stream" is each instrument x each plot. so if Plot="Range 1 Pass 1" then a stream might be "VNIR - Range 1 Pass 1".
"datapoint" is a single measurement at a single point in time, e.g. reflectance. these go into a stream.

Big idea is we have a field of plots (1) and each plot has a list of instruments (2) that each have a bunch of datapoints (3) indicating what that instrument captured in that plot.

FlyingWithJerome · 2017-03-13T18:01:12Z

@max-zilla

Thanks, these two examples make more sense to me. I will double check if I have the pyclowder dependencies on my home machines and do some little researches on them

max-zilla · 2017-03-13T18:09:14Z

@FlyingWithJerome make sure you use the newer PyClowder 2: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse

python setup.py install --force

The --force will overwrite any pyclowder 1 files if you already had them installed.

dlebauer · 2017-03-15T15:14:35Z

@FlyingWithJerome there is some confusion on this side - have you taken on this issue? If so, that is great. Or is there anything that @Zodiase can help with?

One other thought - instead of creating one file per day, can this netcdf file just be appended in time, perhaps creating a new one each year?

FlyingWithJerome · 2017-03-15T16:47:24Z

@dlebauer
The unit conversion issue had already been fixed; I'm right now polishing the code based on Craig's feedback and adding geostream to the pipeline. Max had provided me some useful references on Geostream API.

ghost · 2017-04-04T17:46:41Z

@FlyingWithJerome - can you give an update on this issue? Can it be closed?

FlyingWithJerome · 2017-04-04T18:16:00Z

@max-zilla
Hi Max, I would like to know how to fill in the "geom," "type," and "region" for create_sensor(), since the Environmental Logger has no region, type or geometry. If I had all of them as None, I will get a 400 Bad Request.
BTW, "connector" can be just a random Connector instance?

FlyingWithJerome · 2017-04-04T18:16:44Z

@rachelshekar I'm working on this; will be finished in a not-so-far future

max-zilla · 2017-04-05T14:24:08Z

@FlyingWithJerome for the met extractor, we are using a location off to the side of the field (on the building nearby) to represent "full field" data:

https://github.com/terraref/extractors-meterological/blob/master/datparser/terra_met_datparser.py

def process_message(self, connector, host, secret_key, resource, parameters):
self.sensor_name = "Full Field"
main_coords = [ -111.974304, 33.075576, 0]

sensor_id = get_sensor_id(host, secret_key, self.sensor_name)
		if not sensor_id:
			sensor_id = create_sensor(host, secret_key, self.sensor_name, {
				"type": "Point",
				# These are a point off to the right of the field
				"coordinates": main_coords
			})


stream_name = self.sensor_name + " - Weather Station"
		stream_id = get_stream_id(host, secret_key, stream_name)
		if not stream_id:
			stream_id = create_stream(host, secret_key, sensor_id, stream_name, {
				"type": "Point",
				"coordinates": main_coords
			})

I would replace "Weather Station" with EnvironmentLogger but otherwise you can use the same Full Field sensor we're using for the weather data.

The connector is passed into the process_message method for you.

FlyingWithJerome · 2017-04-05T15:48:28Z

@max-zilla
In PyClowder2 the create_sensor() takes 7 arguments, but you only pass 4?

def create_sensor(connector, host, key, sensorname, geom, type, region):
    ...

max-zilla · 2017-04-05T16:06:32Z

@FlyingWithJerome ah yes, didn't realize the example function was custom before pyClowder 2 had geostreams.

body = {
		"name": name,
		"type": "point",
		"geometry": {
				"type": "Point",
				# These are a point off to the right of the field
				"coordinates": main_coords
			},
		"properties": {
			"popupContent": name,
			"type": {
				"id": "MAC Field Scanner",
				"title": "MAC Field Scanner",
				"sensorType": 4
			},
			"name": name,
			"region": "Maricopa"
		}
	}

So for using PyC2:

geom = {
"type": "Point",
# These are a point off to the right of the field
"coordinates": main_coords
}

type = {
"id": "MAC Field Scanner",
"title": "MAC Field Scanner",
"sensorType": 4
}

region = "Maricopa"

FlyingWithJerome · 2017-04-05T17:56:44Z

@max-zilla
These are extremely helpful! Thanks!

dlebauer · 2017-04-05T19:09:23Z

Just to be clear - the EnvirontmentLogger and WeatherStation should be listed as distinct locations (sensors).

The WeatherStation coordinates are 33.074457 N, 111.975163 W Add extractor for data from EnvironmentLogger --> geostreams database #252 (comment)
The EnvironmentLogger is on top of the gantry system so it is moving around. For the EnvironmentLogger I think it would be better to have a polygon that circumscribes the gantry than to have a point off to the side (though for mapping purposes it might be preferable to keep it off to the side).

max-zilla · 2017-04-05T19:14:51Z

@dlebauer noted. we can change that location/geometry at any time fortunately, will make a note.

FlyingWithJerome · 2017-04-05T19:42:45Z

@dlebauer
I think the EL records have the weather station data, but has no position data (i.e., seems that we cannot trace the EL Logger position like what we do on hyperspectral cameras). Only the weather station in EL records is in a fixed position, and every sensors else (sensor par, sensor co2, etc.) are movable?

max-zilla · 2017-04-05T19:58:36Z

@FlyingWithJerome i think that's correct - that's why David suggests using a polygon to represent the entire field or something else off to the side if there are overlap issues. We can experiment with that a bit later.

FlyingWithJerome · 2017-04-05T20:13:23Z

@max-zilla
Sounds like a plan. I will finish the geostream stuff these days and send to David and you for review

ghost · 2017-04-26T13:29:57Z

@FlyingWithJerome can this issue be closed?

max-zilla · 2017-04-26T13:37:33Z

@rachelshekar yes I merged @FlyingWithJerome 's pull request.

dlebauer assigned Zodiase and jdmaloney Feb 2, 2017

dlebauer changed the title ~~Add extractor for data from second MAC met station --> geostreams database~~ Add extractor for data from EnvironmentLogger --> geostreams database Feb 22, 2017

dlebauer added the help wanted label Feb 22, 2017

dlebauer assigned czender and FlyingWithJerome and unassigned jdmaloney Feb 22, 2017

dlebauer added this to the February 2017 milestone Feb 22, 2017

ghost removed the help wanted label Feb 23, 2017

max-zilla mentioned this issue Mar 14, 2017

Insert plot level height histogram into Clowder geostreams; height into BETYdb #210

Closed

3 tasks

max-zilla closed this as completed Apr 26, 2017

Add extractor for data from EnvironmentLogger --> geostreams database #252

Add extractor for data from EnvironmentLogger --> geostreams database #252

Comments

dlebauer commented Feb 2, 2017 • edited Loading

Zodiase commented Feb 2, 2017

jdmaloney commented Feb 2, 2017 • edited Loading

Zodiase commented Feb 9, 2017

jdmaloney commented Feb 9, 2017

dlebauer commented Feb 9, 2017

dlebauer commented Feb 9, 2017

Andrade-Pedro commented Feb 12, 2017

Zodiase commented Feb 16, 2017

max-zilla commented Feb 16, 2017

dlebauer commented Feb 16, 2017 via email

jdmaloney commented Feb 16, 2017

Zodiase commented Feb 22, 2017

Source File Schema

Document (with example values)

SensorReading

Zodiase commented Feb 22, 2017 • edited Loading

Conversion

For each in $.environment_sensor_readings:

Zodiase commented Feb 22, 2017

dlebauer commented Feb 22, 2017 • edited Loading

dlebauer commented Feb 22, 2017

FlyingWithJerome commented Feb 22, 2017

czender commented Feb 23, 2017

czender commented Feb 23, 2017

FlyingWithJerome commented Feb 23, 2017

ghost commented Feb 23, 2017

robkooper commented Mar 9, 2017

Zodiase commented Mar 9, 2017

max-zilla commented Mar 9, 2017

FlyingWithJerome commented Mar 9, 2017

FlyingWithJerome commented Mar 9, 2017 • edited Loading

czender commented Mar 9, 2017

FlyingWithJerome commented Mar 10, 2017 • edited Loading

max-zilla commented Mar 10, 2017

FlyingWithJerome commented Mar 13, 2017

max-zilla commented Mar 13, 2017

FlyingWithJerome commented Mar 13, 2017

max-zilla commented Mar 13, 2017

dlebauer commented Mar 15, 2017

FlyingWithJerome commented Mar 15, 2017

ghost commented Apr 4, 2017

FlyingWithJerome commented Apr 4, 2017

FlyingWithJerome commented Apr 4, 2017

max-zilla commented Apr 5, 2017 • edited Loading

FlyingWithJerome commented Apr 5, 2017 • edited Loading

max-zilla commented Apr 5, 2017

FlyingWithJerome commented Apr 5, 2017

dlebauer commented Apr 5, 2017

max-zilla commented Apr 5, 2017

FlyingWithJerome commented Apr 5, 2017

max-zilla commented Apr 5, 2017

FlyingWithJerome commented Apr 5, 2017

ghost commented Apr 26, 2017

max-zilla commented Apr 26, 2017

dlebauer commented Feb 2, 2017 •

edited

Loading

jdmaloney commented Feb 2, 2017 •

edited

Loading

Zodiase commented Feb 22, 2017 •

edited

Loading

For each in `$.environment_sensor_readings`:

dlebauer commented Feb 22, 2017 •

edited

Loading

FlyingWithJerome commented Mar 9, 2017 •

edited

Loading

FlyingWithJerome commented Mar 10, 2017 •

edited

Loading

max-zilla commented Apr 5, 2017 •

edited

Loading

FlyingWithJerome commented Apr 5, 2017 •

edited

Loading