Store location and field of view metadata associated with each file in Clowder #101

dlebauer · 2016-05-12T20:56:57Z

Description

Metadata files provide geospatial information that we should add to meta-data and also to the Clowder PostGIS database

convert sensor location and field of view into geojson location metadata in Clowder and
add field of view to PostGIS database to enable cross-database queries
- For more information about Clowder PostGIS database, speak with Luigi Marini and Rob Kooper.

Position of the gantry + camera

assume that the center of the sensor is located at

position + location in camera box

e.g. :
["location in camera box X [m]", "location in camera box Y [m]" + "location in camera box Z [m]"] + "Position x [m]", "Position y [m]" + "Position z [m]"]

Field of view

Assume the value provided for field of view is the height and width of the sides of the bounding box

Transforming to geospatial coordinates

Info from Solmaz on converting XYZ in meters to coordinates (for hyperspectral, but has SE corner origin points) terraref/documentation#9

Check what level of precision is required given resolution of the image. (how many significant digits in WGS85, web mercator).

Example

Here is an example of the current meta-data format (interpretation is discussed in terraref/reference-data#25), until this is clarified please assume that the center of the 'field of view' is located at:

      "optics focus setting (both)": "2m",
      "optics apperture setting (both)": "4",
      "Output data format": "Bayer GR8 3296x2472",
      "sensor purpose": "capture RGB images",
      "location in gantry system": "camera box, facing ground",
      "location in camera box X [m]": "0.877",
      "location in camera box Y [m]": "2.276",
      "location in camera box Z [m]": "0.578",
      "Field of view at 2m in X- Y- direction [m]": "[1.857 1.246]",
      "Bounding Box [m]": "[1.857     1.246]",

An example file on ROGER that you can start with is at /gpfs/largeblockFS/projects/arpae/terraref/raw_data/ua-mac/MovingSensor/ps2Top/2016-05-07/2016-05-07__15-58-43-382/36dfd7fe-e648-427d-b503-fdf47f198545_metadata.json

Reported Z in metadata is calculated from following equation.

ZHeightInM = 2.0f //focal length

0.0f //average height of plants
0.75f //distance ground box at zHeight = 0
0.58f; //camera distance inside box

Context

This feature will be required to find all images based on location (e.g. those associated with a plant or plot)

others for comment: @robkooper @pless @solmazhajmohammadi

caicai89- · 2016-05-17T04:38:36Z

location: position (gantry_system_variable_metadata)
+location in camera box (sensor_fixed_metadata)
Question: 1. Are both the location and position in UTM coordinate?

"Field of view at 2m in X- Y- direction [m]": "[1.857 1.246]"
Question: 2. Does Field of view equal to "Field of view at 2m in X- Y- direction [m]" ?

dlebauer · 2016-05-17T04:43:49Z

This is relative to the SE corner of the gantry. See terraref/documentation#9

TinoDornbusch · 2016-05-17T07:47:12Z

note that the [0,0] coordinate of the gantry is at south east corner.

TinoDornbusch · 2016-05-17T12:39:26Z

South east. Wagner&Müller (the company that programmed the gantry) has strange ideas on definitions & procedures.

TinoDornbusch · 2016-05-17T12:41:34Z

@caicai89- Field of view would describe the size of the rectangle you would see at 2m distance with the sensor.

Note that for line sensors (3d, Hyperspec) this is a y dimension only.

max-zilla · 2016-05-23T16:29:13Z

Followup from meeting on 05/23 with @dlebauer @caicai89- @yanliu-chn @robkooper.

Summary workflow:

Add message for RabbitMQ on dataset.addmetadata
Write extractor to listen for new metadata on dataset - if we find XYZ position in meters, use Southeast origin lat/long + meter offset of camera box + sensor to determine sensor coordinates
Use geostream API to add sensor coordinates to PostGIS database
Later, show map of geostream datapoints and allow querying

Right now, the message for 1 does not exist. Suggest starting just by converting sample coordinates to lat/lon. Then we can implement that in extractor on actual metadata once other parts are done. After converting coordinates is done, we can enable PostGIS database on TERRA Clowder development instance (http://141.142.209.122/clowder/) and create some sample data there to test with.

caicai89- · 2016-05-24T16:22:09Z

@robkooper Could you share the geostream example with me and @yanliu-chn

dlebauer · 2016-05-24T16:43:16Z

Here is the example file from above (renamed to .txt to allow upload here)

2016-04-13_00-38-15_environmentlogger.json.txt

max-zilla · 2016-05-26T15:02:35Z

@caicai89- I think we will need to add a *.dataset.metadata.added message, there does not currently seem to be one.

*.dataset.file.added is valid, for example. Will talk with @robkooper but this should be a simple addition to Clowder along the lines of:

current.plugin[RabbitmqPlugin].foreach { p =>
        val dtkey = s"${p.exchange}.dataset.file.added"
        p.extract(ExtractorMessage(file.id, file.id, clowderurl, dtkey, Map.empty, file.length.toString, ds.id, ""))
      }

...adjusted for metadata and triggered in the right places.

There are separate pull requests to add a dataset.file.added message and to add support in PyClowder for dataset extractors:

caicai89- · 2016-06-08T18:20:17Z

@max-zilla What is the queue name for the queue with message "*.dataset.metadata.added", "dataset.addmetadata"?
@robkooper Have you prepared the sample code for me to learn and implement?

max-zilla · 2016-06-08T19:06:17Z

@caicai89- the queue name should be defined by your extractor in config.py. For example, take a look at my pending pull request for PyClowder example dataset extractor:
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse/sample-extractors/dataset-filecount?at=refs%2Fheads%2Fbugfix%2FCATS-554-add-pyclowder-support-for-dataset

in config.py I give extractorName=datasetFileCount
in dataset-filecount.py you can just use:

extractors.connect_message_bus(extractorName=extractorName, messageType=messageType, processFileFunction=process_dataset,
        checkMessageFunction=check_message, rabbitmqExchange=rabbitmqExchange, rabbitmqURL=rabbitmqURL)

...to automatically connect and listen for messages on Clowder exchange, datasetFileCount queue.

The parameters for on_message() and process_file() will have info about dataset. Can show you sometime if you want.

caicai89- · 2016-06-09T04:50:28Z

@max-zilla thank you!

robkooper · 2016-06-13T20:43:21Z

Quick intro to geostreams:

datapoints are a measurement at a specific time at a specific location, needs a stream_id
streams are grouping of datapoints that logially belong together, for example a season at a plot, needs a sensor_id
sites/sensors is a high level description of a sensor at a location.

In our case we can see a sensor/site as a plot, and a stream as a planting season, and a datapoint a specific image.

Example code that works with the geostreams API:
https://opensource.ncsa.illinois.edu/bitbucket/projects/GEOD/repos/seagrant-parsers-py/browse/USGS/usgs-import.py

A second more complex example : https://opensource.ncsa.illinois.edu/bitbucket/projects/GEOD/repos/seagrant-parsers-py/browse/SeaBird/seabird-import.py

Create sensor : https://opensource.ncsa.illinois.edu/bitbucket/projects/GEOD/repos/seagrant-parsers-py/browse/SeaBird/seabird-import.py#333

Create stream : https://opensource.ncsa.illinois.edu/bitbucket/projects/GEOD/repos/seagrant-parsers-py/browse/SeaBird/seabird-import.py#354

Create datapoint : https://opensource.ncsa.illinois.edu/bitbucket/projects/GEOD/repos/seagrant-parsers-py/browse/SeaBird/seabird-import.py#395

caicai89- · 2016-06-14T03:08:14Z

Thanks! @robkooper

ghost · 2016-07-07T14:04:20Z

@caicai89- this appears to be a discussion ... is there an actual task associated with this issue?

dlebauer · 2016-07-07T14:05:05Z

Task is defined by title
On Thu, Jul 7, 2016 at 9:04 AM Rachel Shekar notifications@github.com
wrote:

@caicai89- https://github.com/caicai89- this appears to be a discussion
... is there an actual task associated with this issue?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#101 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAcX51QwzcauORzGCkGBAZ3N2niqL8gRks5qTQdkgaJpZM4IdeQP
.

caicai89- · 2016-07-07T14:07:00Z

terraref/reference-data#32 (comment) is a discussion

ghost · 2016-07-07T14:15:31Z

@caicai89- have you "Stored location and field of view metadata associated with each file in Clowder"?

caicai89- · 2016-07-07T14:47:16Z

Not yet, last week Max helped me set up the testing environment, and I found some errors during testing. In addition, for the geostreams API part, I still have some questions.

max-zilla · 2016-08-31T13:02:48Z

@caicai89- does the sensor ID creation make sense?

I think in addition to seeing if the metadata is there, we should check dataset name:

does it have a hyphen " - " in the name? if so, sensor name is the part before the hyphen

ds = "co2Sensor - 2016-06-10"
sensorname = ds.split(" - ")[0]

if we found the metadata and the sensor name, we should have a map of sensor name to Stream ID. The list of sensors is basically the SENSOR_DIR column in this sheet: https://docs.google.com/spreadsheets/d/1LLiQSFHbEWoo_FkvG1nl-XoSq_lZ_o6VazfxmPfOH9Y/edit#gid=0

sensorMap = {"co2Sensor": "1"} etc.

so add the datapoint to a stream based on which sensor it is. Your code should also create the necessary stream if it doesn't exist yet.

caicai89- · 2016-09-01T15:21:21Z

@max-zilla Will work on that later.

ghost · 2016-09-22T19:19:15Z

@caicai89- have you made the updates requested above? If not, can you put them in a new issue?

max-zilla · 2016-10-25T13:52:25Z

@dlebauer @robkooper I finished up the remaining tasks on this extractor and deployed to clowder-dev instance. the process is ready:

INFO    : pyclowder.extractors -  Starting a New Thread for Process Dataset
sensor lat/lon: (33.07634044407777, -111.97480020961399)
F.O.V. NW lat/lon: (33.07634876600311, -111.97480697587183)
F.O.V. SE lat/lon: (33.076332122152046, -111.97479344335744)
checking stereoTop.
posting datapoint to stream 5
Successfully added datapoint.

this requires a "sensor" in Geostreams to exist (I created a UA-MAC sensor)
will accept a map of "instrument name" -> "stream ID" in config.py. when a new dataset is processed the instrument name, e.g. stereoTop is checked against this map to see whether a stream was predefined, and a stream is created if not. the following streams should be created and added to config ahead of time like I did on the dev instance (the IDs don't have to match these numbers, but they'll need to be updated):

"stereoTop": "5",
                    "flirIr": "6",
                    "co2Sensor": "7",
                    "cropCircle": "8",
                    "priSensor": "9",
                    "scanner3DTop": "11",
                    "ndviSensor": "10",
                    "ps2Top": "12",
                    "SWIR": "13",
                    "VNIR": "2"

datapoints are then added into the corresponding stream.

I verified via the API that the stream + datapoints exist, but I wasn't seeing the datapoints appear in the Clowder UI. This might be a TERRA-dev Clowder issue (they appeared locally in my testing) but the data are there.

We'll need to deploy PostGIS to the production server before we can deploy this there.

NOTE: This adds the datapoint as a "point" still (geojson support in Clowder is being reviewed) but in the properties of the dp we store the FOV polygon so we can use it later:

metadata = {
        "sources": host+"datasets/"+parameters['datasetId'],
        "file_ids": ",".join(fileIdList),
        "centroid": {
            "type": "Point",
            "coordinates": [sensor_latlon[1], sensor_latlon[0]]
        },
        "fov": {
            "type": "Polygon",
            "coordinates": [[[fov_nw_latlon[1], fov_nw_latlon[0], 0],
                             [fov_se_latlon[1], fov_se_latlon[0], 0] ]]
        }
    }

dlebauer · 2016-10-26T03:47:26Z

Are two points enough to define a polygon? I don't think we can assume that the edges are parallel to lat/lon ( @tingli3 ?)

ghost · 2016-11-02T15:00:09Z

Deployed to clowder-dev. @robkooper - Ready to move to production

ghost · 2016-11-14T19:50:53Z

can this issue be closed?

max-zilla · 2016-11-17T17:24:04Z

I plan to install geostreams stuff today/tomorrow and deploy this on production, then I'll close.

max-zilla · 2016-11-22T15:43:25Z

I installed geostreams database on production and deployed the geospatial metadata extractor. Closing this issue.

dlebauer · 2017-04-19T23:37:07Z

@max-zilla at this point, where can I find the bounding box for the image files? For example, where are the field-of-view bounding boxes for the geotiffs here: https://terraref.ncsa.illinois.edu/clowder/datasets/58dd375a4f0c430e2bff1b21

max-zilla · 2017-04-20T14:44:23Z

@dlebauer those are stored in the geostreams plots. The sensorposition extractor has this JSON data for each dataset/file it processes:

{
            "fov": {
                "type": "Polygon",
                "coordinates": [
                    [
                        [
                            -111.97502740364716,
                            33.07628117585825,
                            0
                        ],
                        [
                            -111.97501541748397,
                            33.07626777606369,
                            0
                        ]
                    ]
                ]
            },
            "sources": "https://terraref.ncsa.illinois.edu/clowder/datasets/587f9adc4f0cd67174e61751",
            "centroid": {
                "type": "Point",
                "coordinates": [
                    -111.9750214105651,
                    33.07627447596113
                ]
            },
            "file_ids": "587f9add4f0cd67174e61757"
        }

...in this case, what you're looking for are file_ids. I just created this pull request:
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/clowder/pull-requests/1128/overview

that will add a datapoints search filter so you can search like this:

http://localhost:9000/clowder/api/geostreams/datapoints?filter={"sources":"https://terraref.ncsa.illinois.edu/clowder/datasets/58dd375a4f0c430e2bff1b21"}

or

http://localhost:9000/clowder/api/geostreams/datapoints?filter={"file_ids":"58dd375a4f0c430e2bff1b21"}

etc. The datapoint FOV will have it. But Zongyang has been overhauling how the stereo RBG geotiffs are generated, so the FOVs in the geostreams will need to also be updated to match.

dlebauer · 2017-08-17T17:28:41Z

@max-zilla has this been implemented? Does geostreams now store the FOV as a PostGIS geometry for each file?

max-zilla · 2017-08-17T18:06:03Z

yes, postGIS stores FOV now in sensorposition. closing.

dlebauer · 2017-08-17T20:28:40Z

@max-zilla can you give examples? For example, if I want the bounding box for

https://terraref.ncsa.illinois.edu/clowder/files/598fa7f74f0c7dbcc8c0da69?dataset=598fa7f74f0c7dbcc8c0da5e&space=571fb3e1e4b032ce83d95ecf

Wouldn't I use the query https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?filter={%22file_ids%22:%22598fa7f74f0c7dbcc8c0da69%22}

? This returns the error in
error.txt

max-zilla · 2017-08-17T20:48:48Z

This is in the new extractor updates I have yet to deploy - no results to look at yet. should have some in next day or two.

dlebauer assigned caicai89- May 12, 2016

dlebauer added the 2 - Working <= 5 label May 26, 2016

dlebauer mentioned this issue Jun 7, 2016

Convert gantry coordinate system from right handed (x positive to left) to conventional (x positive to right) #109

Closed

ghost added the kind/discussion label Jul 7, 2016

ghost modified the milestone: June 2016 Jul 7, 2016

dlebauer modified the milestones: June 2016, July 2016 Jul 11, 2016

dlebauer mentioned this issue Jul 14, 2016

Create generic extractor for geospatial data from *_metadata.json files #117

Closed

ghost modified the milestones: September 2016, July 2016 Sep 22, 2016

czender mentioned this issue Sep 30, 2016

Please document converting geographic coordinate systems terraref/documentation#7

Closed

This was referenced Oct 5, 2016

Converting lemnatec scanner metadata to absolute geographic positions #180

Closed

Extractors for meteorological data #156

Closed

dlebauer assigned max-zilla Oct 7, 2016

ghost unassigned caicai89- Oct 13, 2016

max-zilla closed this as completed Nov 22, 2016

ghost added the 4 - Done label Jan 3, 2017

dlebauer added the help wanted label Apr 19, 2017

dlebauer reopened this Apr 19, 2017

dlebauer mentioned this issue Apr 19, 2017

How to link the BETYdb's traits information back to Globus's RGB data? terraref/reference-data#112

Closed

2 tasks

ghost removed the help wanted label Apr 20, 2017

max-zilla closed this as completed Aug 17, 2017

Chris-Schnaufer mentioned this issue Nov 7, 2019

Port laz/ply extractors: las2height AgPipeline/issues-and-projects#20

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store location and field of view metadata associated with each file in Clowder #101

Store location and field of view metadata associated with each file in Clowder #101

dlebauer commented May 12, 2016 •

edited

Loading

caicai89- commented May 17, 2016

dlebauer commented May 17, 2016 •

edited

Loading

TinoDornbusch commented May 17, 2016

TinoDornbusch commented May 17, 2016

TinoDornbusch commented May 17, 2016

max-zilla commented May 23, 2016

caicai89- commented May 24, 2016

dlebauer commented May 24, 2016

max-zilla commented May 26, 2016

caicai89- commented Jun 8, 2016

max-zilla commented Jun 8, 2016 •

edited

Loading

caicai89- commented Jun 9, 2016

robkooper commented Jun 13, 2016

caicai89- commented Jun 14, 2016

ghost commented Jul 7, 2016

dlebauer commented Jul 7, 2016

caicai89- commented Jul 7, 2016

ghost commented Jul 7, 2016

caicai89- commented Jul 7, 2016

max-zilla commented Aug 31, 2016 •

edited

Loading

caicai89- commented Sep 1, 2016

ghost commented Sep 22, 2016

max-zilla commented Oct 25, 2016 •

edited

Loading

dlebauer commented Oct 26, 2016

ghost commented Nov 2, 2016

ghost commented Nov 14, 2016

max-zilla commented Nov 17, 2016

max-zilla commented Nov 22, 2016

dlebauer commented Apr 19, 2017

max-zilla commented Apr 20, 2017

dlebauer commented Aug 17, 2017

max-zilla commented Aug 17, 2017

dlebauer commented Aug 17, 2017 •

edited

Loading

max-zilla commented Aug 17, 2017

Store location and field of view metadata associated with each file in Clowder #101

Store location and field of view metadata associated with each file in Clowder #101

Comments

dlebauer commented May 12, 2016 • edited Loading

Description

Position of the gantry + camera

Field of view

Transforming to geospatial coordinates

Example

Context

caicai89- commented May 17, 2016

dlebauer commented May 17, 2016 • edited Loading

TinoDornbusch commented May 17, 2016

TinoDornbusch commented May 17, 2016

TinoDornbusch commented May 17, 2016

max-zilla commented May 23, 2016

caicai89- commented May 24, 2016

dlebauer commented May 24, 2016

max-zilla commented May 26, 2016

caicai89- commented Jun 8, 2016

max-zilla commented Jun 8, 2016 • edited Loading

caicai89- commented Jun 9, 2016

robkooper commented Jun 13, 2016

caicai89- commented Jun 14, 2016

ghost commented Jul 7, 2016

dlebauer commented Jul 7, 2016

caicai89- commented Jul 7, 2016

ghost commented Jul 7, 2016

caicai89- commented Jul 7, 2016

max-zilla commented Aug 31, 2016 • edited Loading

caicai89- commented Sep 1, 2016

ghost commented Sep 22, 2016

max-zilla commented Oct 25, 2016 • edited Loading

dlebauer commented Oct 26, 2016

ghost commented Nov 2, 2016

ghost commented Nov 14, 2016

max-zilla commented Nov 17, 2016

max-zilla commented Nov 22, 2016

dlebauer commented Apr 19, 2017

max-zilla commented Apr 20, 2017

dlebauer commented Aug 17, 2017

max-zilla commented Aug 17, 2017

dlebauer commented Aug 17, 2017 • edited Loading

max-zilla commented Aug 17, 2017

dlebauer commented May 12, 2016 •

edited

Loading

dlebauer commented May 17, 2016 •

edited

Loading

max-zilla commented Jun 8, 2016 •

edited

Loading

max-zilla commented Aug 31, 2016 •

edited

Loading

max-zilla commented Oct 25, 2016 •

edited

Loading

dlebauer commented Aug 17, 2017 •

edited

Loading