Data Access API

In addition to CSV download on each variable's page, data can be accessed via the API described here.

Organization

The instruments are referred to as nodes. Each node report measurements of one or more variables periodically. Nodes are grouped into site base on their geographical proximity.

Endpoints

To get a list of sensor nodes at all sites

https://grogdata.soest.hawaii.edu/data/2/config/listing

To get the metadata of a particular node (using node-049 as example)

https://grogdata.soest.hawaii.edu/whatever/data/dashboard.json?nodes=node-049

To get time series data of a variable of a node

https://grogdata.soest.hawaii.edu/data/2/<nodeid>/<variables>.json?time_col=ReceptionTime&begin=<start time in UNIX time>&end=<end time in UNIX time>

Example

https://grogdata.soest.hawaii.edu/data/2/node-008/ReceptionTime,d2w.json?time_col=ReceptionTime&begin=1506401855&end=1506403655

Alternative Endpoint

Instead of specifying the time column to use (ReceptionTime, Timestamp, ts...), this endpoint automatically picks the time column. Using the example above:

https://grogdata.soest.hawaii.edu/data/3/node-008/d2w.json?begin=1506401855&end=1506403655

Note that the response format is different from the endpoint described above.

Example Python scripts to fetch data programmatically

To fetch data (Python3. Requires requests)

With additional plotting (Python3. Requires requests and matplotlib)

Timestamp Convention

When a measurement reaches the database, the time of arrival is recorded as ReceptionTime using server's clock (synchronized with NTP, in UTC).

Some sensors have an internal real-time clock. These would report the time when the measurements are taken. This variable is recorded as Timestamp or more recently, ts using the sensor's built-in real-time clock.

The difference between ReceptionTime and {Timestamp/ts, 1Hz ticker, and Sample Index}

Some field sensors do not have an internal real-time clock, so they do not report a Timestamp or ts variable. These sensors either have a monotonically increasing 1Hz ticker, or their measurements are all indexed (so the time the measurement was taken can be inferred by correlating the ticker/index to ReceptionTime).

During normal operation, the difference between ReceptionTime and Timestamp/ts is negligable. However, when the link between the internet gateways and the database server is temporarily lost (perhaps due to internet blackout, database maintenance...), sensor readings are queued up at the gateways. When the link is restored, all queued messages are sent to the database as fast as the server can handle. In this case, ReceptionTime no longer correlates with the actual time the measurements were taken (example) and must be recovered from the 1Hz ticker and/or the sample index

Converters like this is handy when dealing with POSIX timestamps (no affiliation).

Burst Sampling

Some devices such as the cellular tide gauges queue up several readings before transmitting at once (for power and bandwidth considerations). All readings transmitted in a group would then have ReceptionTime very close to each other since they arrive at the server at around the same time. For time series analysis, Timestamp/ts should be used instead of ReceptionTime.

Water Level Data

Note on the "Distance to Water Surface" Variable, d2w

The ultrasonic tide gauges are mounted above water on fixed structure, pointing downward. Water level is measured indirectly by measuring the distance from the sensor to the water surface. The unit of the variable d2w is millimeter.

For the first Makaha, the conversion from d2w (in millimeter) to water depth (in meter) is:

(50.7 + 1100 - d2w) / 1e3

For the second Makaha:

(50.7 + 2180 - d2w) / 1e3

For Makai Pier:

(6197.6 - d2w) / 1e3