Large-scale gridding

Martin Jakobsson
Marcus Karlsson
Florian Vacek
Rezwan Mohammad

Seabed 2030
coverage calculation

Seabed resolutions

Seabed resolutions

Coverage calculation

Number of data sets/files5,093
Data size [bytes]935,947,192,043
Number of data points26,808,838,976
Number of total grid cells51,051,840,000
Number of ocean grid cells36,144,702,720
Number of mapped ocean grid cells9,103,875,175

Coverage calculation

Number of total grid cells

51,051,840,000

100-metre data

depth
sid
Shelf slope south off Iceland
Shelf slope north off Alaska
West off California
West off California close-up

Input data

x y z sid priority
0 7.602324e+05 8.839106e+06 -363.500000 1 1000
1 7.610190e+05 8.839097e+06 -363.450012 1 1000
2 7.618056e+05 8.839089e+06 -362.790009 1 1000
3 7.625921e+05 8.839080e+06 -361.049988 1 1000
4 7.633786e+05 8.839071e+06 -358.989990 1 1000
…
29712542293 -1.259760e+07 8.806175e+06 -1900.0 14144 0
29712542294 -1.259658e+07 8.806445e+06 -1900.0 14144 0
29712542295 -1.259894e+07 8.805657e+06 -1900.0 14144 0
29712542296 -1.259838e+07 8.805903e+06 -1900.0 14144 0
29712542297 -1.259774e+07 8.805757e+06 -2000.0 14144 0

Input data

Data points

29,712,542,297

Input data

Data points

29,712,542,297

Data types 
x           float64
y           float64
z           float32
sid          uint16
priority     uint64
 
Data size

830 GB

Python

Jupyterlab
Jupyterlab
Numpy
Numpy
Pandas
Pandas
Dask
Dask

Amazon Web Services

aws

Calculation steps

Calculation steps

Calculation steps

Statistical coverage calculation

All 5098 datasets vs. only 11 updated 
00:00:03 00:00:01 stats-check-data
02:19:33 00:06:33 stats-read-data len 26067243872
00:04:29 00:00:00 stats-bin
00:40:48 00:11:57 stats-index
02:50:40 00:32:56 stats-blockmedian
00:01:20 00:00:47 stats-pyramid-filter-bands
00:00:35 00:02:28 stats-pyramid
00:01:13 00:00:54 stats-pyramid-rechunk
00:05:28 00:05:13 stats-render-low
00:01:26 00:01:15 stats-bincount
00:01:26 00:01:16 stats-bincount-count z: 25.000046399606966 %, s: 26.818262095253996 %
00:19:34 00:00:36 stats-extent
00:00:02 00:00:01 stats-agg
00:38:05          stats-regions-centre
00:12:00          stats-regions-rhc
00:14:10          stats-regions-eez
00:37:09          stats-render-merged-projection
08:34:51 01:00:02 stats-total

Calculation steps

Gridding calculation

00:11:59 00:00:57 grid-concat len 29799723872
00:04:22 00:00:17 grid-gebco-weird-projection-9354
00:04:58 00:00:02 grid-gebco-weird-projection-3857
00:04:05 00:00:02 grid-gebco-weird-projection-3996
00:15:28 00:00:13 grid-gebco-weird-projection
00:15:29 00:00:04 grid-gebco-weird-projection-read-data
00:04:39 00:00:03 grid-bin
01:22:40 00:15:14 grid-index
00:17:42          grid-update-json-weirdproj-extent
00:00:01          grid-update-json-weirdproj-extent-write-json
01:07:11 00:02:07 grid-blockmedian-400-m
00:48:04 00:01:46 grid-blockmedian-500-m
00:18:21 00:01:25 grid-blockmedian-250-m
00:16:15 00:01:27 grid-blockmedian-100-m
         00:05:18 merge-blockmedian-update
         00:02:59 blockmedian-rechunk
06:23:08 00:38:34 grid-spline-lowres
00:01:59 00:01:01 grid-spline-lowres-cropped
00:26:40          grid-resample-low-res
00:12:28          grid-resample-high-res
00:23:56          grid-diff-buffered
01:13:54          grid-surface-rr-diff
00:29:36          grid-rr
00:14:11          grid-render-low
00:03:00          grid-rr-rechunk
00:57:16          grid-render-high-tiff
15:25:37 01:11:47 grid-total

Inc mode

depth
depth inc
Chukchi Sea – Araon 14c 2023

IBCAO v5.0

v4.0
v5.0