Global mapping of the Seabed¶

Seabed 2030 project

Compile a map of the global sea floor¶

Depth, bathymetry¶

Data sources¶

Input data¶

In [5]:
df.head()
Out[5]:
x y z sid priority
0 760232.463937 8.839106e+06 -363.500000 1 1000
1 761019.073307 8.839097e+06 -363.450012 1 1000
2 761805.643957 8.839089e+06 -362.790009 1 1000
3 762592.160114 8.839080e+06 -361.049988 1 1000
4 763378.633532 8.839071e+06 -358.989990 1 1000
In [6]:
len(df)
Out[6]:
26067243872
In [7]:
df.dtypes
Out[7]:
x           float64
y           float64
z           float32
sid          uint16
priority     uint64
dtype: object

Data size¶

  • 26,067,243,872 data points, each 30 bytes
  • 729 GB

Multiple sources¶

In [9]:
meta.sources
Out[9]:
file priority sid tid coverage size centre crs mtime ftime ... xmax ymin ymax count count-0 count-100 count-200 count-400 count-800 count-1600
0 05874.xyz 10 12555 10 39 5011 awi 9354.0 2022-10-22 02:32:17.578000+00:00 2022-10-19 12:25:33+00:00 ... -4572600.0 -8126300.0 -7938000.0 232.0 NaN 232.0 NaN NaN NaN None
1 19910019.xyz 10 12556 10 341 20295 awi 9354.0 2022-10-22 02:32:17.578000+00:00 2022-10-19 12:28:34+00:00 ... -4792600.0 -7996300.0 -7961000.0 939.0 NaN 939.0 NaN NaN NaN None
2 19910024.xyz 10 12557 10 30 17094 awi 9354.0 2022-10-22 02:32:17.578000+00:00 2022-10-19 12:28:35+00:00 ... -4533400.0 -8112900.0 -7061600.0 760.0 NaN 471.0 44.0 245.0 NaN None
3 19910025.xyz 10 12558 10 137 30631 awi 9354.0 2022-10-22 02:32:17.578000+00:00 2022-10-19 12:28:35+00:00 ... -4490300.0 -8307200.0 -7981400.0 1422.0 NaN 1422.0 NaN NaN NaN None
4 19910026.xyz 10 12559 10 12 11439 awi 9354.0 2022-10-22 02:32:17.578000+00:00 2022-10-19 12:28:35+00:00 ... -4489300.0 -8013600.0 -7062200.0 506.0 NaN 236.0 142.0 128.0 NaN None
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
4988 ngdc_bathy_guam_100m.tif 0 15354 141 135055 4195534 niwa NaN NaT 2022-03-05 01:22:33+00:00 ... 11448500.0 2012600.0 2149200.0 891736.0 NaN 373835.0 392896.0 125005.0 NaN None
4989 ngdc_bathy_tutuila_100m.tif 0 15355 141 4268 656158 niwa NaN NaT 2022-03-05 01:22:33+00:00 ... -13439500.0 -2225000.0 -2191800.0 115087.0 NaN 69906.0 43228.0 1953.0 NaN None
4990 ngdc_bathy_wakeisland_100m.tif 0 15356 141 2985 393982 niwa NaN NaT 2022-03-05 01:22:33+00:00 ... 13153100.0 2938600.0 2972400.0 93054.0 NaN 15458.0 43726.0 33870.0 NaN None
4991 olex_Natl_15s_v2_A.TIF 583 15359 17 427312 8536687 ldeo NaN NaT 2022-04-20 17:47:18+00:00 ... -3858900.0 -100.0 4477800.0 523432.0 NaN 519981.0 3451.0 NaN NaN None
4992 olex_SInd_15s_clean_v2.tif 583 15361 17 1420518 37157758 ldeo NaN NaT 2022-04-20 17:47:34+00:00 ... 11038500.0 -6868000.0 -100.0 1274722.0 NaN 1170009.0 99070.0 5643.0 NaN None

4993 rows × 23 columns

Data categories¶

In [10]:
meta.categories
Out[10]:
Data categories
10 Singlebeam
11 Multibeam
12 Seismic
13 Isolated sounding
14 ENC sounding
15 Lidar
16 Depth measured by optical light sensor
17 Combination of direct measurement methods
140 Predicted based on satellite-derived gravity d...
141 Interpolated based on a computer algorithm (no...
142 Digital bathymetric contours from charts (not ...
143 Digital bathymetric contours from ENCs (not in...
44 Bathymetric sounding
145 Predicted based on helicopter/flight-derived g...
146 Depth estimated by calculating the draft of a ...
170 Grid compilation including interpolated (not i...
171 Unknown source (not included in total)
172 Steering points (not included in total)
100 Land (negative topography, not included in total)
110 Upcoming, not yet received, (not included in t...
111 Upcoming, received, (not included in total)
112 Upcoming, in moratorium, (not included in total)
113 Upcoming, processing, (not included in total)
114 Completed, (not included in total)
115 Deprecated, (not included in total)
116 Lost, (not included in total)

Python¶

Jupyterlab¶

Jupyterlab

Numpy¶

Numpy

Pandas¶

Pandas

Dask¶

Partitioning¶

Dask partitioning

Dask¶

Distributed computation¶

Dask indexing

aws¶

aws

Calculation steps¶

Calculation steps

Running stats¶

  • 00:00:03 stats-check-data
  • 02:19:33 stats-read-data len 26067243872
  • 00:04:29 stats-bin
  • 00:40:48 stats-index
  • 02:50:40 stats-blockmedian
  • 00:01:20 stats-pyramid-filter-bands
  • 00:00:35 stats-pyramid
  • 00:01:13 stats-pyramid-rechunk
  • 00:05:28 stats-render-low
  • 00:01:26 stats-bincount
  • 00:01:26 stats-bincount-count z: 25.000046399606966 %, s: 26.818262095253996 %
  • 00:19:34 stats-extent
  • 00:00:02 stats-agg
  • 00:38:05 stats-regions-centre
  • 00:12:00 stats-regions-rhc
  • 00:14:10 stats-regions-eez
  • 00:37:09 stats-render-merged-projection
  • 08:34:51 stats-total

Running grid¶

  • 00:11:59 grid-concat len 29799723872
  • 00:04:22 grid-gebco-weird-projection-9354
  • 00:04:58 grid-gebco-weird-projection-3857
  • 00:04:05 grid-gebco-weird-projection-3996
  • 00:15:28 grid-gebco-weird-projection
  • 00:15:29 grid-gebco-weird-projection-read-data
  • 00:04:39 grid-bin
  • 01:22:40 grid-index
  • 00:17:42 grid-update-json-weirdproj-extent
  • 00:00:01 grid-update-json-weirdproj-extent-write-json
  • 01:07:11 grid-blockmedian-400-m
  • 00:48:04 grid-blockmedian-500-m
  • 00:18:21 grid-blockmedian-250-m
  • 00:16:15 grid-blockmedian-100-m
  • 06:23:08 grid-spline-lowres
  • 00:01:59 grid-spline-lowres-cropped
  • 00:26:40 grid-resample-low-res
  • 00:12:28 grid-resample-high-res
  • 00:23:56 grid-diff-buffered
  • 01:13:54 grid-surface-rr-diff
  • 00:29:36 grid-rr dimensions: (x: 400800, y: 400800)
  • 00:14:11 grid-render-low
  • 00:03:00 grid-rr-rechunk
  • 00:57:16 grid-render-high-tiff
  • 15:25:37 grid-total