Notes

• CR = compression ratio = # bits after / # bits before . Smaller compression ratio is better.

Methods

Compressors evaluated

All losssless.

• Heavy:
  • JPEG 2000 (lossless variant)
  • LZW
  • DEFLATE
  • etc. — I included all the ones included by the studies in the experiment design spreadsheet
• Light:
  • Physical only:
    • bitpacking
    • PFOR
    • RLE
  • Logical → physical
    • For every logical transformation, there are 3 different compressors, formed by applying the above physical compressors on the result.
    • Note that since the physical compressors expect unsigned input, ZigZag encoding is applied on the logical transformation result before being passed to the physical stage.
    • Logical transformations:

      • delta

      • 2x delta

      • FOR

      • RLE

      • Predictors — I added these as they’re cheap to compute and very effective. I simply used the collection of predictors that JPEG and PNG use, as they’ve already solved the problem of predictor choice and are very mature codecs.

        

        Neighbourhood pixel identifiers.

        • Based on lossless JPEG: predictors 1 to 7 inclusive in the below table.

          

        • Based on JPEG-LS: Median Edge Detection (MED) / LOCO-I predictor

          

        • Based on PNG: Paeth predictor

          

Datasets evaluated

• Based on relevant papers / reputable libraries from the experiment design spreadsheet. These datasets are constrained to the particular region of the source for comparability:
  • SRTM
  • ETOPO1
  • Custom orthographic DTM
  • Landsat8 bands
  • Sentinel2 bands
• Custom

  • I formed a diverse collection of custom datasets with different distributions / compressibility.

  • These are based on the datasets from the relevant papers / libraries (but extended to more diverse regions), and generally popular ones in our domain.

  • Each dataset is the union of equally sized TIFs over the following regions:

    • NYC (urban)
    • Iowa (agriculture)
    • Himalayas (mountains)
    • Sahara (desert)
    • Amazon (tropical forest)
    • North Atlantic (ocean)

  • Note that some datasets have entirely nodata in some of these regions (e.g. SRTM @ ocean). Those regions are excluded in those datasets.

  • **Datasets by category (in the order of least compressible → most compressible): ** estimated, lossless

    Category	Dataset
    Raw satellite imagery	Landsat8 bands 1, 2, 3, 4, 5, 8 (optical imagery)
    Raw satellite imagery	Sentinel2 bands 2, 3, 4, 8 (optical imagery)
    Derived (processed satellite imagery)	SRTM (DEM)
    Derived (processed satellite imagery)	SRTM High Resolution (DEM)
    Derived (processed satellite imagery)	ETOPO1 (DEM)
    Derived (processed satellite imagery)	ETOPO1 High Resolution (DEM)
    Derived (classified satellite imagery)	WorldCover (land classification)
    Modelled / interpolated	WorldClim (temperature & precipitation)
    Modelled / interpolated	LandScan (population)

• Original TIFs from the MRes project
  • These are considered “old” at this point, but still useful for comparison w/ the project results
  • Some of them are very sparse, exagerrating speedups

Lossy approximation

As some of the datasets are not very compressible through lightweight (or even heavy!) means, and our technique only gives speedups on compressible datasets, I explored ways to transform them such that they lose some detail but become more compressible.

For many tasks, some error is acceptable, especially since geospatial data already has error from sensor noise etc. — so we can sacrifice some error for broader applicability of our technique.

As an intended application of our technique is scientific computation (e.g. computing statistics like NDVI), perceptual denoising (like in JPEG) is not appropriate as it can throw off downstream calculations unpredictably.

There has been a lot of work in the area of bounded denoising [1], and a mature approach in the geospatial domain is LERC. Median filtering is another simple approach with a good result:complexity tradeoff (this article compared it to more complex approaches like BM3D and median actually won out alot with geospatial data). I tried both.

LERC is especially desirable as the errors are bounded by the MaxZError parameter (this article gives a nice summary of the impact of the parameter). It maps very well to our lightweight approach; the “compression” part of it is just bitpacking, which is one of our physical-level compressors.