Refs https://github.com/mapbox/pxm/issues/622.

Let's replace pxm-alpha with rio-alpha. I think @dnomadb has already done some of this work.

There are a number of different ways that nodata can be determined. Depending on how generalizable we need to make this tool, we should consider each of these scenarios.

• The dataset has internal nodata value set in metadata
• The dataset has an alpha band
• The dataset has an internal binary mask (formats like GTiff allow a mask that is not considered a band)
• The dataset has an external binary mask (a *.msk file alongside the main raster)
• The user of the tool supplies the nodata value, either as a single value or a list of per-band values. Useful for when the dataset doesn't say anything about it's nodata situation.

The first four can be handled with the new dataset_masks method in Rasterio which normalizes all of these cases into a single 2D array that can serve as an alpha band.

The case of user-input nodata value would need to be handled separately and would take precedent. Something like:

with rasterio.open(raster) as src:
    if user_nodata:
        alpha = get_alpha_mask(..., nodata=user_nodata)
    else:
        alpha = src.dataset_mask()

• See also http://mapbox.github.io/rasterio/topics/masks.html#nodata-representations-in-raster-files

Or something similar, not an option flag: https://github.com/mapbox/rio-alpha/blob/master/README.md#islossy

cc @jacquestardie @virginiayung

Target date: Thursday, July 26.

_alpha_worker is useful publically, so we should make it officially so:

TODO

• _alpha_worker --> alpha_worker
• Improve docstring for g_args parameter.

cc @perrygeo

Refs https://github.com/mapbox/pxm/issues/622.

• Migrate pxm-findnodata
• Add tests
• Replace argparse with click

Seeing this error with 0.5.0. Probably a regression I introduced in 0.5 😬 Tests didn't catch it because we never tested ndv value with rgba.

File "/usr/local/src/service-worker/venv/local/lib/python2.7/site-packages/rio_alpha/alpha_mask.py", line 20, in mask_exact
    alpha = np.any(np.transpose(img, [1, 2, 0]) != ndv, axis=2)
ValueError: 'axis' entry is out of bounds

PR: #14
Refs https://github.com/mapbox/pxm/issues/622.

• Migrate pxm-alpha to alpha_lossy and alpha_non_lossy modules
• Refactor alpha_lossy and alpha_non_lossy modules
• Add alpha_lossy and alpha_non_lossy command to cli.py
• Tests:
  • property-based
  • test_cli
  • test with fixtures (Refer to example of testing with flex-compare in rio-toa)
• Update docstrings + README
• Clean up, Rebase
• Review 👀
• Merge PR #14 into master
• Update version to 0.3.0

cc @jacquestardie

Let's add CircleCI and Codecov to this repo.

Add Readme and docstrings for each of the following commands in rio-alpha:

• islossy
• alpha
• findnodata

cc @jacquestardie

Running test_cli_findnodata_debug_success fails if you don't install matplotlib first. We should list it as a dependency, possibly a soft "extra" dependency.

$ rio findnodata tests/fixtures/fi_all/W4441A.tiny.tif --discovery --debug
...
ImportError: No module named matplotlib.pyplot

So no need to pass it as a separate option https://github.com/mapbox/rio-alpha#alpha

@jacquestardie @virginiayung

Using the geotiff here: https://github.com/mapbox/rio-alpha/blob/roll-with-it/tests/fixtures/landsat/LC80460272013104LGN01_l8sr.tif

$ rio alpha tests/fixtures/landsat/LC80460272013104LGN01_l8sr.tif /tmp/foo.tif

Raises an exception of ValueError: rollaxis: start (4) must be >=0 and < 4

Traceback (most recent call last):
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/riomucho/__init__.py", line 31, in manualRead
    return work_func(srcs, window, ij, global_args), window
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha.py", line 66, in _alpha_worker
    g_args['ndv'])
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha.py", line 32, in calc_alpha
    alpha = mask_exact(rgb, ndv)
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha_mask.py", line 21, in mask_exact
    alpha = np.any(np.rollaxis(img, 0, depth) != ndv, axis=2)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/numpy/core/numeric.py", line 1452, in rollaxis
    raise ValueError(msg % ('start', start, n + 1))
ValueError: rollaxis: start (4) must be >=0 and < 4
Traceback (most recent call last):
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha.py", line 120, in add_alpha
    rm.run(processes)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/riomucho/__init__.py", line 109, in run
    for data, window in self.pool.imap_unordered(reader_worker, self.windows):
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/riomucho/single_process_pool.py", line 9, in imap_unordered
    yield reader_worker(i)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/riomucho/__init__.py", line 34, in manualRead
    raise e
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/riomucho/__init__.py", line 31, in manualRead
    return work_func(srcs, window, ij, global_args), window
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha.py", line 66, in _alpha_worker
    g_args['ndv'])
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha.py", line 32, in calc_alpha
    alpha = mask_exact(rgb, ndv)
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha_mask.py", line 21, in mask_exact
    alpha = np.any(np.rollaxis(img, 0, depth) != ndv, axis=2)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/numpy/core/numeric.py", line 1452, in rollaxis
    raise ValueError(msg % ('start', start, n + 1))
ValueError: rollaxis: start (4) must be >=0 and < 4
Traceback (most recent call last):
  File "/Users/mperry/env/mapbox35/bin/rio", line 11, in <module>
    load_entry_point('rasterio', 'console_scripts', 'rio')()
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/mperry/work/rio-alpha/rio_alpha/scripts/cli.py", line 77, in alpha
    workers)
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha.py", line 120, in add_alpha
    rm.run(processes)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/riomucho/__init__.py", line 109, in run
    for data, window in self.pool.imap_unordered(reader_worker, self.windows):
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/riomucho/single_process_pool.py", line 9, in imap_unordered
    yield reader_worker(i)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/riomucho/__init__.py", line 34, in manualRead
    raise e
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/riomucho/__init__.py", line 31, in manualRead
    return work_func(srcs, window, ij, global_args), window
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha.py", line 66, in _alpha_worker
    g_args['ndv'])
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha.py", line 32, in calc_alpha
    alpha = mask_exact(rgb, ndv)
  File "/Users/mperry/work/rio-alpha/rio_alpha/alpha_mask.py", line 21, in mask_exact
    alpha = np.any(np.rollaxis(img, 0, depth) != ndv, axis=2)
  File "/Users/mperry/env/mapbox35/lib/python3.5/site-packages/numpy/core/numeric.py", line 1452, in rollaxis
    raise ValueError(msg % ('start', start, n + 1))
ValueError: rollaxis: start (4) must be >=0 and < 4

Rather than alphaing living (semi-randomly) in nodata, we should bring that code to a more general tool / repo in the rio plugin namespace.

• Copy over mapbox/nodata#35
• Implement non-lossy 1st

In rio_color/islossy.py, we're using the neighborhoods parameter, which has been deprecated. We should use connectivity instead.

I tried replace neighborhoods=4 with connectivity=None, but it failed, which is why I'm ticketing this in the first place.

👍

No more Rasterio deprecation warnings and more.

In #9, we list a number of different implementations of nodata masks. It would be handy to have a tool to convert between these representations with a CLI.

As an example, I wrote a script to convert RGBA tiffs to RGB + internal mask - fleshing that out a bit more might be useful. We could do conversions to and from RGBA, RGB + nodata, and RGB + internal/external mask.

rio-alpha may or may not be the place for this but putting it here for consideration...

Refs https://github.com/mapbox/pxm/issues/622.

Let's migrate pxm-islossy here, test it, and swap out argparse for click.

Tl;dr

ndv values need to follow the standard format: "[int(ndv), int(ndv), int(ndv)]" or int(ndv).

Currently, there are two ways to represent ndv in pxm and we rely solely on bash to handle the parsing of ndv strings:

1. single integer
2. 3 separate integers (For example: The ndv values for chilliwack_ortho are 255 255 255)
   The second case is throwing a parsing error when we integerate rio-alpha into pxm-prep, causing pxm-prep.test.sh to fail:

This is caused by two different ndv values formats:

rio islossy accepts "[255, 255, 255]", while rio findnodata outputs "255 255 255"

islossy takes either 1) a single integer or 2) a string of a list of 3 integers that are separated by ,. It uses two functions _parse_ndv and _parse_single to handle both cases and parse the nodata values. We did not rely on Click to handle both input cases (single integer vs. string of list of 3 integers) because Click option only supports a fixed number of arguments. If we set --ndv to have nargs=3, it will throw an error if I pass it only a single integer and vice versa:

We have a couple options:

1. Change pxm-prep script and rio findnodata output format from "255 255 255" to "[255, 255, 255]".
   Similar to pxm-islossy.test.sh:

from `ndv="$1 $2 $3"` and  `--ndv $ndv`

to `ndv="[$1, $2, $3]"` and `--ndv "$ndv"`

2. Change pxm-prep script and rio findnodata output format from 255 255 255 => "255 255 255", also change rio islossy input format from "[255, 255, 255]" to "255 255 255"

from `--ndv $ndv` 
to `--ndv "$ndv"`

3. We can then pass any number of ndv by utilizing click multiple input options:
   Change rio islossy input from --ndv 255 --ndv 255 --ndv 255
   ~~and pxm to --ndv $1 --ndv $2 --ndv $3. ~~
   This method is too messy

cc @jacquestardie

If we've still got the right credentials in Travis settings, it's just a matter of pushing a new tag.

This:

def _convert_rgba(rgb_orig):
    ...
    mod = int(math.ceil(rgb_orig[:, :, 0].shape[min_dimension] / 200))

won't work properly on Python 2 because of integer division.

$ python
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec  5 2015, 12:54:16)
>>> import math
>>> math.ceil(700 / 200)
3.0

The fix is to use the new division operator. It's a good idea to put from __future__ import division at the top of every Python file that does division with anything that could potentially be an int:

>>> from __future__ import division
>>> math.ceil(700 / 200)
4.0

Python 3 does the right thing by default

Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44)
>>> import math
>>> math.ceil(700 / 200)
4

This breaks a couple tests so they'll need to be updated.

Using black.

Bug fixes like #53 .

Target date: 2021-06-28

• Close tickets in milestone
• Update version in rio_alpha/__init__.py to "1.0.2"
• Update change log
• Commit and tag "1.0.2"
• Push tag and check https://pypi.org/project/rio-alpha/#history for new entry

https://github.com/mapbox/rio-alpha#alpha for a CLI tool, what is a list? This needs (a) a clearer explanation (stringified list? idk) in the help string, and (b) possibly an example (eg '[0, 0, 0]')

cc @jacquestardie @virginiayung