Filters

Gwyddion AFM specific flatten_base, polylevel, align_rows, zero_mean, scars_remove, level filters. These would probably have to be rewritten, but are pretty simple.

Also Gaussian filter (in scikit image).

There are multiple hard coded variables throughout the code that are important for optimising processing of different samples or need to changed each time the code is executed (e.g. path to input files).

These need to be identified and listed to facilitate writing a command line interface.

Connected components (grains)

Gwyddion has grains_mark_height, grains_remove_touching_border, grains_remove_by_size, number_grains, grains_invert, grains_intersect, grains_get_values

Would need to lean on https://scikit-image.org/docs/dev/api/skimage.morphology.html, https://scikit-image.org/docs/dev/api/skimage.measure.html, https://scikit-image.org/docs/dev/api/skimage.measure.html#skimage.measure.regionprops

After #66, #67 and #68 are done, take out all the Gwyddion references and replace with the new code.

Some code has docstrings, a readthedocs site will compile these into documentation e.g. https://simple-network-sim.readthedocs.io/en/latest/?badge=latest

Path joins are being done as string manipulations rather than os.path.join. This causes issues in windows when input path includes \ or if the input files are nested in a folder within the path file.

Will try and solve by using os.path.join

Please put ideas here!

Is your feature request related to a problem? Please describe.
We are working with shapes that approximate to the outline of simple geometric objects e.g. the outline of a square. Depending on the size of the AFM tip shapes appear larger or smaller. This can be corrected experimentally, but we'd like to be able to compare data across different tip sizes.

Small tip left, big tip right.

Describe the solution you'd like
Fitting the outline of the shape at it's highest point (like a ridge running round the shape) might give us a true object size that's independent of tip size.

Describe alternatives you've considered
Another approach would be deconvolution, but this would maybe need the tip size which we don't generally know.

There are some unhelpful/undescriptive comments in dnatracing.py that need to be impoved e.g.:

Change the GitHub settings such that people aren't allowed to push directly to the master branch; instead they must make a new branch, do their development and then ask for this to be merged into master via a "Pull Request". This allows multiple people to work on new stuff at the same time.

In filters.py , in the function row_col_quantiles(image: np.array, binary_mask: bool = False), the binary mask checking no longer works, since later on, the script checks if a mask was provided using: if binary_mask:, which fails if binary_mask is of type np.ndarray. I am not sure how to solve this, as this argument needs to be able to take two types of argument, and then the script needs to be able to detect if a value is passed with the function for that argument versus if it has not. The previous solution I had for this was to use: … , binary_mask = None, along with if binary_mask is not None. We could use type checking to see if the binary_mask argument is of type np.ndarray , however this would still not allow for good type-hints.

We could use Unison() to allow for more than one type to be hinted, however this requires an import and may not be best practice.

On Windows, these might form the basis, but need to be checked.

Install

• Install 32 bit Anaconda (https://www.anaconda.com/distribution/#windows) - if you have 64 bit installed as well, make sure you’re using the 32 bit version
• Install 32 bit Gwyddion for Windows (http://gwyddion.net/download.php#stable-windows)
• Download the additional prerequisites (https://sourceforge.net/projects/gwyddion/files/pygtk-win32/)
• Use Anaconda Navigator to import an environment from gwyconda.yml (in the Google Drive)
  Identify the path to the copy of Python in the new conda environment. A way to do this:
  • Open Anaconda Powershell prompt
  • Type conda activate gwyconda (this puts you in the correct environment)
  • Type Get-Command python
• Install (for all users) from the Google Drive Folder making sure to select the folder containing the version of python identified in the previous step
  • pycairo-1.8.10.win32-py2.7.msi
  • pygobject-2.28.3.win32-py2.7.msi
  • Pygtk-2.24.0.win32-py2.7.msi

Test

• Open Anaconda Powershell prompt
• Type conda activate gwyconda (this puts you in the correct environment)
• Type cd 'C:\Program Files (x86)\Gwyddion\bin' or cd <your_gwyddion_path>\bin
• Type python (you should see something like:

Python 2.7.16 |Anaconda, Inc.| (default, Mar 14 2019, 15:49:40) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

• Type import gwy

If you get no errors, you should be good to go…

Current otsuthresholdgrainfinding uses Gwyddion, could (maybe) use from skimage.filters import threshold_otsu.

Great to see the engagement here! #26

To make things easier in future, I think we should enable discussions for, well, discussions. And keep issues for bug reports, enhancements and feature requests. This means that we'd need, at least, some guidance in the readme on this, and perhaps later some issue templates. What do you think @alicepyne? I can turn on discussions with a button push - perhaps someone else in the team could look at the readme?

argparse (https://docs.python.org/3/library/argparse.html) is a od way of turning command line arguments into python variables.

Check existing contributing guidelines and revise to reflect current good practise.

It can be overwhelming and at times confusing about what branches exist for what purpose and who they are created by.

To help alleviate this standardising the nomenclature used in creating branches to be informative can be useful and documenting this in the contributing.md would be appropriate.

One approach to branches is to use the structure of <username>/<github-issue>-<concise-description> which tells anyone...

1. Who created the branch based on their GitHub username (<username>).
2. What GitHub issue ot pertains to (<github-issue>).
3. Some information about the issue.

As an example if I were to create a branch to address this issue I would call it ns-rse/##-contributing-nomenclature. Long and verbose branch names should be an issue when using git at the command line since it includes auto-complete via Tab.

This is just one example of standardised nomenclature and is meant to serve as a starting point for discussion before being the settled upon.

How to cite (title, authors, journal) + link:

https://figshare.shef.ac.uk/articles/software/TopoStats_-_an_automated_tracing_program_for_AFM_images/13103327/1https://figshare.shef.ac.uk/articles/software/TopoStats_-_an_automated_tracing_program_for_AFM_images/13103327/1

To be added to readme.

The colour gradient file used in the code isn't in available in the repository and there is no instructions on how to install it.

Considerable progress has been made on installation instructions, but we still lack a good user guide. 😢

Useful basis material here:

• Guide.on.pygwytracing.pdf
• https://www.sciencedirect.com/science/article/pii/S1046202321000207

The user instructions might ultimately live in readthedocs which is underway for our dev branch: https://topostats.readthedocs.io/en/dev/?badge=dev

The instructions should explain the parameters in the config files https://github.com/AFM-SPM/TopoStats/blob/master/default_config.ini

Some code e.g. skeletonisation could perhaps be removed and replaced with an implementation in an external module, (if one can be found!). It would be useful for someone to look through the code and find opportunities for this...

Add headings and other nice formatting.

Use pySPM (perhaps optionally Nanoscope).

Describe the bug

• When TopoStats processes the images, a single pixel border is added to the image array which has a value of 0.
• i.e. For a 1024x1024px image, the processed tiff without any scale bars/rulers is 1026x1026px. 512x512px spm's become 514x514px

To Reproduce

• Run TopoStats as usual (xvfb-run python pygwytracing.py) with the rulers and scale bars off in the config file.
• Config file attached (as .txt as .ini not supported)
  config.txt

Expected behavior

• Output image dimensions should be the same as input image dimensions

Output

• Screenshot attatched showing difference in input and output size.

(Not required if using Docker) Your computer configuration (please complete the following information):

• Docker used

YAML, TOML and JSON are potential options for this. TOML is probably the most readable, YAML is more broadly used and JSON is very widely used but has a some extra syntax which might make it harder to read. The first step down this road would be to capture what inputs are required to make topostats run. Maybe start listing these below?

Identify functions, classes, etc. and develop tests. A unit test is akin to checking that the brakes work on a car, as opposed to a regression test which would be driving the car to Bologna. pytest is a widely used test framework.

Due to the relative file path used when the path variable is set (line 717), the savestats function (line 906) does not find a place to save this folder and contents as the path returned due to os.path.splitext(os.path.basename(directory))[0] is (line 509).

This can be resolved by adding path = os.path.abspath(path) just below where the path is set as it doesn't change full paths, only those starting with . or a directory name. However, I don't know how this works on Windows machines due to the non-unix-like file paths.

There are multiple functions on plotting the same type of graph. This should be reorganised to make the code clearer and avoid duplicates.

There are three different scripts concerning the plotting of graphs and some of the functions they contain are redundant.

To run on myBinder and on Google Colab. Running on these platforms will (almost certainly) require python 3 and no Gwyddion dependency.

This function uses Gwyddion to save data:

It should be re-implemented using e.g. matplotlib.

When the code doesn't detect any grains in an spm file the image isn't processed because of:

` # if there's no grains skip this image

        if number_of_grains == 0:

            continue`

in the main script.

If this is commented out then no cropped_grains are made and
` # Create cropped datafields for every grain of size set in the main directory

        bbox, orig_ids, crop_ids, data, cropped_grains, cropwidth_pix = boundbox(cropwidth, datafield, grains, dx,
                                                                                 dy, xreal, yreal, xres, yres)`

also needs commented out too, which prevents cropped images being created later.

It would be helpful if all images were processed whether or not they had grains.

Package this repository and make it available on PyPi (and maybe also conda cloud) so it can be used without cloning i.e. via pip install topostats.

For circular molecules with crossings or narrow parts, the algorithm sometimes end the ordering process prematurely when it cannot find the nearest neighbour, leading to the loss of the rest of the unordered points

The functions defined in topostats/filters.py that replace some of the dependency on Gwyddion are a good candidate for refactoring into a class which can then be extended with additional methods such as the recently added remove_x_bowing() and subsequent processing steps that depend on Gwyddion.