theochem / diverseselector Goto Github PK
View Code? Open in Web Editor NEWPython library of algorithms for selecting diverse subsets of data for machine-learning.
Home Page: https://selector.qcdevs.org
License: GNU General Public License v3.0
Python library of algorithms for selecting diverse subsets of data for machine-learning.
Home Page: https://selector.qcdevs.org
License: GNU General Public License v3.0
Make an evil example.
The idea would be to find an important function (from the literature) and then sample it very nonuniformly.
We could try the model in the (retracted, but still a model) paper:
https://www.nature.com/articles/s41586-021-04096-9
This part remains to be finished for the first official release.
For the evaluation metrics, I don't think generating the tests is challenging.
But when it comes to generating tests for different algorithms, things become tricky. Duplicating the original results would be good, but the original dataset is not always available. I would think of having a set of molecules with distinct structures/functional groups and see how the algorithm work. For example, we can have alkanes, alcohols, esters, ketones, et al., then we can test if the implemented method is going to work or not.
Do you have any idea on how to make it work? @PaulWAyers @RichRick1 @JuansaCollins @Khaleeh @alnaba1
Thank you!
We will need to add tests for the following function. Moreover, a wrapper function or Python class simplifies the usage of the diversity functions.
Thank you again for your efforts in building the notebook. @marco-2023
I am copying the list of questions as a reminder/placeholder here for reference as I have merged #147.
To do:
When using DirectedSphereExclusion
and OptiSim
, if we set the r not carefully, we may end up having an incorrect number of molecules. Here are some examples.
# SCFP6 and OptiSim with more molecules than required
selected_ids9 = OptiSim(func_distance=lambda x, y: np.linalg.norm(x - y),
r=0.5, k=3, tolerance=2.0, random_seed=42).select(arr=fingerprints_secfp6, num_selected=12)
print(selected_ids9)
graph_mols(input_sdf="demo_mols_2d.sdf",selected_ids=selected_ids9)
# SCFP6 and OptiSim with less molecules than required
selected_ids9 = OptiSim(func_distance=lambda x, y: np.linalg.norm(x - y),
r=1.5, k=3, tolerance=2.0, random_seed=42).select(arr=fingerprints_secfp6, num_selected=12)
print(selected_ids9)
graph_mols(input_sdf="demo_mols_2d.sdf",selected_ids=selected_ids9)
#================================================
# SECFP6 and directed sphere exclusion with more molecules than required
selector = DirectedSphereExclusion(r=0.5, tolerance=5.0, start_id=0, random_seed=42,
func_distance=lambda x, y: np.linalg.norm(x - y))
selected_ids10 = selector.select(arr=fingerprints_secfp6, num_selected=12)
print(selected_ids10)
graph_mols(input_sdf="demo_mols_2d.sdf",selected_ids=selected_ids10)
# SECFP6 and directed sphere exclusion with less molecules than required
selector = DirectedSphereExclusion(r=1, tolerance=5.0, start_id=0, random_seed=42,
func_distance=lambda x, y: np.linalg.norm(x - y))
selected_ids10 = selector.select(arr=fingerprints_secfp6, num_selected=12)
print(selected_ids10)
graph_mols(input_sdf="demo_mols_2d.sdf",selected_ids=selected_ids10)
[num_mols_debugging.zip](https://github.com/theochem/DiverseSelector/files/8884694/num_mols_debugging.zip)
If you want to reproduce the results, please find the attached notebook.
I know this is restricted by the parameter settings of the corresponding algorithms. Do we have a solution to help people get the right number of molecules? Or we can print a warning when the returned number of molecules is not equal to what we asked for and tell people the right direction to help navigate the right parameters (increase or decrease some parameter values).
This is a challenging question and any suggestions would be appreciated. Thank you.
@PaulWAyers @FarnazH @alnaba1 @RichRick1 @Ali-Tehrani @ramirandaq
As pointed out by @FarnazH , it would be good to be able to identify which molecules/samples are in low-density regions (where we have little data) and also to identify regions where there is very little (or no) data.
This is inspired by the https://chemfp.readthedocs.io/en/chemfp-4x/chemfp_diversity.html?highlight=%20MaxMin#heapsweep-picker
and the paper is https://doi.org/10.1016/j.tcs.2015.02.033. Not sure if the efforts are worth this implementation.
Also, some picking methods can be interesting to explore in this webpage.
What do you think of this? @PaulWAyers @FarnazH @alnaba1 @RichRick1 @Ali-Tehrani
Critically review the code in features module, update docstrings, add tests and comments to the code to make it more clear. Fix tests that are failing in DiverseSelector/test/test_feature.py
.
For example, the feature_reader
seems redundant to me. It is just 2 lines of actual code (but we need a lot of likes to explain/document the function), and we can show how to load the data file in an example/notebook or expect the user to know about it. In a nutshell, I don't think making a wrapper function for every functionality from other packages is helpful/useful. It is best (and to our advantage) to show how our package works in conjunction with other packages (in this case pandas
). Feel free to let me know what you think @FanwangM and @PaulWAyers.
We need a list of methods that require a distance matrix as mandatory. Currently, if a distance matrix is not provided, the program in base.py
(line 96-101) will compute the distance matrix, which is unnecessary when a feature matrix is enough. Therefore, can you help list methods that you are taking care of and show which methods must have a distance matrix as input and which methods require the distance matrix as an optional input? Thank you.
Ramon's group has a clever way of doing diverse selector and we have an in-house implementation and should merge it to this repo.
Generate the documentation and host it with our own domain.
To do:
We needs a good demo.
See more discussions in #73. Key points,
Comments from @PaulWAyers
Similarity and scaled similarity (S(i,j)/SQRT(S(i,i)S(j,j))
5 different methods for computing dissimilarity
One of these five (or not?) should be S(i,i) + S(j,j)- 2S(i,j) where S(i,j) is the similarity. This is the distance via a covariance matrix and is important in Gaussian processes.
One can also consider S(i,j)/SQRT(S(i,i)S(j,j)) and compute the distance-squared from that. That's a correlation matrix/distance.
Information related to determinantal point processes can be found at #4.
This is going to be supported in base.py
.
To do:
Shall we add a new module named converter
to convert the similarity and dissimilarity/distance back and forth?
Some metrics only work on binary fingerprints, such as Tanimoto. But some metrics can work for both cases and some metrics work for only for non-binary matrix, such as molecular descriptors. Can you give a list or table to summary this information? We will need this to refactor the metrics
module. Thank you.
@Khaleeh
An essential component of our package is to have a submodule to compute the dissimilarity/distance/diversity/similarity where normally we assume diversity = 1 - similarity
.
A good summary of this is Table 2 in Drug Dev. Res.,72(1):74 - 84, 2011. Given a fact that this is a public repo, I am not going to share the screenshot here.
The Tanimoto coefficient is a classic and gold stand similarity metric for molecular fingerprints and we should support it. But it was found that it favors small molecules when used in molecule subset selection and a modification was proposed. My guess is that we should make this the default for molecular fingerprint inputs.
I will keep adding new things. Any idea will be appreciated.
@PaulWAyers @JuansaCollins @RichRick1 @Khaleeh @alnaba1
Using the right keywords may help us identify good pieces of literature. Here is a list of keywords that @PaulWAyers and I have used,
1. chemical diversity;
2. compound selection;
3. maximum diversity molecule;
Please feel free to add new keywords if you think that's helpful. If you are adding, adding the numbering would be helpful and I will constantly update the list here for future reference.
Thank you. @Khaleeh @alnaba1 @JuansaCollins @RichRick1
We will add the functionality to support measuring the subset diversity.
"Simple" stratified sampling can be mimicked by using the propert(ies) of interest as the feature(s). (So the feature vector is exactly the propert(ies) to be sampled.)
To achieve sampling with respect to target properties and diversity with respect to a fingerprint/feature-vector, you need to append the target properties to the feature vector. For each property, specify a "repeat": the more times you repeat the property value the higher its weight.
One way to do this would be to pass a weight, then set the repeat-value for each property to be repeat = np.ceil(weight*n_features)
where n_features
is the number of features (length of the fingerprint). (I use ceiling to ensure that as long as weight > 0.0 at least one instance of the property is used). The final feature vector has the property value(s) appended to it repeat
times.
Then we use this set of features.
Add a utility for augmenting the feature array and provide examples to showcase this.
For example, we could measure how much more diverse a selected sample is than a random sample.
If clusters are similar size, then random-ish samples are a lot better than if clusters are very inhomogenous in size.
This is good to have, but let's leave it to the last stage. An example can be found at https://github.com/theochem/B3clf.
This would help people follow a consistent pattern writing codes and making PRs.
The current version of metrics.py
has implemented several functions and maybe that's too many for people to memorize. So we will end up with a simpler API. I know you may not able to join the hackathon next week. Can you update the status of this module? @Khaleeh
Thank you for the nice work!
Not just minimum (p=-infinity) and sum (p=1)) versions, but also we need to support p-sums.
This relates to #4.
This would help people to load different molecule formats.
Sometimes, this function can return us an invalid value, NaN. You can reproduce the error by running the notebook as attached with the latest code in this repo. I think the main problem is we may have zeros in the denominator.
Can you fix this? Thank you.
@Khaleeh
This method is described in J. Chem. Inf. Comput. Sci. 1999, 39, 1, 51–58.
Finalize the functions in distance module by:
The function optimize_radius
in the file utils.py is not using the cluster_ids
parameters. The select_from_cluster
method of the DirectedSphereExclusion
is also not using the cluster_ids
. Because of this, this selection algorithm fails when used with cluster labels.
The utils.py has a large to-do block here Is implementation of this is still needed?
Also, mol_loader() needs tests written for it in test_util.py
The original attempt is to have one line of code to perform the subset selection, but this leads to a large number of arguments. Therefore, I would propose to have two independent modules where we can compute the molecular features and do the subset selection indepedently.
We will need to make it accessible with pypi
and conda
.
We need sensible default values for the grid/sphere size in some of the selection algorithms.
We should also allow for the case where the number of selected molecules is too small. We can (easily?) optimize sphere-size, for example, using the secant method (evaluating at the closest integer) so that the right number of molecules is selected.
(related): As suggested by @FarnazH , we should allow for sphere size to be dictated by the number of molecules, rather than just a size.
For some parts of our package, the documentation is unclear. Can you help double check if the parts you are working on are missing such clear documentation and fix them?
Raises
if there is any error checkingNotes
section. We will need to have equations explicitly expressed in the documentation.If you can think of other things that we should use for cleaning the codes, please comment here. Thank you.
@alnaba1 @RichRick1 @Khaleeh @fwmeng88
This is a method developed by Weitao's group, J. Chem. Inf. Model. 2015, 55, 3, 529–537, which relates to #4.
Implement methods listed in as similarity module https://vlachosgroup.github.io/AIMSim/implemented_metrics.html. Please add detailed documentation to show which similarity functions is corresponding to which distance functions in scikit-learn or scipy.
One question I have shall we separate the similarity and distance measurements? I get confused by some measurements, e.g. Tanimoto index of molecule fingerprints. I would see it as a distance, but they treated it as a similarity, https://vlachosgroup.github.io/AIMSim/implemented_metrics.html. If we decide to distinguish them, we may need to make them into similarity
and distance
modules instead of one module.
@FanwangM can you explain the rationale behind developing the functions euc_bit() and bit_tanimoto()? I have added tests in test_distance.py that demonstrate that they produce the exact same results as using the euclidean distance from sklearn and tanimoto().
The only location I see them used is in nearest_average_tanimoto(). If there is somewhere else in DiverseSelector that they are essential, I can leave them. But otherwise, I think it would make sense for me to replace the bitstring functions with the normal ones and streamline the distance.py module.
It would be beneficial for people to know how they can use our package for practical problems and I think we can make a list for people in the manuscript.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.