update man page to include diversity related parameters about moses HOT 25 CLOSED

I should also add the technical report I wrote about diversity during the Aidyia years (stripped from the non-disclosable content). Another item to add in my Cantorian todo list.

from moses.

i can copy the new stuff from the current help file into the man page. @linas do you remember where the stuff in learning/moses/documentation went so i can update that README.md file as well?

hopefully the snet service will generate more traffic to this repo and the relevant parts of the opencog wiki and i want to fix it up asap.

from moses.

where the stuff in learning/moses/documentation went

I would have moved/copied everything, including the documentation. It's possible, that in all the shuffle, something was lost (it can be found in the deep github history) What I do remember thinking was that some of the original moses documentation was so wildly out of date, that it was just plain wrong -- referring to C++ classes that don't exist, algorithms that weren't implemented, etc. that it was unredeemable. That documentation I threw away. There was a fair amount of it. But I would have thrown that away before you ever used moses, so you shouldn't have any memory of it ...

Should I go hunt down and look through some of this old stuff?

from moses.

The current repo contains this:

$ find |grep pdf
./moses/pleasure/pleasure/PleasureAlgorithm_v3-2.pdf
./doc/moses/QuickGuide.pdf
./doc/comboreduct/ComboReduct.pdf

from moses.

The 2012 version of moses has less documentation:

cd opencog
git checkout 125f5254aa98e64c707dc143484c2eacfbc9362b
git log

shows moses work by Jade in 2012 while

$ find |grep pdf
./pleasure/pleasure/PleasureAlgorithm_v3-2.pdf

so PDF docs were gone by then... let me look earlier

from moses.

Oh, huh... there were html docs in opencog/learning/moses/documentation that I wrote that seem to be gone. Are you thinking of the html docs?

from moses.

uh, no, it seems the html documentation is still there: it's just not in the "docs" directory. Looking at the current git repo:

 $ find |grep html
./moses/moses/documentation/Select_Exemplar.html
./moses/moses/documentation/Genetic_Crossover.html

and when opened in web browser, they display correctly ...

from moses.

So nothing is missing, just unexpected locations maybe

from moses.

I think we should move everything under doc what do you think?

from moses.

Oh, well, actually the README in doc references the other places to find documentation so I suppose it is acceptable to let documentation dispersed. Just the paths should be fixed.

from moses.

i'm referring to the current end of the man page which is duplicated in the opencog wiki:

TODO
       Finish documenting these algo flags: --fs-scorer

         -M  --diversity-pressure  --diversity-exponent  --diversity-normalize
       --diversity-dst --diversity-p-norm --diversity-dst2dp

       -R discretize target var

       These input flags: -G

       Interesting patterns flags: -J -K -U -X

SEE ALSO
       The eval-table man page.

       More information is available at http://wiki.opencog.org/w/MOSES

AUTHORS
       moses was written by Moshe Looks, Nil Geisweiller, and many others.

       This manual page is being written by Linas Vepstas. It is INCOMPLETE.

from moses.

Hah. Let me give you one of my characteristic (patent-pending) trademark answers.

I vaguely recall writing the above. There were not one, but two different efforts to promote diversity, my own, and one that Nil undertook. My own efforts were incomplete and inconclusive. I never fully understood what Nil did. I'm not sure if the above --diversity- flags control my code, or Nil's code. If its my code, then they "do something", that "should improve" speed of convergence/quality-of-result but it was unclear that it "worked well". That's the nature of unfinished work. Promoting diversity should improve results, just on vague theoretical abstract grounds. Actually making it work well in practice is harder.

The other flags -J -K -U -X were left-overs from Moshe Looks original code. They did ... something. It seemed ... interesting. Except it wasn't that interesting, didn't seem that useful, that promising. I did not explore these flags any further, because I decided (rightly or wrongly) that there was nothing promising in that direction.

Instead of documenting these, a valid alternative might be to remove them from the code. It is possible that they have already been removed from the code. It's possible that they are connected to code that doesn't actually work -- code that scrambles and garbages up results. In the "nice try, bad idea" category.

Its a lot of work to convert a "neat idea" into code that actually does something useful, and not all neat ideas can be converted. I believe that the undocumented flags are all "neat ideas" that have not reached the endpoint "useful code". Like hikers in the Himilayas, some died along the way. Maybe a few can still be rescued.

from moses.

well we are attempting to use what ever is actually in the codebase to increase model diversity so if this stuff is "experimental" that should be noted in the help listing and updated in the man page:

$ moses | grep -A4 diversity
  --diversity-pressure arg (=0)         Set a diversity pressure on the 
                                        metapopulation. Programs behaving 
                                        similarily to others are more 
                                        penalized. That value sets the 
                                        importance of that penalty (from 0 to 
--
  --diversity-exponent arg (=-1)        Set the exponent of the generalized 
                                        mean (or sum, if --diversity-normalize 
                                        is set to 0) aggregating the penalties 
                                        between a candidate and the set of all 
                                        candidates better than itself (taking 
                                        into account diversity). If the value 
                                        tends towards 0 it tends to the 
                                        geometric mean, towards +inf it tends 
                                        to the max function. If negative or 
                                        null is it the max function.
--
  --diversity-normalize arg (=1)        If set to 1 then the aggregating 
                                        function is a generalized mean. 
                                        Otherwize it is a generalized sum 
                                        (generalize mean * number of elements).
                                        If --diversity-exponent is set to 
                                        negatively then this doesn't have any 
                                        impact as the aggregating function is 
                                        the max anyway.
                                        
  --diversity-dst arg (=p_norm)         Set the distance between behavioral 
                                        scores, then used to determin the 
                                        diversity penalty.3 distances are 
                                        available: p_norm, tanimoto and 
                                        angular.
                                        
  --diversity-p-norm arg (=2)           Set the parameter of the p_norm 
                                        distance. A value of 1.0correspond to 
                                        the Manhatan distance. A value of 2.0 
                                        corresponds to the Euclidean distance. 
                                        A value of 0.0 or less correspond to 
--
  --diversity-dst2dp arg (=auto)        Set the type of function to convert 
                                        distance into penalty. 4 options are 
                                        available: auto, inverse, complement 
                                        and power. When auto is selected the 
                                        function is selected depending on the 
--
  --fs-diversity-pressure arg (=0)      Multiplicative coefficient of the 
                                        diversity penalty (itself being in 
                                        [0,1]).
                                        
  --fs-diversity-cap arg (=100)         Place a cap on the maximum number of 
                                        feature set to consider. If zero, no 
                                        cap is used (Warning: could be very 
                                        slow). Use this to speed up diversity 
                                        computation on feature sets.
                                        
  --fs-diversity-interaction arg (=-1)  Maximum number of interactions to be 
                                        considered when computing the mutual 
                                        information between feature sets. This 
                                        is used in case the number of selected 
                                        features tends to be high compared to 
--
  --fs-diversity-jaccard arg (=1)       Instead of using the expensive mutual 
                                        information between feature sets to 
                                        measure diversity, it uses a cheap 
                                        Jaccard index. In that case the feature
                                        semantic is ignored
  --fs-inc-redundant-intensity arg (=-1)
                                        Incremental Selection parameter. 

from moses.

I can no longer tell. The style of writing sounds like something I'd write, but the names of the flags sound like code that Nil wrote. I think Nil gets the last word on this.

from moses.

It's best if I take care of it since I wrote most of the diversity code.

from moses.

As I'm going through the options I'm realizing that the --diversity-pressure option should be normalized. Currently a decent value depends on the amplitude of the score, which can range from around 1 to very large values (as shown in the diversity report) which is utterly inconvenient to the user. Instead it should be automatically scaled so that the user only need to set it from 0 to 1.

I'd be tempted to break backward compatibility since likely almost nobody has used it. Alternatively, I could add a new option like

--diversity-pressure-autoscale

and recommend that one instead of --diversity-pressure, which is pity because it has a longer name, but that way compatibility is preserved.

What do you guys think?

from moses.

Or I can have a flag

--diversity-autoscale

which would rescale --diversity-pressure, so that the user can enter a value from 0 to 1 without having to worry about the scale of the fitness function.

It does mean the user will typically have to set that extra flag on top of setting the pressure, I don't like that too much, but maybe it's better than having 2 almost the same options, and again it preserves backward compatibility.

from moses.

Also the diversity penalty reported in the behavioral score should be renamed uniformity penalty.

It wouldn't change any option names, diversity pressure, etc, would remain the same, but the term diversity penalty reported in the bscore is really dumb (and so is the person that came up with it).

from moses.

i vote for one less flag over backward-compatibility, although maximum fine-tuning control is in keeping with the spirit of moses and it's 1000+ line help page ;)

from moses.

i vote for one less flag over backward-compatibility

I'm not sure I understand, do you mean

one more flag over bc?

or

one less flag under bc?

Anyway, I thought that bc was more important, especially since moses is a legacy piece that is being superseded by asmoses, and so I went with implementing that flag.

We can always set this flag to true by default the day we crank up moses (or more likely asmoses) major version number.

from moses.

Oh, sorry, I had misunderstood your sentence, you really mean one less flag. Since I've already implemented that flag I'm bending toward keeping it, but if we all agree that it should be removed, I'm OK with that too.

from moses.

The man page can still be improved, but I suppose it's better left for another issue. I'm closing, feel free to reopen @mjsduncan if something has been left out.

from moses.

thanks for the update, nil! i'd re-open if i could but i don't have the permissions to, since this issue title specifically refers to the man page and the man page is linked to in the opencog wiki, so your update isn't reflected in the documentation that has the most exposure.

from moses.

Done as well

https://wiki.opencog.org/w/MOSES_man_page

from moses.

thanks, nil!

from moses.