Comments (21)
If Boosting was refactored into its own class, I would add this boosting function:
boost-factor = log(activation-frequency) / log(target-sparsity)
I like this function because it has a zero-crossing at activation-frequency = 1
and an asymptote to infinity at activation-frequency = 0
.
from htm.core.
On the topic of using SDRs hold the potential pools:
This could be useful for debugging a custom topology, using the sdr metrics classes. Sparsity is the number of potential connections to each output, and active freq is the number of pot syns per input. Set period >= #outputs for true average instead of exp-rolling avg.
If min-sparsity is zero then an output has no synapses.
If min-AF is zero then input has no synapses.
This wont actually work because the caller (SP) would need to setup the metrics class, but this is something i hope to improve upon in the metrics classes.
Edit: I added this feature to the SDR Metrics classes.
from htm.core.
I think that all of these changes would break API compatibility. That doesnt mean it can't be done, just that it wont be easy.
Another way is subclasses which over ride certain methods. Its not as flexible of a solution but it seems simpler and less disruptive.
from htm.core.
Another way is subclasses
Yes, those classes would not be visible to the public API and SP would call them.
AbstractInhibition
InhibitionLocal
InhibitionGlobal
SP { Inhibition = InhibitionLocal }
something like this.
from htm.core.
That sounds even like a good default, eventhough we'd have to do some tests.
Another motivation for this modularity: while testing performance impact of PR #119 , I figured there are no tests for using local inhibition, at all! The separate class would make that more obvious.
I was thinking about your comment how to make this less intrusive, API-wise.
SpatialPooler(..., inhibition*=nullptr)
would allow us to create instance of Inhibition and provide it to override the default.
from htm.core.
@ctrl-z-9000-times now that David is working on pybind, do you want to join me here and then to #93?
I figured we should proceed here first as it separates the SP into smaller logical blocks, where none of Boosting,Inhibition,Topology are used in TM, therefore shielding them will make the port simpler.
from htm.core.
I would rather work on #93 first. It seems more straightforward than this one.
from htm.core.
Here are my thoughts on each of these topics:
Boosting: I would like to use the logarithmic boosting function as a default. Maybe we could sneak this in by turning it on when boosting-strength == -1
?
Topology: Conceptually, this is the function topology( mini-column-location ) -> potential-pool
. The SP could keep a pointer to this function, with setter/getter methods, to allow other ppl to hook in their own topology. This allows the defaults to remain, and for new functions to be added. Python function callbacks can be added.
I would add the topology function:
Topology( X )
RandomSample( num = potential_pool_size
data = for Input in InputSpace:
Normal(loc = Input, mean = X, Std = Radius)
This is the simpler 1-D case. The N-Dimensional extension of this function samples from the product of the normal-distributions in all dimensions. This has the feature of rotational symmetry, which is nice.
Update: I tried this topology function and it didn't perform as well as the one that's already in the spatial pooler. I tested it on the MNIST dataset.
Inhibition: This is a core part of the SP, and I don't think its going to be easy to change them without modifying the whole class. Local inhibition currently is a lot of the code in the SP, and my solution also changes the API.
from htm.core.
@breznak ,
I have a proposal for how to implement topology.
We should change the private method:
vector<UInt> SpatialPooler::initMapPotential_(UInt column, bool wrapAround);
Into protected method:
virtual SpatialPooler::initMapPotential( SDR &column, SDR &potentialPool );
Subclasses of SpatialPooler can then override the default topology.
- Attributes:
potentialRadius_
,potentialPct_
, andwrapAround_
will all be made protected so that subclasses can use them if they want to. WrapAround
is not an argument any more, since it's specific to this implementation of topology.- Both the mini-column index and the input index are changed into SDRs for convenience.
- No API changes, only additions.
from htm.core.
Subclasses of SpatialPooler can then override the default topology.
This does sound as a good approach! 👍
I usually tend to prefer a composition solution (Topology class and a member is passed/created, instead of overriding in classes). My reasoning for this is combinatorial explosion - if you want Topology: Wrapping, NonWrapping,Whatnot... AND Inhibition: Local,Global,YourLocalMacroCols,... you either end up with a million overriding classes, the composition let's the user to mix their salad 🥗 . What do you think?
Another advantage is it makes the main (SP) class leaner by hiding some varables in Topology.
Both the mini-column index and the input index are changed into SDRs for convenience.
Does this help, or is it more confusing? Such SDR contains exactly one ON-bit, the column, right?
from htm.core.
Do [SDR's] help, or is it more confusing? Such SDR contains exactly one ON-bit, the column, right?
I think SDR's are the way to go:
- If I give you a
UInt
it could mean a lot of things, but an SDR has a very specific meaning. It represents a group of physical locations (typically neurons), with a sub-set of them being selected/active. - It's true that there will be a single ON-bit, but that's not really a problem is it?
- SDR do the conversions! There is some confusion about what data format is best this this task. The SDR class allows both the caller and the callee to decide for themselves which data format makes sense for themselves, and allows the other side to safely and quickly extract the data.
- For topology the user will probably want the coordinates of the mini-column (sparse format), however the initialize method gives it the index (flat-sparse format).
- The current topology function computes the potential pool as a flat-sparse vector, converts it to a dense array and returns it. Then the caller (
SP::initialize
) converts it back into a flat-sparse vector.
from htm.core.
If I give you a UInt it could mean a lot of things, but an SDR has a very specific meaning.
Agreed, SDR is the way to go here.
I was thinking about adding more vectorization (by means of "batch learning", ...) to the code, so instead of UInt, the parameter would be vector (even if size()==1), but/and SDRs would allow this behavior too.
from htm.core.
I usually tend to prefer a composition solution [over subclass solution]
I understand this, I just think it's easier to use subclasses until the # of subclasses gets out of hand. For example if topology was an instance of a class which was passed in, then the user would need to create the topology object every time they create an SP, which isn't an easy default. Subclasses also allow the topology code to peek at the SP's protected data, which is convenient and sometimes necessary (especially for boosting)
from htm.core.
easier to use subclasses until the # of subclasses gets out of hand
and that's my worry, with topology (2x: Wrapping, Normal); inhibition (3x: Global, Local, MacroColumns); Boosting (2x: Numenta, "Yours") -> 12 combinations. Not that we'd test them all anyway (probably), but parametric approach makes the individual composition easier for the user (and harder for the initial developer). Good example is parameter optimization.
create the topology object every time they create an SP, which isn't an easy default
That's why I propose a "factory":
auto sp = SP(.., string topologyType="normal" /* "normal", "wrapping", etc*/, ....);
where
SP
private:
TopologyAbstract topology;
initialize() {
if(topologyType == "normal") {
topology = NormalTopology(...);
} else {...}
Subclasses also allow the topology code to peek at the SP's protected data, which is convenient and sometimes necessary (especially for boosting)
True, use it when it's necessary. On the other hand, it hides topic-related stuff into the "submodule": class Topology, Boosting, Inhibition...
So in the end, it could look like SP(.., "topologyNormal", "boostingLogaritmic", "inhibitionLocal");
What do you think?
from htm.core.
topology (2x: Wrapping, Normal); inhibition (3x: Global, Local, MacroColumns); Boosting (2x: Numenta, "Yours")
I think you overestimated this:
Topology wrapping is an argument, both versions are implemented by the same code. Also topology has 3 arguments which the factory would need.
Local inhib is slow, has no unit tests, and probably shouldn't be used. My new inhibition subclass can do both local and global inhib.
There are two variants of boosting, which the user might want to compare. Or we could do the comparison and give the user just the better if the two, assuming that one is unambiguously better than the other.
from htm.core.
Local inhib is slow, has no unit tests, and probably shouldn't be used. My new inhibition subclass can do both local and global inhib.
The lack of test and speed is true, so you intend this (new inh) to eventually replace the other two? That's a bold idea, in a good way. I'll dig some more into the new inhibition..
from htm.core.
Hi @breznak,
I'm starting to write the Column Pooler (see issue #94) and since it's new code we can make a new API for the class. For Topology this is what I have so far:
typedef function<void(SDR& cell, SDR& potentialPool)> Topology_t;
...
ColumnPooler(... , Topology_t potentialPool, ...) {
...
class DefaultTopology : public Topology_t
{
...
DefaultTopology(Real potentialPct, Real radius, bool wrapAround)
: potentialPct(potentialPct), potentialRadius(radius), wrapAround(wrapAround)
...
void operator()(SDR& cell, SDR& potentialPool) {
In MNIST_SP:
ColumnPooler htm(
...
/* potentialPool */ DefaultTopology(.9, 4., false),
from htm.core.
Could you please explain me why topology is needed here? Thanks
from htm.core.
@Thanh-Binh: This is the topology between the feed forward inputs and the proximal dendrites. It is very much like the spatial pooler's topology, except that instead of allowing a spatial pooler to cover a large topological area, it allows the column pooler to do the same.
from htm.core.
@ctrl-z-9000-times thanks for your explanation ..
from htm.core.
I think this issue gains importance given aim for parallelization #255 , and recent development of ColumnPooler (SP, CP, share inhibition, topology, ...) Extracting them to separate classes would help to drive common interface.
from htm.core.
Related Issues (20)
- Region.setParameters(name, value) not available HOT 1
- "HelloSPTPTest" Fails on a clean installation HOT 2
- No CMAKE_CXX_COMPILER could be found. HOT 1
- ModuleNotFoundError: No module named 'htm.bindings'
- Trouble Submitting Pull Request HOT 1
- Build python module without extra dependencies HOT 3
- htm.corr-nspkg.pth error HOT 1
- Build with Xcode 13.3.x fails with DEPRECATED error HOT 8
- Attempting to build from source (python): import error HOT 2
- Histogram evoked drive color HOT 1
- No module named 'htm.bindings.xxx'
- Getting error when installing by building from source HOT 1
- 'cp950' codec can't decode HOT 1
- Does not work with Python 3.11 HOT 1
- build error in debug mode
- prediction quality using ApicalTiebreakTemporalMemory? HOT 1
- Debian installation attempt fails: ModuleNotFoundError: No module named 'htm.bindings' HOT 6
- Request for monitor mixin examples
- htm.dummy extension HOT 5
- "resetIn" flag on tmRegion doesn't seem to actually work.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from htm.core.