contextlab / quail Goto Github PK

The Kahana Lab's Behavioral Toolbox has MATLAB implementations of many of the Quail functions (and desired Quail functions), including the temporal factor calculation (temp_fact) and generalized clustering calculations (dist_fact).

Note: this is not really an "issue" per se; feel free to close once you've gotten what you wanted from the link. (@andrewheusser, you had mentioned wanting some reference code.)

Needed: @KirstensGitHub add text describing how this works, and ideally share code

integrate fingerprint analyses into pyrec. The code already exists in another repo. We can A) keep the repo separate and import the package and a dependency, or B) integrate it into pyrec and get rid of the other repo. thoughts?

The software will crash if there are different numbers of lists per subject. We should handle this somehow. For instance, if a subject data is lost for one list, it would still be nice to analyze their data...

Thinking about how users will interact with this package.. During our meeting last week, we decided to implement the package as a set of functions, instead of method attached to the pyro object.

pyro=Pyro(data)
x = spc(pyro)
spc_plot(x)

vs.

pyro=Pyro(data)
pyro.spc(listgroup=listgroup).plot()

#or even more concise
Pyro(data).spc(listgroup=listgroup).plot()

Still happy with the functions as it will be the most straight forward to implement, but wanted to make one last case for the object-oriented design before we get too far with the API :)

difference between early and late lists, by experiment; follow up with unpaired t-test?

Analyze can return a nested list of analyzed data:

quail.analyze([egg1,egg2], analysis=['accuracy','spc'])
# returns
[[egg1_analyzed_acc, egg1_analyzed_spc],[egg2_analyzed_acc, egg2_analyzed_spc]]

The plot function should be able to handle these as well.

this toolbox has "correct" implementations of many of the functions we want (although it's in MATLAB): http://memory.psych.upenn.edu/Behavioral_toolbox

Plot can currently handle lists of analyzed data:

quail.plot([spc_data, pfr_data,...])

It could be convenient to also allow for lists of kwargs:

quail.plot([spc_data, pfr_data], title=['SPC','PFR'])

To facilitate doing analyses separated by experiment and list number, I propose adding two flags:

• subjgroup provides a per-subject group label (e.g. labeling which experiment each subject participated in). analysis functions carry out analyses separately for each group, and plotting functions show a different curve for each group (e.g. in a different color). for example, this will allow us to easily do analyses for different experiments, in a single command.

• listgroup provides a per-list label (independent of subjects). analysis functions carry out analyses separately for each listgroup (in addition to each subjgroup), and plotting functions show a different curve for each listgroup (e.g. in a different color). for example, this will allow us to easily do analyses for the first 8 vs last 8 lists, see how memory fingerprints change over the course of the experiment, and other similar sorts of things, also in a single command.

The way I'm imagining this would be implemented is to first divide the data based on subjgroup, passing in the listgroup flags to the function that carries out analyses for each subject. Then inside the subject-level analysis function, analyses would be performed separately on each listgroup.

An efficient way to set this up would be to have "general" analysis function that takes in a pyro object and an (analysis) function handle. The general analysis function would then have an outer loop over subjgroup and an inner loop over the listgroups for the current subjgroup, and would call the analysis function handle for actually doing the analysis on that subjgroup-listgroup piece of the data and then aggregrate the results in some way (e.g. by returning a new dataframe or pyro object).

Plotting could work similarly-- we could have a general plotting function that takes in a results object and a (plot) function handle and loops over subjgroup and listgroup, adding each piece of the data in the innermost loop (listgroup of the subjgroup) to the current plot (or aggregating the results in a way that could easily be plotted at the end).

example:

results = pyr.analyze(pyro, type='spc')
pyr.plot(results)

this function would take a file or folder of audio files of free recall data and return the decoded results.

example:

rec_data = quail.decode_speech('/path/to/file.flac')

where rec_data is a list of words and

rec_data = quail.decode_speech(folder='/path/to/folder')

where rec_data is a list of lists of words.

related to #38, add a new function that creates a matrix version of the pnr for each output position

A bar chart that plots average free recall accuracy with error bars. You can use the Seaborn barplot. This should be able to handle listgroup and subjgroup arguments as well

need a tutorial for the following:

+speech decoding (with ffmpeg/google cloud setup instructions)
+fingerprint tutorial
+crack egg (add to egg tutorial)
+recmat2egg (add to egg tutorial)

Just checking in on a point Allison brought up a while back about whether the probability of last recall should take the final word spoken as the last word or if it should skip words not on the most recently presented list (effectively becoming probability of last correctly-recalled word)

couldn't remember if we had discussed previously

current: takes last utterance, irrespective of whether the word was on the previous list

for neuroimaging experiments, automatically extract neural features (power spectra, brain maps, etc.) and include these in each presentation/recall stimulus feature set.

other things to think about/implement:

• infer the types of features from the data type
• allow the user to pass in a feature extraction function to convert brain data to feature vectors

quail is currently optimized for list-learning experiments. however, the same general framework could be applied to continuous experiments, like movie viewing/recall, social interactions, etc. for example:

• tag each stimulus and response with a feature vector
• treat viewing/listening/etc. as "presentations" -- e.g. watching a movie or listening to a conversation is like studying a list of words
• treat produced behaviors (speech, writing, quiz responses, brain responses) as recalls -- e.g. speech turns into feature vectors, which are treated like recalling words

the existing behavioral summaries (serial position curves, pnr curves, lag-CRPs, memory fingerprints) could be extended to work with continuous data by making the x-axis of these plots reflect time rather than stimulus number (or relative stimulus number).

Posting here so we can discuss and add to this, and maybe use the project board to keep track of progress?

Extension of the pfr analysis, computes probability of recall over encoding positions for a word in the nth output position.

pfr should call the pnr function with n=1

estimate chunk size from recall

plot temporal cluster score as a function of chunk size?

it would be nice to have an easy way to plot a subset of conditions, for example:

quail.plot(analyzed_egg, conditions=['forward','backward']) #dont plot random condition

This would be useful for getting a quick snapshot of an egg, such as the number of subjects, number of lists, number of words in each list, meta data etc.

It'd be nice to have some documentation (e.g. a readme file) with some simple examples. And maybe a Sphinx-generated API? Some ipython notebooks w/ simple example datasets would be cool too.

we could use tufte-style slope plots (SPC, PFR, PNR, Lag-CRP) and bar plots (fingerprints) to beautify out plot styles

inspiration: https://github.com/juanshishido/tufte

i think we should define a data object (e.g. recalls matrix + some additional info). then we can have two types of functions:

• analysis functions-- take a data object and return a new data object
• plot functions-- take a data object and return a plot

we could depend on hypertools for plotting trajectories and seaborn for other stuff

side-by-side comparison of multiple or all experiments

With our current model, I think it would be pretty easy to support custom list-wise analyses. For instance, the user could do something like this:

custom_analysis = lambda lst: [item+1 for item in lst]
pyr.analyze(pyro, analysis='custom', analysis_function=custom_analysis)

admittedly a stupid custom analysis but you get the point..think this would be desirable?

would be nice to be able to remotely access data. taking some inspiration from pandas, this could be set up with a load function that parses local files as well as files given a web address

When creating an egg, we could have these optional fields to make the syntax simpler for later functions. For example:

egg = quail.Egg(pres=pres, rec=rec, listgroup=['early']*8+['late']*8, subjgroup=['exp1']*10+['exp2']*8)

analyzed_data = quail.analyze(egg, analysis='spc')

the averaging would be inferred from the egg, unless listgroup/subjgroup are passed to the analysis function - in which case the egg groupings would be overwritten. This would allow users to save eggs with the conditions included.

just finished implementing support for nested listgroup kwarg, so that users can specify a unique listgroup for each subject. This will be useful when conditions are randomized/counterbalanced across lists over subjects.

It would be nice to have a shortcut for counterbalanced designs. I propose that we support listgroup dictionaries, something like:

listgroup = {
'cbs' : [['random']*4+['forward']*4],['forward']*4]+['random']*4]],
'subj_cb' : [0,1]
}

internally, this would generate a nested listgroup, but would be less code on the user end to set it up. thoughts?

turns out there is already a pyrec on pip... so we may want to think about another name to avoid confusion. @paxtonfitzpatrick ran into this recently, accidentally installing the wrong version of pyrec.

pyrecall has not been claimed...

there's a lot of repeated/copied code-- e.g. copying the same command for experiments 1 -- 6. instead, these should be written as loops. we can specify a list of labels to go with each experiment number, and then each command should loop through that list to create the plots.

The result is accurate, but the column names for the result of the lagcrp analysis should by -5, -4, -3, .... 3, 4, 5 etc instead of 0, 1, 2, etc.

just saving this to i remember to add, thoughts about the implementation are welcome

like pyrex, but singular and spelled wrong?

this may have to do with the data that we pass to pyro, or some strange bug inside the pyro code. I think its likely the former

example here: https://github.com/ContextLab/quail/tree/loadmat
however, that code is messy, un-tested, and shouldn't be pulled into master

ideally we should support a wide variety of data formats, including everything here: http://memory.psych.upenn.edu/Data_Archive

for the writeup/release, i think we'll want to show examples from a variety of experiments to demonstrate the robustness of the toolbox

We should substantially fill out our data loading capabilities to support a range of data formats. We should also provide good documentation for writing parsers for arbitrary data formats.

These will be critical for convincing others to use our tools-- if they can't easily use them with their own datasets, they won't care about pyrec.

The best way to get a sense of which formats we'll want to support may be to talk to other labs...but at the very least we should discuss.

this is because most of the analyses require knowing about all the presented words, and the positions argument slices out a portion of the data....will have to think about a better way to implement this

List length can bias our measure of temporal clustering (pointed out to me by Karl Healey's talk at CEMS 2017). For instance, short lists will have higher cluster scores than long lists because when recalling short lists, the subject can only possibly recall nearby items.

To account for this, we could permute recall sequences and then measure where the 'real' clustering score falls with respect to this distribution.

Not clear to me whether this would also effect our other feature dimensions?

@KirstensGitHub, let's copy in all of the files from https://github.com/kennethreitz/samplemod, and put the files you've already added into a folder called pyrec. Then we can use that as a template, and get rid of anything we dont need

I could see this working in two ways:

1. a function that takes an audio file and returns a transcript. this could be based on google speech, or our own speech model.

2. a higher level wrapper function that takes a group of encoding lists and audio files, and creates a pyro object for further analysis

For the accuracy analysis (maybe others?), it would be convenient to be able to group different positions. For example, if we wanted to plot 1 bar each accuracy for early, middle and late positions, you'd have to do an spc, and then manually select the positions. we could have a posgroup argument that allows you to run a separate analysis for multiple groups of positions. e.g.

quail.analyze(egg, analysis='accuracy', listgroup=['average']*16, posgroup=['start']*3+['middle']*10+['end']*3)

it would be really cool to integrate some models into Quail at some point (e.g. second release)

for example:

• rather than building a separately implemented mechanism into AdaptiveFR, we could call Quail functions to get the order of the next list
• our current experiment uses clustering scores to order future lists
• we could instead use arbitrary functions of prior data (e.g. models fit to prior data) to order those future lists

in addition, it'd be nice to have an easy way for people to fit some popular memory models to their data. (we could start with CMR or pTCM.) i'd love to also include marc howard's multi-timescale version of TCM, but that one is tricky to implement, so perhaps not a good place to start.

add plot_type to plot function called grid to map plot over subjects

e.g.

quail.plot(analyzed_data, plot_type='grid')

i think the current "build passing" badge on quail is actually linking to hypertools...