Git Product home page Git Product logo

presleyp / speriment Goto Github PK

View Code? Open in Web Editor NEW
6.0 5.0 1.0 306 KB

A program to make it easier to write online experiments. Speriment takes a simple description of your experiment in Python and uses it to run online experiments with the use of PsiTurk and Mechanical Turk.

License: GNU General Public License v2.0

Python 30.94% JavaScript 43.25% HTML 0.91% TypeScript 24.73% CSS 0.17%

speriment's Introduction

#Speriment

##Making experiments easier to express

###What is Speriment? Speriment is a package, inspired by SurveyMan, to help you write an online experiment for use with PsiTurk. PsiTurk describes itself as a tape player - it can run any "tape", that is, JavaScript program, as an online experiment on Mechanical Turk. But you have to provide the tape. Speriment allows you to write a simple Python script, using only the most basic programming skills, to create that JavaScript program. Instead of writing your own code to shuffle items, display HTML, record answers, and so on, you simply describe the structure and contents of your experiment.

It's currently in beta. It should work on any Unix machine but has only been tested on a Mac. Please add an Issue if you find any bugs.

###Guides

Speriment experiments are made by nesting Python objects.

To get started, look at this example script to see the basics.

Then you can look for the specific features you need here. Features you may be interested in include:

  • counterbalancing
  • Latin Squares
  • pseudorandomization
  • training loops
  • multi-page items (such as for self-paced reading tasks)
  • optional keyboard-only response selection (better for collecting response times)
  • distribution of text and resources on a per-participant basis
  • condition presentation of items depending on previous responses

For help with the syntax while writing your script, check out the API at RawGit.

The workflow for installing and running Speriment is explained here. A description of the output you'll analyze is given here.

###How do I contribute?

Contributions are super welcome. But before you start coding, start an Issue or comment on an existing one. There are lots of new features to add, and we want to make sure they'll play nice together before anyone spends time implementing stuff.

speriment's People

Contributors

adamliter avatar ppizzo-rms avatar presleyp avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

Forkers

adamliter

speriment's Issues

Let OuterBlocks have criterion

This would be easy to do but I have to think about how it should work. If InnerBlocks don't have criterion, fine. If they do, does it calculate only over their last iteration?

make tool to create db column for a json property

It would be nice to be able to look at, for instance, answers to freetext questions by querying the database. But they're buried in the JSON object in the datastring column. A tool that takes the name of the JSON property and creates a column for it would help.

periodically change status of excluded participants

Make them not count for decision about which condition/counterbalance to use next. Could work for exclusion criteria that are based on one participant, ie ReactionTime but not whether they're an outlier.

update docs

  • commit changes to API so it shows on rawgit
  • add docs on using MySQL - for instance, need to start its server before starting PsiTurk server

sql option to speriment-output

Maybe options to filter by experiment version or date, or maybe allow a SQL query to be passed in and only return results from that. Just excluding by worker ID isn't very convenient.

Make complex Pages.

This probably means introducing a new Trial class that contains one or more Pages.

make it easier to give bonuses

Can I write a script to do PsiTurk commands? Take a function written by the user to decide who should get a bonus? Could be reused in the decision to exclude people.

Resources in feedback

Should be easy, just let feedback be Pages. Good for letting people study image-word association.

Latin Square not working with PsiTurk

  • PsiTurk's condition variable has sensical values
  • Latin Square unit tests pass
  • works in viewpage.html
  • With PsiTurk, groups get initialized correctly
  • It seems to go wrong in the latin square method, but only when PsiTurk is involved

Upgrading Qunit causes filter problem

Very strange bug where Qunit insists on a filter, apparently of a certain length. Version 1.14 is fine but I'll want to figure this out for the future.

Improve validation

So much more needed - specifically I want to remember to check for exchangeable-counterbalance overlap.

consent form CSS

PsiTurk's CSS doesn't look right with my long consent forms. I put a fix in speriment.css. If you need it, add <link rel=stylesheet href="/static/lib/node_modules/speriment/css/speriment.css" type="text/css">
to templates/consent.html.

Link to PsiTurk documentation in README is broken

Unless there's a http(s):// in front of a link, I think (GitHub flavored) markdown treats links as relative. Because of this, the link to the PsiTurk documentation in the README currently points to https://github.com/presleyp/Speriment/blob/master/psiturk.readthedocs.org, rather than https://psiturk.readthedocs.org.

Constraint on sampling across pages

This is the most critical thing missing for my alternations experiment. I want to match pictures to words differently across participants, but once a picture-word association has been chosen for a participant, I need the related word to get the related picture so that singulars and plurals share semantics as expected.

I'm not sure the best way to do this. A general way would be to add something that knows how to set per-participant ground truth and then train on that. But that is probably too general; constraints and training styles probably vary too much.

Another way would be an object like SampleFrom, something like MapFrom. Whereas SampleFrom takes a string from a list, MapFrom would take a string from a dictionary. The key used to access the string would be a page id, so it would mean "get me the corresponding value to the value page 1 used." The experimenter could write in the other half of the correspondence, by putting in correlated text in these two pages for instance.

So I could have two banks of pictures. The singular bank would be a list. The plural bank would be a dictionary from singulars to plurals. When a string was sampled from the singular bank, the page id it went to would become the key to the plural in the plural bank. When MapFrom is called, it accesses that plural.

It's not terribly pretty but it's a lot prettier than all the other ways I can think of. I'll have to think about more complex versions we might want and ways to make it more flexible. At least this would mean you could map from different pages in the same page, and different aspects of a page.

Optional Questions

Right now, all non-text questions have to have answers for Next to be enabled. It would be better to have all questions require answers uniformly, and give all an argument optional that can be set to allow participants to skip questions.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.