cloudyr / pymturkr Goto Github PK

A Client for the MTurk Requester API

R 100.00%

pymturkr's Introduction

pyMTurkR: A Client for the MTurk Requester API

pyMTurkR is an R package that allows you to interface with MTurk's Requester API.

pyMTurkR provides access to the latest Amazon Mechanical Turk (MTurk) Requester API (version '2017-01-17'), using reticulate to wrap the boto3 SDK for Python. pyMTurkR is a replacement for the now obsolete MTurkR.

Using this package, you can perform operations like: creating HITs, updating HITs, creating custom Qualifications, reviewing submitted Assignments, approving/rejecting Assignments, sending bonus payments to Workers, sending messages to Workers, blocking/unblocking Workers, and many more. See the pyMTurkR documentation for a full list of operations available.

Why make this?

pyMTurkR was created because on June 1, 2019 Amazon deprecated the MTurk API (version '2014-08-15') that MTurkR was using, rendering it obsolete. This package was created to maintain MTurk access for R users while migrating to the new MTurk API (version '2017-01-17').

pyMTurkR is not a native R language package. It uses reticulate to import and wrap the boto3 module for Python. Cross-language dependency is not necessarily a bad thing, and from the user perspective there is probably no difference, besides a few extra installation steps. Welcome to the wonderful world of R-python interoperability.

Installation

1. Install `python` and `pip`

1.1. Install python 2 (>= 2.7) or python 3 (>= 3.3) (download page)

1.2. Install pip for python

Windows users: see "Installing with get-pip.py" here
Mac users: run sudo easy_install pip to install pip

1.3. Install reticulate for R

install.packages("reticulate")

1.4. Check that reticulate can now find python. You may have to restart RStudio.

reticulate::py_config()

You will see information about your python configuration, including one or more system paths where python is installed.

1.5. Check that pip can be found

system("pip --version")

You will see something like pip 18.0 from /usr/local/lib/python2.7/site-packages/pip

2. Install `boto3`

2.1. Find the default python path that reticulate is using

reticulate::py_config()

Take note of the path in the first line (e.g., "/usr/bin/python").

2.2. Find the path that the system pip command is using

system("pip --version")

For example, in "pip 18.0 from /usr/local/lib/python2.7/site-packages/pip" the python path is "/usr/local/lib/python2.7". If the path here does not match the py_config() path, then you may need to manually set the path using use_python().

reticulate::use_python("/usr/local/lib/python2.7", required = TRUE)

You may have to restart RStudio before setting the path if you have run other reticulate operations.

2.3. Install boto3 using pip

system("pip install boto3")

Or alternatively, run this command in the system terminal (sudo pip install boto3 for Mac users).

2.4. Check if you can now import boto3. You may have to restart RStudio.

reticulate::import("boto3")

For additional install options, see "Installing Python Packages" in the reticulate docs.

3. Install `pyMTurkR`

3.1. Finally, you can install pyMTurkR.

install.packages("pyMTurkR")

Or the development version

devtools::install_github("cloudyr/pyMTurkR")

Usage

Before using the package, you will need to retrieve your "Access" and "Secret Access" keys from Amazon Web Services (AWS). See "Understanding and Getting Your Security Credentials".

Set AWS keys

The Access and Secret Access keys should be set as environment variables in R prior to running any API operations.

Sys.setenv(AWS_ACCESS_KEY_ID = "MY_ACCESS_KEY")
Sys.setenv(AWS_SECRET_ACCESS_KEY = "MY_SECRET_ACCESS_KEY")

Set environment (Sandbox or Live)

pyMTurkR will run in "sandbox" mode by default. It is good practice to test any new methods, procedures, or code in the sandbox first before going live. To change this setting and use the live environment, set pyMTurkR.sandbox to FALSE.

options(pyMTurkR.sandbox = FALSE)

Examples

library("pyMTurkR")
Sys.setenv(AWS_ACCESS_KEY_ID = "ABCD1234")
Sys.setenv(AWS_SECRET_ACCESS_KEY = "EFGH5678")
options(pyMTurkR.sandbox = FALSE)
AccountBalance()

Changelog

For development updates see the changelog.

Package maintainer / author

pyMTurkR is written and maintained by Tyler Burleigh.

Additional credits

pyMTurkR borrows code from MTurkR, written by Thomas J. Leeper. pyMTurkR's logo borrows elements from Amazon, R, and python logos; the "three people" element is thanks to Muammark / Freepik.

pymturkr's People

Contributors

Stargazers

Watchers

Forkers

erikosorensen rydenbutler lukaswallrich jinkim3 vineetbansal isabella232 nickygch gregeporter

pymturkr's Issues

GetAssignment only returns first page of assignments

I have a HIT with 261 submitted assignments. When I use the GetAssignment function, it reads, "261 Assignments Retrieved". However, I can only view what I think is the first page of the assignments. As in, when I save the assignments as a dataframe in R, I only have 61 cases. I've tried the various options within the GetAssignment function (e.g., pagetoken) and nothing has worked.

mydata <- GetAssignment(hit = "[my hit]", status = "Submitted")

^When I run this, the "mydata" dataframe only has 61 cases, rather than 261.

In MTurkR, this problem was resolved by adding the option, "return.all = TRUE" into the command. However, this is not an argument now.
mydata <- GetAssignment(hit = "[my hit]", status = "Submitted", return.all = TRUE)

GetAssignment returns duplicate assignments

Hi! I really appreciate having this package to use to access the MTurk API.

df <- GetAssignment(hit = "HITid")

This returns a data frame with duplicate assignments: e.g., if the HIT has 9 assignments, the data frame has 18 rows with rows 1-9 duplicated in rows 10-18.

CreateQualificationType() error from test.duration

Hi, I ran into an error using the CreateQualificationType function. I'm guessing somewhere in the function test.duration is converted to a string unintentionally, but I couldn't tell for sure by looking at it. I've included a minimal reproducible example. (Thank you by the way for creating pyMTurkR in the wake of the deprecation of the old API!)

#files in pyMTurkR_issue.zip
QuestionForm <- paste0(readLines('Test.xml'), collapse='')
AnswerKey <- paste0(readLines('AnswerKey.xml'), collapse='')

#works
qual1 <- CreateQualificationType(name = "qual 1",
                                description = "Enter code and accept HIT for discussion",
                                status = "Active")

#same error when trying to add test.duration
qual2 <- CreateQualificationType(name = "qual 2",
                                description = "Enter code and accept HIT for discussion",
                                status = "Active",
                                test.duration = seconds(minutes = 5),
                                test = QuestionForm, answerkey = AnswerKey)

qual3 <- CreateQualificationType(name = "qual 3",
                                description = "Enter code and accept HIT for discussion",
                                status = "Active",
                                test.duration = 300,
                                test = QuestionForm, answerkey = AnswerKey)

qual4 <- CreateQualificationType(name = "qual 4",
                                description = "Enter code and accept HIT for discussion",
                                status = "Active",
                                test.duration = "300",
                                test = QuestionForm, answerkey = AnswerKey)

Error in py_call_impl(callable, dots$args, dots$keywords) :
ParamValidationError: Parameter validation failed:
Invalid type for parameter TestDurationInSeconds, value: 300, type: <type 'str'>, valid types: <type 'int'>, <type 'long'>
Error in CreateQualificationType(name = "qual 2", description = "Enter code and accept HIT for discussion", :
Unable to create qualification

pyMTurkR_issue.zip

ConnectionClosedError for CreateQualificationType

Hi. Thanks for the useful package.
I am trying out the "CreateQualificationType" command.
I have created the Question Form and Answer Keys following your vignette and example command.
However, when I run "CreateQualificationType", I get the following error message:

QualificationWithTest <- CreateQualificationType(name = 'My Test', description = 'Qualifies workers to participate in My HITs', status = 'Active', test = TestQuestions, test.duration = 60 * 60, retry.delay = NULL, answerkey = TestKey)

Error in py_call_impl(callable, dots$args, dots$keywords) :
ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://mturk-requester.us-east-1.amazonaws.com/".

Detailed traceback:
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 276, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 573, in _make_api_call
operation_model, request_dict, request_context)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 592, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 137, in _send_request
success_response, exception):
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 231, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 211, in _emit
response = handler(**kwargs)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 183, in call
if self._checker(attempts, response, caught_exception):
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 251, in call
caught_exception)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 317, in call
caught_exception)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 223, in call
attempt_number, caught_exception)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 244, in _send
return self.http_session.send(request)
File "C:\Users\JY\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\httpsession.py", line 294, in send
endpoint_url=request.url

Error in CreateQualificationType(name = "My Test", description = "Qualifies workers to participate in My HITs", :
Unable to create qualification

Could you please guide me how to get around with this error message?
I will be looking forward to hearing from you soon. Thank you!

ContactWorker() issue

Is anyone else getting this error when running ContactWorker()?

Error in py_call_impl(callable, dots$args, dots$keywords) :
RequestError: An error occurred (RequestError) when calling the NotifyWorkers operation: There was an error with this request. (1613172686271)

Establish the API client in the global environment to make fewer API requests

Currently, every time an API operation is called a new boto3 client is created (this is in the first line of each function as client <- GetClient()). Sometimes when lots of operations are being run this can exceed the API rate limit and cause those operations to fail.

A better way to handle the API requests might be to establish a client in the global environment and send the API requests through that client. (and before each function runs, check that that client is still valid; if not, create a new one).

Issue approving workers

Hi. Thanks so much for adapting and maintaining this package! You are doing awesome work.

I tried to approve some assignments and getting an error:

> ApproveAssignment(assignments = hitassigns$AssignmentId,
+                   feedback = approvemsg)
Error in names(object) <- nm : 
  'names' attribute [4] must be the same length as the vector [3]

I believe because ApproveAssignment misspecifies the number of columns (3 instead of 4):

Assignments <- emptydf(length(assignments), 3, c("AssignmentId", 
        "Feedback", "OverrideRejection", "Valid"))

Should be:

Assignments <- emptydf(length(assignments), 4, c("AssignmentId", 
        "Feedback", "OverrideRejection", "Valid"))

Need Url + BugReports fields in DESCRIPTION

To make it easier to find this repo 😄 (e.g. run usethis::use_github_links())

ExtendHIT does't accept vector of HIT ids

Hi. I am using "ExtendHIT" function. I tried to use a vector of HIT ids but keep getting an error message. I tried transforming the vector in various ways like "as.character()", etc. Still no luck.
When I feed in only one HIT ID, it works. However, a vector of HITs like the one below doesn't work.

So, I am just looping over each HIT id in a vector. However, I would like to just put a vector in the code as suggested in the package manual. Could you please help me with some solution? Thanks!

ExtendHIT(hit = as.character(c("371QPA24C1NJQUPVEKWBBRSWE3H1TJ", "3FCO4VKOZ3CVCXM0D5OPDFJ8EFU7EW")), add.seconds = seconds(days=7))

`Error in py_call_impl(callable, dots$args, dots$keywords) :
ParamValidationError: Parameter validation failed:
Invalid type for parameter HITId, value: ['371QPA24C1NJQUPVEKWBBRSWE3H1TJ', '3FCO4VKOZ3CVCXM0D5OPDFJ8EFU7EW'], type: <class 'list'>, valid types: <class 'str'>

Detailed traceback:
File "C:\Users\Ju Yeon Park\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 276, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\Users\Ju Yeon Park\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 559, in _make_api_call
api_params, operation_model, context=request_context)
File "C:\Users\Ju Yeon Park\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 607, in _convert_to_request_dict
api_params, operation_model)
File "C:\Users\Ju Yeon Park\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\validate.py", line 297, in serialize_to_request
raise ParamValidationError(report=report.generate_report())

Error: No HITs Retrieved
Error in py_call_impl(callable, dots$args, dots$keywords) :
ParamValidationError: Parameter validation failed:
Invalid type for parameter ExpireAt, value: NA, type: <class 'str'>, valid types: <class 'datetime.datetime'>, timestamp-string

Error in py_call_impl(callable, dots$args, dots$keywords) :
ParamValidationError: Parameter validation failed:
Invalid type for parameter ExpireAt, value: NA, type: <class 'str'>, valid types: <class 'datetime.datetime'>, timestamp-string

In addition: Warning message:
In ExtendHIT(hit = as.character(c("371QPA24C1NJQUPVEKWBBRSWE3H1TJ", :
Invalid Request
Warning message:
In ExtendHIT(hit = as.character(c("371QPA24C1NJQUPVEKWBBRSWE3H1TJ", :
Invalid Request`

Unable to start boto3 client

While re-running some code that was working a few months ago (Nov or Dec 2019), I am running into the following problem. It happens as soon as I deploy a function that requires boto3 (e.g. AccountBalance(), CreateHIT() etc.). The first time I tried it, I was prompted to install Miniconda, and said 'yes', even though I already have Anaconda installed. Then, I get the following message:

Error in py_module_import(module, convert = convert): ModuleNotFoundError: No module named 'boto3'
     Unable to start boto3 client.
Error in fun(Title = "Classify a radio fragment about climate change",  : 
  could not find function "fun"
Warning message:
In CreateHIT(annotation = paste0(fulltop, " Clara Coding"), assignments = 1,  :
  Invalid Request

I have made sure that boto3 is installed for both python2 and python3. I am getting this error on two different machines with different OS's.

Thanks for any help you can give me!

Clara

Failure with dev testthat

────────────────────────────────────────────────────────────────────────────────
⠏ |   0       | UpdateQualificationType                                         
Error in fun(Name = "X74462665", Description = "Qual002", QualificationTypeStatus = "Active",  : 
  could not find function "fun"
✖ |   0 1   1 | UpdateQualificationType
────────────────────────────────────────────────────────────────────────────────
Skip (test-UpdateQualificationType.R:2:3): UpdateQualificationType
Reason: CheckAWSKeys() is not TRUE

ERROR (test-UpdateQualificationType.R:30:3): UpdateQualificationType with Qualification Test
Error: Unable to create qualification
Backtrace:
 1. pyMTurkR::CreateQualificationType(...) test-UpdateQualificationType.R:30:2

Can you please take a look? I'm planning on submitting to CRAN in about a month.

question with crowd-form element

Hi,

Thanks a million for your work on this package! A real lifesaver now that the old API is officially gone.

I was wondering whether CreateHIT's question argument can deal with crowd-form elements, like the ones used in MTurk's templates for sentiment analysis, audio naturalness and more. This was working for me in MTurkR, but in pyMTurkR when I run:

q <- GenerateHTMLQuestion(file=question_file)
hit <-CreateHIT(title = "Get qualified to label radio fragments about climate change",
                 description = "Classify a political radio fragment, and get qualified for more",
                 reward = ".20",
                 annotation = "Clim Coding Step 1b",
                 assignments = 100,
                 duration = MTurkR::seconds(hours=1),
                 expiration = MTurkR::seconds(days = 7),
                 question = q$string
                )

I get the following error:

Error in parse(text = request) : <text>:11:23: unexpected symbol
10:     <crowd-classifier 
11:         categories="['Skeptical
                          ^
Warning message:
In CreateHIT(title = "Get qualified to label radio fragments about climate change",  :
  Invalid Request

To be fair, this didn't even work on the MTurkR GUI until recently: questions like this, even unmodified templates, couldn't be given layout IDs because of parsing errors... On the plus side, Amazon fixed this, so now generating a layout ID through the GUI is a viable alternative for me.

I'm including question_file, an .xml file, in attachment.

Thanks,

Clara

question_step1b.txt

Hit creation fails with qualifications

Hi Tyler,

I am trying to create HITs or HITTypes with Qualifications, but even when I use the code straight from the documentation, there seems to be a bug (or I am making some silly mistake).

With two qualifications, as in the GenerateQualificationRequirement example, I get the following error (plus an equivalent one for QualificationRequirements2) - not sure if the c() function is meant to introduce the different endings:

Unknown parameter in input: "QualificationRequirements1", must be one of: AutoApprovalDelayInSeconds, AssignmentDurationInSeconds, Reward, Title, Keywords, Description, QualificationRequirements

When I then try to set up a HIT with just one qualification, I get the following error:
Invalid type for parameter QualificationRequirements, value: {'QualificationTypeId': '2ARFPLSP75KLA8M8DH1HTEQVJT3SY6', 'Comparator': 'Exists', 'RequiredToPreview': True}, type: <class 'dict'>, valid types: <class 'list'>, <class 'tuple'>

I hope you can help?

All the best,

Lukas

qualification requirement error

Hi!

In version 0.5.9 (as well as version 0.4.6) I am having trouble with qualification requirement code that was working properly before. When I generate a qualification requirement using the example code in the documentation, I get an error once I use it as part of CreateHIT. Code and errors below. Thanks in advance!

Clara

#make qual.req using example code
 quals.list <- list(
     list(QualificationTypeId = "2F1KVCNHMVHV8E9PBUB2A4J79LU20F",
          Comparator = "Exists",
          IntegerValues = 1,
          RequiredToPreview = TRUE
     ),
     list(QualificationTypeId = "00000000000000000071",
          Comparator = "EqualTo",
          LocaleValues = list(Country = "US"),
          RequiredToPreview = TRUE
     )
 )
GenerateQualificationRequirement(quals.list) -> qual.req

#use it in HIT
q <- GenerateHTMLQuestion(file=question_file)
hit <-CreateHIT(title = "Get qualified to label radio fragments about climate change",
                 description = "Classify a political radio fragment, and get qualified for more",
                 reward = ".20",
                 annotation = "Clim Coding Step 1b",
                 assignments = 100,
                 duration = MTurkR::seconds(hours=1),
                 expiration = MTurkR::seconds(days = 7),
                 question = q$string,
                 qual.req = qual.req
 )

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ParamValidationError: Parameter validation failed:
Invalid type for parameter QualificationRequirements, value: reticulate::tuple(reticulate::dict(list(QualificationTypeId = "2ARFPLSP75KLA8M8DH1HTEQVJT3SY6", Comparator = "Exists", RequiredToPreview = TRUE)), reticulate::dict(list(QualificationTypeId = "00000000000000000071", Comparator = "EqualTo", RequiredToPreview = TRUE, LocaleValues = list(list(Country = "US"))))), type: <class 'str'>, valid types: <class 'list'>, <class 'tuple'>

Detailed traceback: 
  File "/nfs/home/C/claravandeweerdt/.local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/nfs/home/C/claravandeweerdt/.local/lib/python3.6/site-packages/botocore/client.py", line 634, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/nfs/home/C/claravandeweerdt/.local/lib/python3.6/site-packages/botocore/client.py", line 682, in _convert_to_request_dict
    api_params, operation_model)
  File "/nfs/home/C/claravandeweerdt/.local/lib/python3.6/site-packages/botocore/validate.py", line 297, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())

Warning message:
In CreateHIT(title = "Get qualified to label radio fragments about climate change",  :
  Invalid Request

Grant bonus for more than 100 people

Hi, Tyler.

I am trying to use GetAssinment and GrantBonus. The GetAssignment function returns only 100 assignments, event though my hit has more 1000. I saw that you closed a previous issue related to this. Do you have any updates?
Thank you.

Auto-approval delay always goes to default

When creating a HIT, the auto.approval.delay parameter seems to have no effect: the HIT always ends up with an AutoApprovalDelayInSeconds of 2592000, i.e. the default (and maximum) of 30 days.

q <- GenerateHTMLQuestion(file=question_file)
hit <-CreateHIT(title = "Get qualified to label radio fragments about climate change",
                description = "Classify a political radio fragment, and get qualified for more",
                reward = ".20",
                annotation = "Clim Coding Step 1b",
                assignments = 100,
                duration = pyMTurkR::seconds(hours=1),
                expiration = pyMTurkR::seconds(days = 7),
                question = q$string,
                auto.approval.delay = 1) #gets ignored
)
HIT 309D674SHZ8HPWUGNRYOTF35Q72BCI created

GetHIT(hit="309D674SHZ8HPWUGNRYOTF35Q72BCI")$HITs$AutoApprovalDelayInSeconds
HIT (309D674SHZ8HPWUGNRYOTF35Q72BCI) Retrieved
[1] "2592000"

cloudyr / pymturkr Goto Github PK

pymturkr's Introduction

pyMTurkR: A Client for the MTurk Requester API

Why make this?

Installation

1. Install python and pip

2. Install boto3

3. Install pyMTurkR

Usage

Set AWS keys

Set environment (Sandbox or Live)

Examples

Changelog

Package maintainer / author

Additional credits

pymturkr's People

Contributors

Stargazers

Watchers

Forkers

pymturkr's Issues

Recommend Projects

Recommend Topics

Recommend Org

1. Install `python` and `pip`

2. Install `boto3`

3. Install `pyMTurkR`