I wanted to create a mlr task from a data set that I just uploaded to OpenML. Unfortunately, there seems to be a problem with some characters in the data set's description!?

Do I have to change the description or is there another way to fix this?

R Log:
Downloading data set 'libras_move' from OpenML repository.
Intermediate files (XML and ARFF) will be stored in : C:\Users\tob\AppData\Local\Temp\RtmpU183Lr
Downloading file: C:\Users\tob\AppData\Local\Temp\RtmpU183Lr/data_set_desc_libras_move_v1.xml from:
http://openml.org/api/?f=openml.data.description&data.id=299
Input is not proper UTF-8, indicate encoding !
Bytes: 0xED 0x73 0x63 0x61
Error : 1: Input is not proper UTF-8, indicate encoding !
Bytes: 0xED 0x73 0x63 0x61

Error in parseXMLResponse(file, "Getting data set description", "data_set_description") :
Error in parsing XML for type data_set_description in file: C:\Users\tob\AppData\Local\Temp\RtmpU183Lr/data_set_desc_libras_move_v1.xml
All intermediate XML and ARFF files are now removed.

traceback()
5: stop(obj)
4: stopf("Error in parsing XML for type %s in file: %s", type, file)
3: parseXMLResponse(file, "Getting data set description", "data_set_description")
2: parseOpenMLDataSetDescription(file = fn.data.set.desc)
1: downloadOpenMLDataAsMlrTask("libras_move", clean.up = TRUE)

sessionInfo()
R version 3.1.0 (2014-04-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=German_Germany.1252 LC_CTYPE=German_Germany.1252 LC_MONETARY=German_Germany.1252
[4] LC_NUMERIC=C LC_TIME=German_Germany.1252

attached base packages:
[1] grid stats graphics grDevices utils datasets methods base

other attached packages:
[1] ROCR_1.0-5 gplots_2.13.0 plyr_1.8.1 cluster_1.15.2 BatchJobs_1.4 R.matlab_3.0.1
[7] fail_1.2 OpenML_1.0 DMwR_0.4.1 lattice_0.20-29 mlr_2.2 BBmisc_1.7
[13] ParamHelpers_1.3 devtools_1.5

loaded via a namespace (and not attached):
[1] abind_1.4-0 bitops_1.0-6 brew_1.0-6 caTools_1.17 checkmate_1.2
[6] class_7.3-10 codetools_0.2-8 DBI_0.2-7 digest_0.6.4 evaluate_0.5.5
[11] gdata_2.13.3 gtools_3.4.1 httr_0.3 KernSmooth_2.23-12 memoise_0.2.1
[16] parallel_3.1.0 parallelMap_1.1 quantmod_0.4-0 R.methodsS3_1.6.1 R.oo_1.18.0
[21] R.utils_1.32.4 Rcpp_0.11.2 RCurl_1.95-4.1 rJava_0.9-6 rjson_0.2.14
[26] rpart_4.1-8 RSQLite_0.11.4 RWeka_0.4-23 RWekajars_3.7.11-1 sendmailR_1.1-2
[31] splines_3.1.0 stringr_0.6.2 survival_2.37-7 tools_3.1.0 whisker_0.3-2
[36] XML_3.98-1.1 xts_0.9-7 zoo_1.7-11

Travis is a Continuous Integration platform. It's quite handy for automatic build testing. More information at https://travis-ci.org/

Talk to J.R. for this.

Include the mlr version that was used to generate the implementation.

My suggestion would be to include sessionInfo() output, so all the packages and their versions are included and therefore facilitates reproducibility.

At least these ones do not work:

blacklist = c("baseball","cmc","hypothyroid","mfeat-factors","sick","spambase","mushroom","page-blocks")

May be change foreign::read.arff() to RWeka::read.arff()

I guess this a problem on the R side. Need to check.

openML.impl <- createOpenMLImplementationForMLRLearner(learner)

I got:

Error in createOpenMLImplementationForMLRLearner(learner) :
could not find function "OpenMLImplementation"

We'd like to run code chunks and print the output in the tutorial, so the user sees what happens. However, most of the code interacts with the server. Therefore, a valid hash must be supplied. How can we solve this problem? Right now, the chunks are not evaluated, so we have no output.

Maybe we should add a special openML-account for this (e.g., for Mr. "John Sample")? But then, users will know its login data and could "abuse" the account.

Assume you download stuff on a cluster. Now the if the cachedir of diffferent R processes points to the same physical dir, we might have a problem right?

A guess we dont want to implment locks?
Or does R support that?

Does that imply parallel jobs cant use the same cache?

This seems like a complicated topic, whatever we decide we should document it carefully....

1. setOMLConfig just sets the configuration for the current R session and does not modify the config file. Wouldn't it be better to replace the config file with the parameters set by setOMLConfig for a global configuration? Or is setOMLConfig only thought as session-specific configuration?

2. This is rather less importand but after modifying the config file manually, the package has to be detached and loaded again to get the modified changes from the config file. What if we extend setOMLConfig so that the default call setOMLConfig(conf=list()) always reloads the config file and sets the configuration specified in the config file?

If you provide a path relative to your home ("~/.openml/cache/..."), you get a nice java IO error...

The predictions upload should be one based as it is when the user downloads a task.

Names should be the same as on the server. the only difference should be . instead of _.

exceptions only in VERY RARE cases

check current objects for that rule.

At the moment, the password in the config file or console is in plain text. Find a better solution. Maybe we should consider a password prompt as shown here:
http://stackoverflow.com/questions/13033573/how-to-handle-db-passwords-in-r-connection-strings

I am not sure why we need the did column for the evaluations in the output.data slot of a OMLRun object:

run = getOMLRun(run.id = 1L)
run$output.data$evaluations$did
## [1] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA

I have checked the did column for the first 100 run.ids and each of them return a vector of NAs, so do we really need this column?

I made a similar observation for run$output.data$evaluations$label, run$output.data$evaluations$sample.size and run$output.data$evaluations$stdev. Can we remove these columns when they contain only NAs? Or can we even remove them completely?

Authenticating user at server: [email protected]
[1] "http://www.openml.org/api/?f=openml.authenticate"
Error: Internal Server Error

We need informative msg

> getMetaLearningFeatures()
SQL was processed: 28 rows selected. 
Fehler in getMetaLearningFeatures() : 
  konnte Funktion "getDataQualities" nicht finden

see here

The class "OMLConfig" is assigned each time when one runs getOMLConfig().

Example: Running this code for (i in 1:10) cat(length(class(getOMLConfig())), fill=TRUE) should print always the same length, currently the length increases.

1. Extend the current pages slightly somewhat.

2. Add some general info on OpenML, but always link if info is already there.

3. Show downloading of performance info.

• Needs to be changed, there is some ::: stuff in there.
• Do we really want OpenML data sets as mlr tasks? OpenML tasks should become mlr tasks!
• Document

Make the code work for now, but discuss whether this is reasonable then!

ot = downloadOpenMLTask(3008)
Downloading task 3008 from OpenML repository.
Intermediate files (XML and ARFF) will be stored in : /tmp/Rtmpm70wOk
Downloading file: /tmp/Rtmpm70wOk/task.xml from:
http://openml.org/api/?f=openml.tasks.search&task_id=3008
Downloading file: /tmp/Rtmpm70wOk/data_set_description.xml from:
http://openml.org/api/?f=openml.data.description&data.id=299
Downloading file: /tmp/Rtmpm70wOk/data_set.ARFF from:
http://openml.org/files/download/52200/libras_move.arff
Downloading file: /tmp/Rtmpm70wOk/data_splits.ARFF from:
http://openml.org//api_splits/get/3008/Task_3008_splits.arff
All intermediate XML and ARFF files are now removed.
ot$
ot$id ot$pars ot$data.desc.id ot$estimation.procedure ot$evaluation.measures
ot$type ot$target.features ot$data.desc ot$preds
ot$data.desc

Dataset libras_move :: (openML ID = 299, version = 1)

    Collection Date  : 2009
    Upload Date      : 2014-08-20 20:56:22
    Licence          : Public
    URL              : http://openml.org/files/download/52200/libras_move.arff

Error in if (x$language != "") catf("\tLanguage : %s", x$language) :
missing value where TRUE/FALSE needed

Like downloadOpenMLDataAsMlrTask, but do not convert to mlr, but return the raw results.

Then remove downloadOpenMLDataAsMlrTask, and write a converter for that result object to mlr.

measure time of each train test split, for training and prediction separately.

also do one final model fit for the whole data, predict on whole data. throw results away.

depending on how large the model is, upload this as well. also in human readable form.

also upload rscimark vector, once per machine, cache this.

and thus the tutorial is somewhat hard to find.

This should be an option to subset the results to the given "stati".

default should be "active". in 99% of cases you dont want the other crap

We use unserialize to retrieve complex mlr learner objects from source files of OpenML flows. This is not working anymore. Try the following for example:

flow = getOMLFlow(1057)
source(flow$source.path)
sourcedFlow(task.id = 1L)

wasn't able to find a specific function to upload datasets to openml, is there any available or need to make a query?

@dominikkirchhoff @giuseppec

Please sort this out ASAP.

Also please write down how one changes the wiki and works on it, so new people like @giuseppec know how to work on it.

Giuseppe is emailing tomorrow.

Link in ?configuration points to BatchJobs. We also need some documentation in the package, i.e. location and format of the configuration file.

not sure why, but I'm having a problem with java. On
devtools::install_github("openml/r")
I get

• installing source package ‘OpenML’ ...
  ** R
  ** inst
  ** tests
  ** byte-compile and prepare package for lazy loading
  No Java runtime present, requesting install.

I just installed Java (v8), and
from the terminal it seems OK:

stephens$ java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)

any ideas?

In uploadMlrLearner, createOpenMLImplementationForMlrLearner, (etc.)

Do an extra roxygen section / file (without real code) OMLDataSet.R

Document everything which is in the class. For the "data" and "desc" try to link the global API Docs.
(otherwise too much to write down)

Here we create a resample description although an openMLtask already contains a given Data Split so we should create a resample instance directly.

> task.openML$estimation.procedure

Estimation Method :: holdout
Parameters         ::
    number_repeats = 1
    number_folds = NA
    percentage = 33
    stratified_sampling = true
Data Splits        :: 
'data.frame':   2000 obs. of  4 variables:
 $ type : Factor w/ 2 levels "TRAIN","TEST": 2 2 2 2 2 2 2 2 2 2 ...
 $ rowid: int  475 904 1484 751 1253 1024 1356 1376 389 10 ...
 $ rep  : num  1 1 1 1 1 1 1 1 1 1 ...
 $ fold : num  1 1 1 1 1 1 1 1 1 1 ...

Hi, I uploaded this dataset: http://openml.liacs.nl/d/345
and noticed there are periods in the feature names. Is this still an issue in r? AFAI can remember we agreed that periods are quite common and will be handled on the r side?

Just checking :)

Brainstorm a few ideas.

Here is a simple one: In Bagging and random forests there is a difference about
a) when you randomly select the festures. In the tree nodes (RF) or the for the whole tree (B)
b) whether you prune the tree (B: maybe) or not (RF)

Now is there a big difference in performance? If yes which of the 2 is more important?
Yes, one could probably also look at Breimans paper for this but why not demonstrate it via a nice OMLR study?

There probably dozens of other interesting similar questions.

Resampling of the first 100 tasks with all adequate mlr learners.
Submitted: 1710 (100%)
Done: 1692 (98.95%)
Expired: 18 (1.05%)

There were 295 errors (17%). Here are some common error messages:

• [114 times] Error in contrasts<-(*tmp*, value = contr.funs[1 + isOF[nn]]) : contrasts can be applied only to factors with 2 or more levels

Quite clearly, this is due to "pseudo factors" that have only one level. I guess, we should remove those automatically in runTask().

• [24 times] Error in knn.cv(x, cl, k) : NA/NaN/Inf in foreign function call (arg 6)
• [21 times] Error in scale.default(newdata[, scaling(object)$scaled, drop = FALSE], : length of 'center' must equal the number of columns of 'x'
• [19 times] Error in qda.default(x, grouping, ...) : rank deficiency in group [...].

If you want me to submit some of the rarer errors, too, or post more details on those above, please tell me.

Hey,

at the moment, in getOMLFlow we do not download source/binary files at all. How should the function behave?

1. where to save the files (cache?)
2. always download all, or filter by file format or something like that?

As an "only R"-User, I'd not be interested in anything other than R-files, so a filter would be nice, I guess.

• it is currently required for all actions to authenticate, this is nonsense?
• the session hash should be stored in and retrieved from options (if not directly provided) after first call to authenticateUser.

It seems that downloadOpenMLDataAsMlrTask() can only be used once per R Session, because some files in the temporary data folder can't be overwritten.

Here's what I've done:
task1 = downloadOpenMLDataAsMlrTask("ozone_level")
-> works fine

task2 = downloadOpenMLDataAsMlrTask("abalone")
-> Error

Error Message:
Downloading data set 'abalone' from OpenML repository.
Intermediate files (XML and ARFF) will be stored in : C:\Users\tob\AppData\Local\Temp\Rtmp6D2vf4
Downloading file: C:\Users\tob\AppData\Local\Temp\Rtmp6D2vf4/data_set_description.xml from:
http://openml.org/api/?f=openml.data.description&data.id=183
Error in downloadOpenMLDataAsMlrTask("abalone") :
Assertion on 'file' failed: File at path already exists: 'C:\Users\tob\AppData\Local\Temp\Rtmp6D2vf4\data_set.arff'
All intermediate XML and ARFF files are now removed.

Of course, it's possible to change the directory via the dir-option, but for the download of multiple data sets it would be nice just to use the same temporary path over and over.

sessionInfo()
R version 3.1.0 (2014-04-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
locale:
[1] LC_COLLATE=German_Germany.1252 LC_CTYPE=German_Germany.1252 LC_MONETARY=German_Germany.1252
[4] LC_NUMERIC=C LC_TIME=German_Germany.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] OpenML_1.0 devtools_1.5
loaded via a namespace (and not attached):
[1] BBmisc_1.7 checkmate_1.2 codetools_0.2-8 digest_0.6.4 evaluate_0.5.5
[6] grid_3.1.0 httr_0.3 memoise_0.2.1 mlr_2.1 parallel_3.1.0
[11] parallelMap_1.1 ParamHelpers_1.3 plyr_1.8.1 Rcpp_0.11.2 RCurl_1.95-4.1
[16] rJava_0.9-6 rjson_0.2.14 RWeka_0.4-23 RWekajars_3.7.11-1 splines_3.1.0
[21] stringr_0.6.2 survival_2.37-7 tools_3.1.0 whisker_0.3-2 XML_3.98-1.1

Use SQL call.
Ask Jan / Joaquin for example.

The result should be a data.frame with columns like

dataset, n_obs, n_feats, n_classes, ....

The task structure defined by the XSD

https://github.com/openml/website/blob/master/openml_OS/views/pages/rest_api/xsd/openml.task.search.xsd

is not the same as the one we currently use in R.

at lest these file names

test_base_*

test_server_*

The 2nd group tests all objects from the server, or so

xmlParse accepts character(n) as input if asText == TRUE. We do not need to write everything to temp files first.