Task-tracking-issue to upgrade ARFF to use some of the nice C++11 features.

I tested your project using the case 1. This case compiled and workerd perfectly, but the case 2, case 3 and case 4 not worked. I believe that not worked because was developed one script only for the case 1.

I would like of one example for the case 3 or case 4.

Trying to import https://github.com/erikw/etsf01_project/blob/master/testdata/cocomonasa.arff, I am getting the exception " First token must be of 'RELATION'! It is 'VALUE_TOKEN'" I tried to solve it and I realize that the comment tag '%' after a record at data block is causing the exception https://github.com/erikw/etsf01_project/blob/master/testdata/cocomonasa.arff#L436 the comment block after the record

I tested the ARFF project using the case 1 in arff_scanner_t.cpp file.
This case compiled and workerd perfectly, but for the case 2, case 3 and case 4 not worked.
I think that not worked because was developed one script only for the case 1.
I did not find a script for the case 2, case 3 or case 4.

I would like of an example (script) for the case 4 to conduct experiments.

-> Experiment 1

./arff-test

Result->

[==========] Running 21 tests from 9 test cases.
[----------] Global test environment set-up.
[----------] 5 tests from arff_value
[ RUN ] arff_value.arff_value_enum
[ OK ] arff_value.arff_value_enum (0 ms)
[ RUN ] arff_value.usage_default
[ OK ] arff_value.usage_default (0 ms)
[ RUN ] arff_value.usage_float
[ OK ] arff_value.usage_float (0 ms)
[ RUN ] arff_value.usage_string
[ OK ] arff_value.usage_string (1 ms)
[ RUN ] arff_value.usage_missing
[ OK ] arff_value.usage_missing (0 ms)
[----------] 5 tests from arff_value (1 ms total)

[----------] 1 test from arff_instance
[ RUN ] arff_instance.usage
[ OK ] arff_instance.usage (0 ms)
[----------] 1 test from arff_instance (0 ms total)

[----------] 1 test from arff_attr
[ RUN ] arff_attr.usage
[ OK ] arff_attr.usage (0 ms)
[----------] 1 test from arff_attr (0 ms total)

[----------] 2 tests from arff_lexer
[ RUN ] arff_lexer.usage1
[ OK ] arff_lexer.usage1 (0 ms)
[ RUN ] arff_lexer.usage2
[ OK ] arff_lexer.usage2 (1 ms)
[----------] 2 tests from arff_lexer (1 ms total)

[----------] 1 test from arff_data
[ RUN ] arff_data.usage
[ OK ] arff_data.usage (0 ms)
[----------] 1 test from arff_data (0 ms total)

[----------] 5 tests from arff_utils
[ RUN ] arff_utils.throw_ex
[ OK ] arff_utils.throw_ex (0 ms)
[ RUN ] arff_utils.num2str
[ OK ] arff_utils.num2str (0 ms)
[ RUN ] arff_utils.str2num
[ OK ] arff_utils.str2num (0 ms)
[ RUN ] arff_utils.to_lower
[ OK ] arff_utils.to_lower (0 ms)
[ RUN ] arff_utils.icompare
[ OK ] arff_utils.icompare (1 ms)
[----------] 5 tests from arff_utils (1 ms total)

[----------] 2 tests from arff_scanner
[ RUN ] arff_scanner.good_usage
[ OK ] arff_scanner.good_usage (0 ms)
[ RUN ] arff_scanner.bad_usage
[ OK ] arff_scanner.bad_usage (0 ms)
[----------] 2 tests from arff_scanner (0 ms total)

[----------] 2 tests from arff_token
[ RUN ] arff_token.arff_token2str
[ OK ] arff_token.arff_token2str (0 ms)
[ RUN ] arff_token.ArffToken
[ OK ] arff_token.ArffToken (0 ms)
[----------] 2 tests from arff_token (0 ms total)

[----------] 2 tests from arff_parser
[ RUN ] arff_parser.usage
[ OK ] arff_parser.usage (1 ms)
[ RUN ] arff_parser.missing_usage
[ OK ] arff_parser.missing_usage (26 ms)
[----------] 2 tests from arff_parser (27 ms total)

[----------] Global test environment tear-down
[==========] 21 tests from 9 test cases ran. (30 ms total)
[ PASSED ] 21 tests.

-> Experiment 2

I understand that the arff_scanner_t.cpp file was designed only to read the case 1, but I changed the line (ArffScanner sc("cases/case4.arff"); ) that was case1.arff to case4.arff

include

include <gtest/gtest.h>

include <arff_scanner.h>

TEST(arff_scanner, good_usage) {
ArffScanner sc("cases/case4.arff");
char c = 'a';
for(int i=0;i<26;++i,++c) {
EXPECT_EQ(c, sc.next());
EXPECT_EQ(1, sc.line());
EXPECT_EQ(i+1, sc.column());
}
EXPECT_EQ('\n', sc.next());
EXPECT_EQ('z', sc.previous());
EXPECT_EQ('\n', sc.current());
EXPECT_FALSE(sc.eof());
c = 'A';
for(int i=0;i<26;++i,++c) {
EXPECT_EQ(c, sc.next());
EXPECT_EQ(2, sc.line());
EXPECT_EQ(i+1, sc.column());
}
EXPECT_EQ('\n', sc.next());
EXPECT_FALSE(sc.eof());
EXPECT_TRUE(sc.next() < 0);
EXPECT_TRUE(sc.eof());
EXPECT_TRUE(sc.is_newline(ArffScanner::NEWLINE));
EXPECT_FALSE(sc.is_newline('\0'));
}

TEST(arff_scanner, bad_usage) {
EXPECT_THROW(ArffScanner("does_not_exist"), std::runtime_error);
}

Result->

tests/arff_scanner_t.cpp:10: Failure
Value of: sc.next()
Actual: 'T' (84, 0x54)
Expected: c
Which is: 'f' (102, 0x66)
tests/arff_scanner_t.cpp:10: Failure
Value of: sc.next()
Actual: 'i' (105, 0x69)
Expected: c
Which is: 'g' (103, 0x67)
tests/arff_scanner_t.cpp:10: Failure
Value of: sc.next()
Actual: 't' (116, 0x74)
Expected: c
Which is: 'h' (104, 0x68)
tests/arff_scanner_t.cpp:10: Failure
Value of: sc.next()
Actual: 'l' (108, 0x6C)
Expected: c
Which is: 'i' (105, 0x69)
tests/arff_scanner_t.cpp:10: Failure
Value of: sc.next()
Actual: 'e' (101, 0x65)
Expected: c
Which is: 'j' (106, 0x6A)
tests/arff_scanner_t.cpp:10: Failure
Value of: sc.next()
Actual: ':' (58, 0x3A)
Expected: c
Which is: 'k' (107, 0x6B)
tests/arff_scanner_t.cpp:10: Failure
Value of: sc.next()
Actual: ' ' (32, 0x20)
Expected: c
Which is: 'l' (108, 0x6C)
tests/arff_scanner_t.cpp:10: Failure
Value of: sc.next()
Actual: 'I' (73, 0x49)
Expected: c
.
.
.

[ FAILED ] arff_scanner.good_usage (8 ms)
[ RUN ] arff_scanner.bad_usage
[ OK ] arff_scanner.bad_usage (0 ms)
[----------] 2 tests from arff_scanner (8 ms total)

[----------] 2 tests from arff_token
[ RUN ] arff_token.arff_token2str
[ OK ] arff_token.arff_token2str (0 ms)
[ RUN ] arff_token.ArffToken
[ OK ] arff_token.ArffToken (0 ms)
[----------] 2 tests from arff_token (0 ms total)

[----------] 2 tests from arff_parser
[ RUN ] arff_parser.usage
[ OK ] arff_parser.usage (1 ms)
[ RUN ] arff_parser.missing_usage
[ OK ] arff_parser.missing_usage (0 ms)
[----------] 2 tests from arff_parser (1 ms total)

[----------] Global test environment tear-down
[==========] 21 tests from 9 test cases ran. (12 ms total)
[ PASSED ] 20 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] arff_scanner.good_usage