• Use "segment‐challenge.arff" as training set and "segment‐test.arff" as test set;
• Choose KNN with default K value;
• Output the accuracy;
• Evaluate on the training set.

• Load "segment‐challenge.arff";
• 70% of "segment‐challenge.arff" is used as training set with the rest as test set;
• Choose KNN with default K value;
• Output the accuracy;
• Evaluate on the training set.

• Widely used approach for estimating test error.
• Estimates can be used to select best model, and to give an idea of the test error of the final chosen model.
• Idea is to randomly divide the data into 𝑘 equal-sized parts. We leave out one part, fit the model to the other 𝑘 − 1 parts (combined), and then obtain predictions for the left-out part.
• This is done in turn for each part 1, 2, . . . 𝑘, and then the results are combined.

• A schematic display of 5-fold CV. A set of n observations is randomly split into five non-overlapping groups. Each of these fifths acts as a validation set (shown in beige), and the remainder as a training set (shown in blue). The test error is estimated by averaging the five resulting estimates.

• Cross-validation allows us to use (almost) all the training data to evaluate our model (most useful in low-data situations). 10-fold or 5-fold is most commonly used.

Suppose we have 100 instances, and we want to estimate accuracy with cross validation. What is the final accuary?

• Load "segment‐challenge.arff";

• Use 10-CV to evaluate the model (KNN with default K value).

• Choose the best K value (1-10);

• Load "diabetes.arff";

• Use 10-CV to evaluate the model (KNN with default K value).

• Choose the best K value (1-10);

• Which feature(s) has the highest impact on the "Friendly"?
• Data source "Rabbit_Sydney_Data".
• What else knowledge do you discover via Weka?