This repository provides the data set, named IPC, for the evaluation of machine learning methods on images. It contains two collections of labeled images, presplit for training/validation/testing. The labels correspond to a total time it takes each planner to solve the corresponding planning task, and the two collections correspond to lifted and grounded representations of planning tasks. For additional details on planners and images, see Katz et al. (2018) and (Sievers et al. 2019).

Automated planning is one of the foundational areas of AI. Even the simplest formalism is known to be PSPACE-hard, and therefore no single planner can be expected to work well for all tasks and domains. Thus, portfolio-based techniques has become increasingly popular. In particular, deep learning emerges as a promising methodology for online planner selection. A prominent example is the winner of the Optimal Track of the International Planning Competition (IPC) 2018: the planner Delfi (Katz et al. 2018).

Delfi is a portfolio planner, choosing a planner out of the portfolio for a planning task, by reasoning about tasks represented as images, constructed from structural graph representations of the planning tasks (Katz et al. 2018). Two examples are the problem description graph (Pochter et al. 2011) for a grounded representation, and the abstract structure graph (Sievers et al. 2017) for a lifted representation.

The file runtimes.csv describes the collection of planning tasks with time needed for solving each task, for a collection of 29 planners. These planners include the 17 planners in the portfolio of Delfi, as well as the planners that participated in IPC 2018. The details of some of the planners may be found in Katz et al. (2018).

The timeout limit for each task is 1800s. For planners that fail to solve the task before timeout, the target value is artificially set as 10000.

The tasks not solved by any of the planners in the portfolio within the timeout limit 1800s are ignored in the construction of the data set. In particular, some of these tasks occur in IPC 2018. Hence, the test set contains tasks strictly fewer than those in IPC 2018.

There are two folders with images: grounded and lifted. Each of these folders contains a collection of images, named by the corresponding planning task.

Additionally, the folder problems contains files with task names, with one suggested separation to training/validation/test sets.

@InProceedings{sievers-et-al-aaai2019,
  author =       "Silvan Sievers, Michael Katz, Shirin Sohrabi, Horst Samulowitz, and Patrick Ferber",
  title =        "Deep Learning for Cost-Optimal Planning: Task-Dependent Planner Selection",
  booktitle =    "Proceedings of the Thirty-Third {AAAI} Conference on
                  Artificial Intelligence ({AAAI} 2019)",
  year =         "2019"
}

• Michael Katz, Shirin Sohrabi, Horst Samulowitz, and Silvan Sievers. Delfi: Online planner selection for cost-optimal planning. In Ninth International Planning Competition (IPC-9): planner abstracts, 2018.
• Nir Pochter, Aviv Zohar, and Jeffrey S. Rosenschein. Exploiting problem symmetries in state-based planners. In AAAI, 2011.
• Silvan Sievers, Gabriele Röger, Martin Wehrle, and Michael Katz. Structural symmetries of the lifted representation of classical planning tasks. In ICAPS 2017 Workshop on Heuristics and Search for Domain-independent Planning, 2017.
• Silvan Sievers, Michael Katz, Shirin Sohrabi, Horst Samulowitz, and Patrick Ferber. Deep Learning for Cost-Optimal Planning: Task-Dependent Planner Selection. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), 2019.

• Patrick Ferber, University of Basel
• Michael Katz, IBM Research
• Horst Samulowitz, IBM Research
• Silvan Sievers, University of Basel
• Shirin Sohrabi, IBM Research

Please direct questions to Michael Katz, [email protected].