The IJCAI-09 workshop on Competitions in Artificial Intelligence and Robotics

To be held at the International Joint Conference on Artificial Intelligence (IJCAI-09)
Pasadena, California, USA, July 12, 2009

Home :: Important Dates :: Submission :: Organization :: Program

Home:

Call for Papers (PDF)

Many subfields of artificial intelligence and robotics regularly host competitions, such as the RoboCup soccer and RoboCup Rescue competitions (robotics), the SAT competition (boolean satisfiability), the International Planning Competition (action planning), the Trading Agent Competition (agents), the CADE ATP System Competition (automated theorem proving), the Annual Reinforcement Learning Competition (reinforcement learning), the Diagnostic Competition (model-based diagnosis), or the CSP solver competition (constraints).

Competitions impact research communities in many ways, including a scientific, engineering and community dimension. Scientifically, they offer a way to evaluate the state of the art of a subfield by providing a common benchmark on which different approaches to a problem can be compared. From the engineering perspective, they help technology in an area to mature by requiring development of systems that work robustly on unseen problems or by promoting the development of tools or reusable system components for the problem addressed by the competition. From the community perspective, they inspire discussion and attract publicity for a field and help enroll young researchers in a research community.

There are many subfields of artificial intelligence and robotics in which competitions have had a clear influence on the research landscape in past years:

• In robotics, the RoboCup competitions (originally on robotic soccer, recently also in search and rescue scenarios) have attracted huge publicity and inspired a large number of researchers to work on its challenges. RoboCup has effectively evolved into an own subfield where research activity is to a large extent guided by the requirements defined by the competition. More recently, the DARPA Grand Challenge has spurned a flurry of research activity on autonomous navigation in large outdoor areas, leading to impressive improvements of the state of the art.
• In satisfiability testing, the SAT competitions have provided a continuous challenge for solvers that has inspired significant algorithmic innovations for SAT solvers as well as huge improvements in implementation quality (e.g., low-level performance).
• In classical planning, the International Planning Competitions have focused the research community on a common representation language, PDDL, and a set of common benchmarks which have greatly helped comparing different classical planning systems to each other. They have also led to a huge increase in scalability of planning systems on a wide range of problem domains.

But competitions haven't had the same degree of impact in all subfields of artificial intelligence or robotics. In model-based diagnosis for instance the community has just started converging on a generally accepted way of evaluating and comparing different approaches or technologies. Some researchers argue that the missing confidence in the methods used to evaluate approaches has been an obstacle to progress in this area.

Despite the potential advantages resulting from competitions, they have been a source of controversy in many subfields of artificial intelligence. Whereas supporters believe that competitions accelerate research, opponents argue that they often focus research on synthetic problems or preclude research directions that are less aligned with current competitions.

We believe that the methods used to evaluate and compare research have strong implications on future research directions and therefore need to be well designed. Once communities have accepted regular competitions, it can be difficult to create new directions in research. Another important aspect is the question of how competition should evolve as research evolves. Therefore a careful design as well as the actively guided evolution of competitions is essential for its success in the field as well as for the success of the field.