SCIENCE@home

SynthACticBench: A Capability-Based Synthetic Benchmark for Algorithm Configuration

Valentin Margraf,Anna Lappe,Marcel Wever,C. Benjamins,Eyke Hüllermeier,Marius Lindauer

Published 2025 in Annual Conference on Genetic and Evolutionary Computation

ABSTRACT

Algorithm configuration deals with the automatic optimization of an algorithm's parameters to maximize its performance on a distribution of problem instances, such as Boolean satisfiability or the traveling salesperson problem. While significant progress has been made in developing optimizers for algorithm configuration - so-called algorithm configurators - their evaluation remains computationally expensive and often relies on real-world scenarios with hard-to-control characteristics. This makes it challenging to analyze their strengths and weaknesses systematically. To address this, we introduce SynthACticBench, a synthetic benchmark specifically designed to isolate and investigate key properties of algorithm configuration problems. Our benchmark distinguishes between properties related to the configuration space and those associated with the objective function. We define a configurator's ability to handle a particular property as its capability -for example, the capability to manage hierarchical configuration spaces. Using SynthACticBench, we evaluate two state-of-the-art algorithm configurators, SMAC and irace, examining their complementary capabilities and analyzing their performances across diverse benchmark functions. By providing a controlled, scalable, and capability-based evaluation environment, SynthACticBench facilitates a more targeted analysis of algorithm configurators, helping to advance research in the field. The benchmark is available at: https://github.com/annaelisalappe/SynthACticBench/.

PUBLICATION RECORD

  • Publication year

    2025

  • Venue

    Annual Conference on Genetic and Evolutionary Computation

  • Publication date

    2025-07-13

  • Fields of study

    Computer Science

  • Identifiers
    DOI 10.1145/3712256.3726330
  • External record

    Open on Semantic Scholar

  • Source metadata

    Semantic Scholar

CITATION MAP

EXTRACTION MAP

CLAIMS

  • No claims are published for this paper.

CONCEPTS

  • No concepts are published for this paper.

REFERENCES

Showing 1-38 of 38 references · Page 1 of 1

CITED BY