General Program Synthesis Benchmark Suite

Recent interest in the development and use of non-trivial benchmark problems for genetic programming research has highlighted the scarcity of general program synthesis (also called "traditional programming") benchmark problems. We present a suite of 29 general program synthesis benchmark problems systematically selected from sources of introductory computer science programming problems. This suite is suitable for experiments with any program synthesis system driven by input/output examples. We present results from illustrative experiments using our reference implementation of the problems in the PushGP genetic programming system. The results show that the problems in the suite vary in difficulty and can be useful for assessing the capabilities of a program synthesis system.

• Publication year

  2015

• Venue

  Annual Conference on Genetic and Evolutionary Computation

• Publication date

  2015-07-11

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/2739480.2754769
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Detailed Problem Descriptions for General Program Synthesis Benchmark Suite Technical Report UM-CS-2015-006

  2015influential reference
• Solving Uncompromising Problems With Lexicase Selection

  2015cited by this paper
• The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs

  2015cited by this paper
• R: A language and environment for statistical computing.

  2014cited by this paper
• Benchmarks that matter for genetic programming

  2014cited by this paper
• Word count as a traditional programming benchmark problem for genetic programming

  2014influential reference
• Effective simplification of evolved push programs using a simple, stochastic hill-climber

  2014cited by this paper
• Implicit Fitness Sharing for Evolutionary Synthesis of License Plate Detectors

  2013cited by this paper
• Evolution vs. Intelligent Design in Program Patching

  2013cited by this paper
• Evolution vs. Intelligent Design in Program Patching

  2013cited by this paper
• Better GP benchmarks: community survey results and proposals

  2012cited by this paper
• Genetic programming needs better benchmarks

  2012cited by this paper
• Tag-based modules in genetic programming

  2011cited by this paper
• A Unifying Framework for Analysis and Evaluation of Inductive Programming Systems

  2009cited by this paper
• Genetic Programming and Autoconstructive Evolution with the Push Programming Language

  2002cited by this paper
• Fitness Sharing in Genetic Programming

  2000cited by this paper
• Genetic Programming: On the Programming of Computers by Means of Natural Selection

  1992cited by this paper

• TinyverseGP: Towards a Modular Cross-domain Benchmarking Framework for Genetic Programming

  2025cites this paper
• Fully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models

  2025cites this paper
• Was Tournament Selection All We Ever Needed? A Critical Reflection on Lexicase Selection

  2025cites this paper
• Assessing and Analyzing the Correctness of GitHub Copilot’s Code Suggestions

  2025cites this paper
• Grammar-obeying program synthesis: A novel approach using large language models and many-objective genetic programming

  2025cites this paper
• GP and LLMs for Program Synthesis: No Clear Winners

  2025influential citation
• A Comparison of Large Language Models and Genetic Programming for Program Synthesis

  2025influential citation
• Exploring the Effect of Genetic Improvement for Large Language Models-Generated Code

  2025cites this paper
• Program Synthesis with LLM-Predicted Minimal Specialized Grammars

  2025cites this paper
• Evolving Typed Token Processing Networks

  2025cites this paper
• Review and Mapping of Search-Based Approaches for Program Synthesis

  2025influential citation
• Effective Adaptive Mutation Rates for Program Synthesis

  2024cites this paper
• Going Bananas! - Unfolding Program Synthesis with Origami

  2024influential citation
• Generational Computation Reduction in Informal Counterexample-Driven Genetic Programming

  2024cites this paper
• Runtime phylogenetic analysis enables extreme subsampling for test-based problems

  2024influential citation
• Origami: (un)folding the abstraction of recursion schemes for program synthesis

  2024cites this paper
• DALex: Lexicase-like Selection via Diverse Aggregation

  2024cites this paper
• Program Synthesis on Single-Layer Loop Behavior in Pure Functional Programming

  2024cites this paper
• Comparing Large Language Models and Grammatical Evolution for Code Generation

  2024cites this paper
• A Comprehensive Analysis of Down-sampling for Genetic Programming-based Program Synthesis

  2024cites this paper
• Improving Lexicase Selection with Informed Down-Sampling

  2024cites this paper
• A Performance Analysis of Lexicase-Based and Traditional Selection Methods in GP for Symbolic Regression

  2024cites this paper
• Lexicase Selection Parameter Analysis: Varying Population Size and Test Case Redundancy with Diagnostic Metrics

  2024cites this paper
• A Pilot Study in Surveying Data Challenges of Automatic Software Engineering Tasks

  2024cites this paper
• Facilitating Function Application in Code Building Genetic Programming

  2024cites this paper
• Enhancing Program Synthesis with Large Language Models Using Many-Objective Grammar-Guided Genetic Programming

  2024cites this paper
• Untangling the Effects of Down-Sampling and Selection in Genetic Programming

  2023cites this paper
• Improve Software Development: An Overview of Program Synthesis

  2023cites this paper
• Program Synthesis with Generative Pre-trained Transformers and Grammar-Guided Genetic Programming Grammar

  2023cites this paper
• Many-objective Grammar-guided Genetic Programming with Code Similarity Measurement for Program Synthesis

  2023influential citation
• Reachability Analysis for Lexicase Selection via Community Assembly Graphs

  2023influential citation
• General Boolean Function Benchmark Suite

  2023cites this paper
• Human-Driven Genetic Programming for Program Synthesis: A Prototype

  2023influential citation
• On the Trade-Off between Population Size and Number of Generations in GP for Program Synthesis

  2023cites this paper
• Towards a General Boolean Function Benchmark Suite

  2023influential citation
• The Problem Solving Benefits of Down-sampling Vary by Selection Scheme

  2023cites this paper
• Particularity

  2023cites this paper
• Solving Novel Program Synthesis Problems with Genetic Programming using Parametric Polymorphism

  2023cites this paper
• Phylogeny-informed fitness estimation

  2023influential citation
• Probabilistic Lexicase Selection

  2023influential citation
• Fully Autonomous Programming with Large Language Models

  2023cites this paper
• HOTGP - Higher-Order Typed Genetic Programming

  2023influential citation
• A Static Analysis of Informed Down-Samples

  2023cites this paper
• Down-Sampled Epsilon-Lexicase Selection for Real-World Symbolic Regression Problems

  2023cites this paper
• MTGP: Combining Metamorphic Testing and Genetic Programming

  2023cites this paper
• Calculating lexicase selection probabilities is NP-Hard

  2023cites this paper
• Informed Down-Sampled Lexicase Selection: Identifying Productive Training Cases for Efficient Problem Solving

  2023influential citation
• Optimizing Neural Networks with Gradient Lexicase Selection

  2023cites this paper
• A Comprehensive Survey on Program Synthesis With Evolutionary Algorithms

  2023influential citation
• Measuring the ability of lexicase selection to find obscure pathways to optimality

  2022cites this paper
• Genetic programming benchmarks

  2022cites this paper
• Multi-donor Neural Transfer Learning for Genetic Programming

  2022cites this paper
• Knowledge-Driven Program Synthesis via Adaptive Replacement Mutation and Auto-constructed Subprogram Archives

  2022influential citation
• Effects of the Training Set Size: A Comparison of Standard and Down-Sampled Lexicase Selection in Program Synthesis

  2022influential citation
• Multi-objective Grammar-guided Genetic Programming with Code Similarity Measurement for Program Synthesis

  2022cites this paper
• Artificial selection methods from evolutionary computing show promise for directed evolution of microbes

  2022cites this paper
• A Hazard Analysis Framework for Code Synthesis Large Language Models

  2022cites this paper
• Program synthesis with evolutionary algorithms: Status quo: hot off the press track (GECCO 2022)

  2022influential citation
• Going faster and hence further with lexicase selection

  2022cites this paper
• Lexicase selection at scale

  2022cites this paper
• Incorporating sub-programs as knowledge in program synthesis by PushGP and adaptive replacement mutation

  2022cites this paper
• Iterative Genetic Improvement: Scaling Stochastic Program Synthesis

  2022cites this paper
• Environments with local scopes for modules in genetic programming

  2022influential citation
• Why functional program synthesis matters (in the realm of genetic programming)

  2022influential citation
• Functional code building genetic programming

  2022cites this paper
• Applying genetic programming to PSB2: the next generation program synthesis benchmark suite

  2022cites this paper
• A suite of diagnostic metrics for characterizing selection schemes

  2022cites this paper
• Population Diversity Leads to Short Running Times of Lexicase Selection

  2022influential citation
• PSB2: the second program synthesis benchmark suite

  2021influential citation
• Getting a Head Start on Program Synthesis with Genetic Programming

  2021influential citation
• Problem-Solving Benefits of Down-Sampled Lexicase Selection

  2021influential citation
• A generalizability measure for program synthesis with genetic programming

  2021influential citation
• Evaluating Large Language Models Trained on Code

  2021cites this paper
• Push

  2021cites this paper
• GLEAM: genetic learning by extraction and absorption of modules

  2021cites this paper
• An Exploration of Exploration: Measuring the ability of lexicase selection to find obscure pathways to optimality

  2021cites this paper
• Recent Developments in Program Synthesis with Evolutionary Algorithms

  2021influential citation
• What can phylogenetic metrics tell us about useful diversity in evolutionary algorithms?

  2021cites this paper
• GLEAM

  2021cites this paper
• Relationships between parent selection methods, looping constructs, and success rate in genetic programming

  2021cites this paper
• Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming

  2021influential citation
• Evolving and Analyzing Modularity with GLEAM (Genetic Learning by Extraction and Absorption of Modules)

  2021cites this paper
• Tag-based Genetic Regulation for Genetic Programming

  2020cites this paper
• Challenges of Program Synthesis with Grammatical Evolution

  2020cites this paper
• Effect of Parent Selection Methods on Modularity

  2020cites this paper
• Explaining and Exploiting the Advantages of Down-sampled Lexicase Selection

  2020cites this paper
• Modeling black-box components with probabilistic synthesis

  2020cites this paper
• Program Synthesis in a Continuous Space Using Grammars and Variational Autoencoders

  2020cites this paper
• Code Characterization With Graph Convolutions and Capsule Networks

  2020cites this paper
• Guiding Deep Molecular Optimization with Genetic Exploration

  2020cites this paper
• Benchmarking parent selection for program synthesis by genetic programming

  2020cites this paper
• Counterexample-driven genetic programming without formal specifications

  2020cites this paper
• Tag-based regulation of modules in genetic programming improves context-dependent problem solving

  2020cites this paper
• Transfer Learning of Genetic Programming Instruction Sets

  2020cites this paper
• Why and when are loops useful in genetic programming?

  2020cites this paper
• Code building genetic programming

  2020cites this paper
• Genetic Source Sensitivity and Transfer Learning in Genetic Programming

  2020cites this paper
• Genetic Programming: 23rd European Conference, EuroGP 2020, Held as Part of EvoStar 2020, Seville, Spain, April 15–17, 2020, Proceedings

  2020cites this paper
• On the importance of specialists for lexicase selection

  2020influential citation
• Summed Batch Lexicase Selection on Software Synthesis Problems

  2020cites this paper