NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions

The Natural Language for Optimization (NL4Opt) Competition was created to investigate methods of extracting the meaning and formulation of an optimization problem based on its text description. Specifically, the goal of the competition is to increase the accessibility and usability of optimization solvers by allowing non-experts to interface with them using natural language. We separate this challenging goal into two sub-tasks: (1) recognize and label the semantic entities that correspond to the components of the optimization problem; (2) generate a meaning representation (i.e., a logical form) of the problem from its detected problem entities. The first task aims to reduce ambiguity by detecting and tagging the entities of the optimization problems. The second task creates an intermediate representation of the linear programming (LP) problem that is converted into a format that can be used by commercial solvers. In this report, we present the LP word problem dataset and shared tasks for the NeurIPS 2022 competition. Furthermore, we investigate and compare the performance of the ChatGPT large language model against the winning solutions. Through this competition, we hope to bring interest towards the development of novel machine learning applications and datasets for optimization modeling.

• Publication year

  2023

• Venue

  Neural Information Processing Systems

• Publication date

  2023-03-14

• Fields of study

  Computer Science

• Identifiers
  DOI 10.48550/arXiv.2303.08233arXiv 2303.08233
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• A Novel Approach for Auto-Formulation of Optimization Problems

  2023cited by this paper
• OPD@NL4Opt: An ensemble approach for the NER task of the optimization problem

  2023cited by this paper
• Highlighting Named Entities in Input for Auto-Formulation of Optimization Problems

  2022influential reference
• VTCC-NLP at NL4Opt competition subtask 1: An Ensemble Pre-trained language models for Named Entity Recognition

  2022cited by this paper
• Linear programming word problems formulation using EnsembleCRF NER labeler and T5 text generator with data augmentations

  2022influential reference
• Tag Embedding and Well-defined Intermediate Representation improve Auto-Formulation of Problem Description

  2022cited by this paper
• Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptions

  2022cited by this paper
• Are NLP Models really able to Solve Simple Math Word Problems?

  2021cited by this paper
• SemEval-2021 Task 8: MeasEval – Extracting Counts and Measurements and their Related Contexts

  2021cited by this paper
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

  2021cited by this paper
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks

  2020cited by this paper
• Identifying locations for new bike-sharing stations in Glasgow: an analysis of spatial equity and demand factors

  2020cited by this paper
• A Review of Cross-Domain Text-to-SQL Models

  2020cited by this paper
• Benchmarking Meaning Representations in Neural Semantic Parsing

  2020cited by this paper
• A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers

  2020cited by this paper
• Analytics and Optimization Reduce Sewage Overflows to Protect Community Waterways in Kentucky

  2020cited by this paper
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

  2019cited by this paper
• SemEval-2019 Task 10: Math Question Answering

  2019cited by this paper
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

  2019cited by this paper
• Unsupervised Cross-lingual Representation Learning at Scale

  2019cited by this paper
• Dialog-to-Action: Conversational Question Answering Over a Large-Scale Knowledge Base

  2018cited by this paper
• NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System

  2018cited by this paper
• Deep Contextualized Word Representations

  2018cited by this paper
• Graph Convolutional Networks for Text Classification

  2018cited by this paper
• Towards Deep Learning Models Resistant to Adversarial Attacks

  2017cited by this paper
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

  2017cited by this paper
• MAWPS: A Math Word Problem Repository

  2016cited by this paper
• Research on pricing method of public bicycle service: A case study in Guangzhou

  2016cited by this paper
• Language to Logical Form with Neural Attention

  2016cited by this paper
• Semantic Parsing via Paraphrasing

  2014cited by this paper
• Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

  2014cited by this paper
• Explaining and Harnessing Adversarial Examples

  2014cited by this paper
• Production Planning by Mixed Integer Programming

  2010cited by this paper
• An overview of pricing models for revenue management

  2003cited by this paper
• The (Dantzig) simplex method for linear programming

  2000cited by this paper
• Supply chain partnerships: Opportunities for operations research

  1997cited by this paper
• A new polynomial-time algorithm for linear programming

  1984cited by this paper
• Model Building in Mathematical Programming

  1979cited by this paper

• LLM Agents for Combinatorial Efficient Frontiers: Investment Portfolio Optimization

  2026cites this paper
• ConstraintBench: Benchmarking LLM Constraint Reasoning on Direct Optimization

  2026cites this paper
• OptiRepair: Closed-Loop Diagnosis and Repair of Supply Chain Optimization Models with LLM Agents

  2026cites this paper
• ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

  2026influential citation
• Automated Optimization Modeling via a Localizable Error-Driven Perspective

  2026cites this paper
• Constructing Industrial-Scale Optimization Modeling Benchmark

  2026influential citation
• MIRROR: A Multi-Agent Framework with Iterative Adaptive Revision and Hierarchical Retrieval for Optimization Modeling in Operations Research

  2026influential citation
• ProOPF: Benchmarking and Improving LLMs for Professional-Grade Power Systems Optimization Modeling

  2026cites this paper
• EvoOpt-LLM: Evolving industrial optimization models with large language models

  2026cites this paper
• Business optimization for digital manufacturing: A fine-tuned large language model approach

  2026cites this paper
• NEMO: Execution-Aware Optimization Modeling via Autonomous Coding Agents

  2026cites this paper
• OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling

  2026cites this paper
• Grammar-Aware Literate Generative Mathematical Programming with Compiler-in-the-Loop

  2026influential citation
• Mathematical programming model mining: A systematic field survey

  2026cites this paper
• NC2C: Automated Convexification of Generic Non-Convex Optimization Problems

  2026cites this paper
• OR-Toolformer: Modeling and Solving Operations Research Problems with Tool Augmented Large Language Models

  2025cites this paper
• Artificial intelligence for optimization: Unleashing the potential of parameter generation, model formulation, and solution methods

  2025cites this paper
• "It Was a Magical Box": Understanding Practitioner Workflows and Needs in Optimization

  2025cites this paper
• Large Language Models in Operations Research: Methods, Applications, and Challenges

  2025cites this paper
• StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models

  2025influential citation
• OptiMind: Teaching LLMs to Think Like Optimization Experts

  2025cites this paper
• REG: A Regularization Optimizer for Robust Training Dynamics

  2025influential citation
• CALM Before the STORM: Unlocking Native Reasoning for Optimization Modeling

  2025cites this paper
• SAC-Opt: Semantic Anchors for Iterative Correction in Optimization Modeling

  2025cites this paper
• Mathematical Programming Through the Lens of LLMs: Systematic Evidence and Empirical Gaps

  2025cites this paper
• Global Constraint LLM Agents for Text-to-Model Translation

  2025cites this paper
• Large Language Models and Operations Research: A Structured Survey

  2025cites this paper
• SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided Search

  2025cites this paper
• LinearizeLLM: An Agent-Based Framework for LLM-Driven Exact Linear Reformulation of Nonlinear Optimization Problems

  2025cites this paper
• OPT2CODE: A retrieval-augmented framework for solving linear programming problems

  2025cites this paper
• AlphaOPT: Formulating Optimization Programs with Self-Improving LLM Experience Library

  2025cites this paper
• Toward Conversational Decision Support Systems: Integrating LLMs in the Operations Research Methodology

  2025cites this paper
• LISTEN to Your Preferences: An LLM Framework for Multi-Objective Selection

  2025cites this paper
• ORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modeling

  2025cites this paper
• SOCRATES: Simulation Optimization with Correlated Replicas and Adaptive Trajectory Evaluations

  2025cites this paper
• An LLM-powered MILP modelling engine for workforce scheduling guided by expert knowledge

  2025cites this paper
• An Agent-Based Framework for the Automatic Validation of Mathematical Optimization Models

  2025cites this paper
• LM4Opt-RA: A Multi-Candidate LLM Framework with Structured Ranking for Automating Network Resource Allocation

  2025cites this paper
• Behavior and Representation in Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selection

  2025cites this paper
• Solver-Independent Automated Problem Formulation via LLMs for High-Cost Simulation-Driven Design

  2025cites this paper
• Do LLMs Understand Constraint Programming? Zero-Shot Constraint Programming Model Generation Using LLMs

  2025cites this paper
• CP-Bench: Evaluating Large Language Models for Constraint Modelling

  2025cites this paper
• Benchmarking Solver-agnostic Constraint Modeling Frameworks

  2025cites this paper
• OptiChat: Bridging Optimization Models and Practitioners with Large Language Models

  2025cites this paper
• Decision Information Meets Large Language Models: The Future of Explainable Operations Research

  2025cites this paper
• OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling

  2025influential citation
• LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging

  2025cites this paper
• EquivaMap: Leveraging LLMs for Automatic Equivalence Checking of Optimization Formulations

  2025influential citation
• A Comprehensive Survey of Linear, Integer, and Mixed-Integer Programming Approaches for Optimizing Resource Allocation in 5G and Beyond Networks

  2025cites this paper
• Text2Zinc: A Cross-Domain Dataset for Modeling Optimization and Satisfaction Problems in MiniZinc

  2025cites this paper
• From User Preferences to Optimization Constraints Using Large Language Models

  2025cites this paper
• CO-Bench: Benchmarking Language Model Agents in Algorithm Search for Combinatorial Optimization

  2025cites this paper
• OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents

  2025cites this paper
• CHORUS: Zero-shot Hierarchical Retrieval and Orchestration for Generating Linear Programming Code

  2025cites this paper
• Optimization Problem Solving Can Transition to Evolutionary Agentic Workflows

  2025cites this paper
• AgentMILO: A Knowledge-Based Framework for Complex MILP Modelling Conversations with LLMs

  2025cites this paper
• Learning Virtual Machine Scheduling in Cloud Computing through Language Agents

  2025cites this paper
• Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling

  2025cites this paper
• Collaborative Problem-Solving in an Optimization Game

  2025cites this paper
• DualSchool: How Reliable are LLMs for Optimization Education?

  2025cites this paper
• ORMind: A Cognitive-Inspired End-to-End Reasoning Framework for Operations Research

  2025cites this paper
• DCP-Bench-Open: Evaluating LLMs for Constraint Modelling of Discrete Combinatorial Problems

  2025influential citation
• HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimization

  2025cites this paper
• SMARTAPS: Tool-augmented LLMs for Operations Management

  2025cites this paper
• Auto-Formulating Dynamic Programming Problems with Large Language Models

  2025influential citation
• CP-Agent: Agentic Constraint Programming

  2025cites this paper
• Automated Constraint Specification for Job Scheduling by Regulating Generative Model With Domain-Specific Representation

  2025cites this paper
• EvoCut: Strengthening Integer Programs via Evolution-Guided Language Models

  2025cites this paper
• Automated Optimization Modeling through Expert-Guided Large Language Model Reasoning

  2025influential citation
• R-ConstraintBench: Evaluating LLMs on NP-Complete Scheduling

  2025cites this paper
• Teaching LLMs to Think Mathematically: A Critical Study of Decision-Making via Optimization

  2025cites this paper
• Large Language Model-Based Automatic Formulation for Stochastic Optimization Models

  2025cites this paper
• Improving Adaptability in Optimization-Based Decision Support Systems Through Large Language Models

  2025cites this paper
• LLM-QUBO: An End-to-End Framework for Automated QUBO Transformation from Natural Language Problem Descriptions

  2025cites this paper
• CP-Model-Zoo: A Natural Language Query System for Constraint Programming Models

  2025cites this paper
• A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving

  2025cites this paper
• Gala: Global LLM Agents for Text-to-Model Translation

  2025cites this paper
• GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation

  2024influential citation
• Optimization modeling and verification from problem specifications using a multi-agent multi-stage LLM framework

  2024influential citation
• Constraint Modelling with LLMs Using In-Context Learning

  2024cites this paper
• LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch

  2024influential citation
• A Systematic Survey on Large Language Models for Algorithm Design

  2024cites this paper
• Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks

  2024cites this paper
• Autoformulation of Mathematical Optimization Models Using LLMs

  2024influential citation
• BPP-Search: Enhancing Tree of Thought Reasoning for Mathematical Modeling Problem Solving

  2024influential citation
• Evaluating LLM Reasoning in the Operations Research Domain with ORQA

  2024cites this paper
• LM4OPT: Unveiling the potential of Large Language Models in formulating mathematical optimization problems

  2024cites this paper
• From Large Language Models and Optimization to Decision Optimization CoPilot: A Research Manifesto

  2024cites this paper
• OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models

  2024cites this paper
• Synthetic Dialogue Dataset Generation using LLM Agents

  2024cites this paper
• Ner4Opt: named entity recognition for optimization modelling from natural language

  2024cites this paper
• Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process

  2024cites this paper
• A Survey of Optimization Modeling Meets LLMs: Progress and Future Directions

  2024influential citation
• LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages

  2024cites this paper
• Towards Human-aligned Evaluation for Linear Programming Word Problems

  2024cites this paper
• PresAIse, a prescriptive AI solution for enterprise

  2024cites this paper
• Benchmarking LLMs for Optimization Modeling and Enhancing Reasoning via Reverse Socratic Synthesis

  2024cites this paper
• OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale

  2024cites this paper
• OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling

  2024influential citation
• Solving General Natural-Language-Description Optimization Problems with Large Language Models

  2024cites this paper