{"corpus_id":280291954,"paper_sha":"6478c185582f5e48c54677c683a9297ff2916118","doi":"10.48550/arXiv.2507.08794","arxiv_id":"2507.08794","pmid":null,"pmcid":null,"mag_id":null,"dblp_id":"journals/corr/abs-2507-08794","acl_id":null,"title":"One Token to Fool LLM-as-a-Judge","year":2025,"publication_date":"2025-07-11","venue":"arXiv.org","journal":{"name":"ArXiv","pages":null,"volume":"abs/2507.08794"},"journal_issn":null,"journal_title":null,"publication_types":["JournalArticle"],"pubmed_pub_types":null,"s2_fields_of_study":["Computer Science"],"reference_count":65,"citation_count":36,"influential_citation_count":3,"is_open_access":false,"arxiv_categories":["cs.LG","cs.CL"],"arxiv_license":"http://creativecommons.org/licenses/by/4.0/","arxiv_journal_ref":null,"mesh_headings":null,"chemicals":null,"comments_corrections":null,"source_flags":1,"s2_open_access_pdf_url":null,"s2_open_access_landing_url":null,"s2_open_access_license":null,"s2_open_access_status":null,"pmc_open_access_pdf_url":null,"pmc_open_access_landing_url":null,"pmc_open_access_license":null,"pmc_open_access_status":null,"unpaywall_open_access_pdf_url":null,"unpaywall_open_access_landing_url":null,"unpaywall_open_access_license":null,"unpaywall_open_access_status":null,"abstract":"Large language models (LLMs) are increasingly trusted as automated judges, assisting evaluation and providing reward signals for training other models, particularly in reference-based settings like Reinforcement Learning with Verifiable Rewards (RLVR). However, we uncover a critical vulnerability even in this reference-based paradigm: generative reward models are systematically susceptible to reward hacking. We find that superficial inputs, which we term''master keys''such as non-word symbols (e.g.,'':''or''.'') or generic reasoning openers (e.g.,''Thought process:''or''Let's solve this problem step by step.''), can consistently elicit false positive rewards without any substantive reasoning. Our systematic evaluation demonstrates this is a widespread failure affecting a diverse range of models, including leading proprietary systems such as GPT-o1 and Claude-4. These results challenge the assumed robustness of LLM judges and pose a significant threat to their reliability. To address this, we propose a simple yet effective data augmentation strategy using truncated model outputs as adversarial negative examples. The resulting Master Reward Models (Master-RMs) demonstrate state-of-the-art robustness against these''master key''attacks while maintaining high performance in standard evaluation settings. We supplement these findings with a comprehensive analysis of the vulnerability across model scales, prompt variations, and common inference-time strategies, offering insights to guide future research on robust LLM evaluation. We release our robust, general-domain reward models and the synthetic training data at https://huggingface.co/sarosavo/Master-RM and https://huggingface.co/datasets/sarosavo/Master-RM.","claims":[{"public_id":"cl_9367e0669af0cc39c2ac38d4683abd1a","status":"active","text":"Data augmentation with truncated model outputs as adversarial negative examples produces Master Reward Models with state-of-the-art robustness against master key attacks while preserving standard evaluation performance.","confidence":0.96,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_9367e0669af0cc39c2ac38d4683abd1a"},{"public_id":"cl_65f5967694f32bc361be2687b94fc892","status":"active","text":"Generative reward models are systematically susceptible to reward hacking even in reference-based evaluation settings.","confidence":0.98,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_65f5967694f32bc361be2687b94fc892"},{"public_id":"cl_59b8b4320e0ff8c56488520b1cc6142a","status":"active","text":"Non-word symbols and generic reasoning openers can elicit false positive rewards without substantive reasoning.","confidence":0.97,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_59b8b4320e0ff8c56488520b1cc6142a"},{"public_id":"cl_702b1d84caab9dfffa40ec07981420a6","status":"active","text":"The robustness issue varies across model scale, prompt variation, and common inference-time strategies.","confidence":0.89,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_702b1d84caab9dfffa40ec07981420a6"},{"public_id":"cl_7ab799a270781a7fbde4aae15aec9cd7","status":"active","text":"The vulnerability is widespread across a diverse range of models, including GPT-o1 and Claude-4.","confidence":0.93,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_7ab799a270781a7fbde4aae15aec9cd7"}],"concepts":[{"public_id":"co_172233a41816cc42838208c89cf3b56e","status":"active","name":"non-word symbols","description":"Symbolic characters such as punctuation marks used here as trivial input cues.","types":["input pattern"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_172233a41816cc42838208c89cf3b56e"},{"public_id":"co_17ddd6217fd03adaab6087182959d448","status":"active","name":"LLM-as-a-Judge","description":"A setting where large language models are used to evaluate or score model outputs.","types":["evaluation framework"],"aliases":["LLM judges","automated judges"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_17ddd6217fd03adaab6087182959d448"},{"public_id":"co_45f3645a0ee759cf714f7dd927b3859d","status":"active","name":"truncated model outputs","description":"Partially cut off model responses used as adversarial negative examples for training.","types":["training data"],"aliases":["truncated outputs"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_45f3645a0ee759cf714f7dd927b3859d"},{"public_id":"co_4eceb0c6343f8981cc41f5d38e1ff818","status":"active","name":"model scales","description":"Different sizes or parameter scales of models examined in the analysis.","types":["analysis factor"],"aliases":["model scale"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_4eceb0c6343f8981cc41f5d38e1ff818"},{"public_id":"co_6942a60a866fc314a64434b136ce8b81","status":"active","name":"reward hacking","description":"Manipulating a reward model to obtain high reward without genuinely satisfying the task objective.","types":["phenomenon"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_6942a60a866fc314a64434b136ce8b81"},{"public_id":"co_6f57ea1ce3044fe788068b9c3ff63a22","status":"active","name":"generic reasoning openers","description":"Stock phrases that imitate the start of chain-of-thought reasoning.","types":["input pattern"],"aliases":["reasoning openers"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_6f57ea1ce3044fe788068b9c3ff63a22"},{"public_id":"co_87add40ff07144196662c5594b3c17d0","status":"active","name":"adversarial negative examples","description":"Training examples designed to represent undesirable outputs and improve robustness.","types":["training data"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_87add40ff07144196662c5594b3c17d0"},{"public_id":"co_8c9f6d8bf2e6b8dd42917c0c493efda6","status":"active","name":"inference-time strategies","description":"Decoding or prompting strategies applied when the model is used at inference.","types":["analysis factor"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_8c9f6d8bf2e6b8dd42917c0c493efda6"},{"public_id":"co_a54b16c4235d9d10aa935c0ea367f1cf","status":"active","name":"GPT-o1","description":"A proprietary large language model evaluated as a reward model or judge in the study.","types":["model"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_a54b16c4235d9d10aa935c0ea367f1cf"},{"public_id":"co_aef1d9c55d80434b5d3ec0b50e2a8113","status":"active","name":"prompt variations","description":"Different prompt formulations used to test model sensitivity.","types":["analysis factor"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_aef1d9c55d80434b5d3ec0b50e2a8113"},{"public_id":"co_ba45cee954fb80ee76aea31715e246cb","status":"active","name":"Master Reward Models","description":"Reward models trained with adversarial augmentation to resist superficial attack patterns.","types":["model"],"aliases":["Master-RMs"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_ba45cee954fb80ee76aea31715e246cb"},{"public_id":"co_ca9cd36340400aac0eec53aad1d53cf8","status":"active","name":"false positive rewards","description":"Incorrectly assigned positive reward signals for outputs lacking substantive correctness or reasoning.","types":["outcome"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_ca9cd36340400aac0eec53aad1d53cf8"},{"public_id":"co_d04f0a7173533c92f65f7a9936bd75f0","status":"active","name":"reference-based settings","description":"Evaluation settings in which a candidate output is judged against a reference response or target.","types":["evaluation setting"],"aliases":["reference-based paradigm"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_d04f0a7173533c92f65f7a9936bd75f0"},{"public_id":"co_f9adf76e94483be72c7ddc2b58e749c5","status":"active","name":"Claude-4","description":"A proprietary large language model evaluated as a reward model or judge in the study.","types":["model"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_f9adf76e94483be72c7ddc2b58e749c5"},{"public_id":"co_fad00fe4856f575315f84f7c3fab6fa8","status":"active","name":"master keys","description":"Superficial input patterns that can trigger undeserved positive judgments from reward models.","types":["attack pattern"],"aliases":["master key"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_fad00fe4856f575315f84f7c3fab6fa8"},{"public_id":"co_ff0effdd73cd85074f2a645c968739a0","status":"active","name":"generative reward models","description":"Reward models that generate judgments or scores from text inputs rather than using only fixed discriminative scoring.","types":["model"],"aliases":["reward models"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_ff0effdd73cd85074f2a645c968739a0"}],"external_ids":{"DOI":"10.48550/arXiv.2507.08794","ArXiv":"2507.08794","PubMed":null,"PubMedCentral":null,"MAG":null,"DBLP":"journals/corr/abs-2507-08794","ACL":null},"open_access":{"is_open_access":true,"pdf_url":"https://arxiv.org/pdf/2507.08794","landing_url":"https://arxiv.org/abs/2507.08794","source":"arxiv","pdf_url_source":"derived_arxiv","license":"http://creativecommons.org/licenses/by/4.0/","reason":null},"reference_availability":{"status":"available","references_indexed":true,"full_text_available":true,"full_text_source":"arxiv","count_basis":"semantic_scholar_metadata","extraction_status":"not_applicable","reason":null},"source":{"provider":"episteme2","base_corpus":"semantic_scholar_dump","freshness_mode":"unknown","basis":["semantic_scholar_metadata","postgres_metadata"],"limits":["paper metadata is based on indexed upstream scholarly datasets","claims and concepts are available only for extracted papers","absence of claims or concepts means no extracted graph data is available in this response"],"status":"available","degraded":false,"degraded_reasons":[],"diagnostics":{"status":"available","degraded":false,"degraded_reasons":[],"metadata_status":"available","graph_status":"available","abstract_status":"available"},"source_flags":1},"paper_id":634458,"paper_uid":"465b73f6-f723-47eb-829d-52bbedf6e54b","canonical_identity":{"paper_id":634458,"paper_uid":"465b73f6-f723-47eb-829d-52bbedf6e54b","identity_status":"available","lookup_basis":"semantic_scholar_external_id","compatibility_path":"corpus_id"},"url":"https://sah.borca.ai/papers/280291954"}