Systematic Study of Different Structural Large Language Models in Semantic Textual Similarity

Semantic Textual Similarity(STS) is one of the challenging problems in the field of natural language processing and plays an important role in many applications such as question answering, recommendation systems and information retrieval. For the STS tasks, previous work has primarily been based on encoder-only structural models, as researchers widely believe that such structure can better capture feature information compared to the other structural models. However, there has been a lack of systematic comparative studies to investigate the performance of different structural models on STS. To fill this gap, we systematically compared different structural models and explored methods for extracting sentence embeddings tailored to different structures and conducted extensive experiments across 7 text similarity datasets. The results show that, the decoder-only model LlaMA2 has shown superior overall performance on the SentEval benchmark without fine-tuning and as the parameters increased, the performance of the decoder-only model improves gradually. Additionally, for any model structure, without doing any projection, the intermediate layers of the model actually degrade the quality of sentence embeddings, affecting the model's performance on STS tasks.

• Publication year

  2025

• Venue

  Cybersecurity and Cyberforensics Conference

• Publication date

  2025-07-28

• Fields of study

  Not labeled

• Identifiers
  DOI 10.23919/CCC64809.2025.11178482
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Retrieval-Augmented Generation for AI-Generated Content: A Survey

  2024cited by this paper
• Retrieval-Augmented Generation for Large Language Models: A Survey

  2023cited by this paper
• Scaling Sentence Embeddings with Large Language Models

  2023cited by this paper
• LLaMA: Open and Efficient Foundation Language Models

  2023cited by this paper
• Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

  2021cited by this paper
• All NLP Tasks Are Generation Tasks: A General Pretraining Framework

  2021cited by this paper
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

  2019cited by this paper
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  2019cited by this paper
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

  2019cited by this paper
• SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

  2017cited by this paper
• SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation

  2016cited by this paper
• SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

  2014cited by this paper
• A SICK cure for the evaluation of compositional distributional semantic models

  2014cited by this paper
• *SEM 2013 shared task: Semantic Textual Similarity

  2013cited by this paper
• *SEM 2012: The First Joint Conference on Lexical and Computational Semantics -- Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)

  2012cited by this paper

• No citing papers are available for this paper.