Reliable Measures for Aligning Japanese-English News Articles and Sentences

We have aligned Japanese and English news articles and sentences to make a large parallel corpus. We first used a method based on cross-language information retrieval (CLIR) to align the Japanese and English articles and then used a method based on dynamic programming (DP) matching to align the Japanese and English sentences in these articles. However, the results included many incorrect alignments. To remove these, we propose two measures (scores) that evaluate the validity of alignments. The measure for article alignment uses similarities in sentences aligned by DP matching and that for sentence alignment uses similarities in articles aligned by CLIR. They enhance each other to improve the accuracy of alignment. Using these measures, we have successfully constructed a large-scale article and sentence alignment corpus available to the public.

• Publication year

  2003

• Venue

  Annual Meeting of the Association for Computational Linguistics

• Publication date

  2003-07-07

• Fields of study

  Computer Science

• Identifiers
  DOI 10.3115/1075096.1075106
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Automatic Alignment of Japanese and English Newspaper Articles using an MT System and a Bilingual Company Name Dictionary

  2002cited by this paper
• Machine Translation vs. Dictionary Term Translation - a Comparison for English-Japanese News Article Alignment

  1998cited by this paper
• A Maximum Entropy Approach to Identifying Sentence Boundaries

  1997cited by this paper
• Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval

  1994influential reference
• Bilingual Text, Matching using Bilingual Dictionary and Statistics

  1994influential reference
• A Program for Aligning Sentences in Bilingual Corpora

  1993influential reference
• A Simple Rule-Based Part of Speech Tagger

  1992cited by this paper

• Low resource neural machine translation model optimization based on semantic confidence weighted alignment

  2024cites this paper
• JaParaPat: A Large-Scale Japanese-English Parallel Patent Application Corpus

  2024influential citation
• North Korean Neural Machine Translation through South Korean Resources

  2023cites this paper
• Automatic Resource Augmentation for Machine Translation in Low Resource Language: EnIndic Corpus

  2023cites this paper
• Reconstruct to Retrieve: Identifying interesting news in a Cross-lingual setting

  2023cites this paper
• Fuzzy Influenced Process to Generate Comparable to Parallel Corpora

  2023cites this paper
• Parallel sentence extraction to improve cross-language information retrieval from Wikipedia

  2021cites this paper
• A Systematic Literature Review on Extraction of Parallel Corpora from Comparable Corpora

  2021influential citation
• A Comparison of Sentence-Weighting Techniques for NMT

  2021cites this paper
• Bilingual Text Extraction as Reading Comprehension

  2020influential citation
• A Survey of Domain Adaptation for Machine Translation

  2020cites this paper
• Beyond English-Centric Multilingual Machine Translation

  2020cites this paper
• ParaCrawl: Web-Scale Acquisition of Parallel Corpora

  2020cites this paper
• Construction d’un corpus parallèle à partir de corpus comparables pour la simplification de textes médicaux en français [Building a parallel corpus from comparable corpora for French biomedical text simplification]

  2020cites this paper
• Towards Computational Linguistics in Minangkabau Language: Studies on Sentiment Analysis and Machine Translation

  2020cites this paper
• A Test Set for Discourse Translation from Japanese to English

  2020cites this paper
• A Comparison of Unsupervised Methods for Ad hoc Cross-Lingual Document Retrieval

  2020cites this paper
• SpanAlign: Sentence Alignment Method based on Cross-Language Span Prediction and ILP

  2020influential citation
• Exploiting Sentence Order in Document Alignment

  2020cites this paper
• Cross-lingual text similarity exploiting neural machine translation models

  2020cites this paper
• WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

  2019cites this paper
• Détection automatique de phrases parallèles dans un corpus biomédical comparable technique / simplifié (Automatic detection of parallel sentences in comparable biomedical corpora)

  2019cites this paper
• Leveraging Additional Resources for Improving Statistical Machine Translation on Asian Low-Resource Languages

  2019influential citation
• Online Parallel Data Extraction with Neural Machine Translation

  2019cites this paper
• RANLP 2019 12 th Workshop on Building and Using Comparable Corpora

  2019cites this paper
• Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax

  2019cites this paper
• New Areas of Application of Comparable Corpora

  2019cites this paper
• CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web

  2019cites this paper
• Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation

  2019cites this paper
• Multilingual Dynamic Topic Model

  2019cites this paper
• Self-Induced Curriculum Learning in Neural Machine Translation

  2019cites this paper
• Introduction

  2019cites this paper
• Generic and Specialized Word Embeddings for Multi-Domain Machine Translation

  2019cites this paper
• Syntax-based Preordering for Statistical Machine Translation

  2019cites this paper
• A Survey of Domain Adaptation for Neural Machine Translation

  2018cites this paper
• Filtering and Mining Parallel Data in a Joint Multilingual Space

  2018cites this paper
• Automatic Extraction of Bilingual Sentences by Similarity based on Earth Mover's Distance using Word Embeddings and Difference of Sentence Length

  2018cites this paper
• Cross-lingual Knowledge Projection Using Machine Translation and Target-side Knowledge Base Completion

  2018cites this paper
• A Multilingual Dataset for Evaluating Parallel Sentence Extraction from Comparable Corpora

  2018cites this paper
• Identification of Parallel Sentences in Comparable Monolingual Corpora from Different Registers

  2018cites this paper
• Effective Parallel Corpus Mining using Bilingual Sentence Embeddings

  2018cites this paper
• Recent works on Parallel Sentence Extraction from Comparable Corpora

  2018cites this paper
• Crowdsourcing and Translation Quality: Novel Approaches in the Language Industry and Translation Studies

  2018cites this paper
• Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings

  2018cites this paper
• MultiNews: A Web collection of an Aligned Multimodal and Multilingual Corpus

  2017cites this paper
• Using English as Pivot to Extract Persian-Italian Parallel Sentences from Non-Parallel Corpora

  2017cites this paper
• Controlling Target Features in Neural Machine Translation via Prefix Constraints

  2017cites this paper
• Automatically Extracting Parallel Sentences from Wikipedia Using Sequential Matching of Language Resources

  2017cites this paper
• Addressing unknown word problem for neural machine translation using distributee representations of words as input features

  2017cites this paper
• Analyzing Semantic Change in Japanese Loanwords

  2017influential citation
• Overview of the Second BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora

  2017cites this paper
• Crowdsourcing and Online Collaborative Translations: Expanding the limits of Translation Studies

  2017cites this paper
• Cupral 2017 The First Workshop on Curation and Applications of Parallel and Comparable Corpora Proceedings of the Workshop

  2017cites this paper
• Phrase Table Induction Using In-Domain Monolingual Data for Domain Adaptation in Statistical Machine Translation

  2017cites this paper
• Towards Preparation of the Second BUCC Shared Task : Detecting Parallel Sentences in Comparable Corpora

  2017cites this paper
• Towards the Generation of Bilingual Chinese-English Multi-word Expressions from Large Scale Parallel Corpora: An Experimental Approach

  2017cites this paper
• A Multilingual Parallel Corpus for Improving Machine Translation on Southeast Asian Languages

  2017cites this paper
• Parallel fragments : Measuring their impact on translation performance

  2017cites this paper
• Efficient Extraction of Pseudo-Parallel Sentences from Raw Monolingual Data Using Word Embeddings

  2017cites this paper
• The ADAPT Bilingual Document Alignment system at WMT16

  2016cites this paper
• Integrated Parallel Sentence and Fragment Extraction from Comparable Corpora: A Case Study on Chinese--Japanese Wikipedia

  2016cites this paper
• Otedama: Fast Rule-Based Pre-Ordering for Machine Translation

  2016cites this paper
• Ninth Workshop on Building and Using Comparable Corpora Workshop Programme

  2016cites this paper
• Sequential Decisions and Predictions in Natural Language Processing

  2016cites this paper
• Exploring the Effectiveness of Combined Web-based Corpus Tools for Beginner EFL DDL

  2016cites this paper
• Who is data-driven learning for? Challenging the monolithic view of its relationship with learning styles

  2016cites this paper
• C-BiLDA extracting cross-lingual topics from non-parallel texts by distinguishing shared from unshared content

  2016cites this paper
• A DDL approach to learning noun and verb phrases in the beginner level EFL classroom

  2016cites this paper
• Extracting parallel phrases from comparable data for machine translation†

  2016cites this paper
• Building and using multimodal comparable corpora for machine translation†

  2016cites this paper
• Effects of class-based statistical machine translation on unknown names

  2016cites this paper
• FaDA: Fast Document Aligner using Word Embedding

  2016cites this paper
• Teaching and language corpora

  2015cites this paper
• Quantifying Cross-lingual Semantic Similarity for Natural Language Processing Applications

  2015cites this paper
• Integrated Parallel Data Extraction from Comparable Corpora for Statistical Machine Translation

  2015influential citation
• Probabilistic topic modeling in multilingual settings: An overview of its methodology and applications

  2015cites this paper
• Inflating Training Data for Statistical Machine Translation using Unaligned Monolingual Data

  2015cites this paper
• English to Japanese spoken lecture translation system by using DNN-HMM and phrase-based SMT

  2015cites this paper
• Towards automated creation of high quality domain-specific machine translation resources

  2015cites this paper
• SCHOOL OF COMPUTING AND INFORMATICS UNIVERSITY OF NAIROBI MASTERS OF SCIENCE IN COMPUTER SCIENCE A Parallel Corpus Based Translation Using Sentence Similarity By RUORO SIMON WACHIRA

  2015cites this paper
• Syntax-based Rewriting for Simultaneous Machine Translation

  2015cites this paper
• Building a Korean-English Parallel Corpus by Measuring Sentence Similarities Using Sequential Matching of Language Resources and Topic Modeling

  2015cites this paper
• Development of a scale to measure learners’ perceived preferences and benefits of data-driven learning

  2015cites this paper
• Resampling approach for instance-based domain adaptation from patent domain to newspaper domain in statistical machine translation

  2015influential citation
• Parallel Sentence Extraction Based on Unsupervised Bilingual Lexicon Extraction from Comparable Corpora

  2015influential citation
• Unsupervised Algorithms for Cross-Lingual Text Analysis, Translation Mining, and Information Retrieval (Algoritmen voor ongesuperviseerde cross-linguale tekstanalyse, het identificeren van vertalingen en informatieontsluiting)

  2014cites this paper
• Iterative Bilingual Lexicon Extraction from Comparable Corpora with Topical and Contextual Knowledge

  2014cites this paper
• Unsupervised Algorithms for Cross-Lingual Text Analysis, Translation Mining, and Information Retrieval

  2014cites this paper
• La Traduction automatique statistique dans un contexte multimodal. (Statistical machine translation in a multimodal context)

  2014cites this paper
• Alignement de phrases parallèles dans des corpus bruités

  2014cites this paper
• Present state of machine translation with a focus on statistical approach

  2014cites this paper
• A Survey on Large Scale Corpora and Emotion Corpora

  2014cites this paper
• Improving machine translation via triangulation and transliteration

  2014influential citation
• Constructing a Chinese—Japanese Parallel Corpus from Wikipedia

  2014cites this paper
• An Iterative Approach for Mining Parallel Sentences in a Comparable Corpus

  2014influential citation
• Automatic Speech Recognition and Machine Translation System for MIT English Lectures Using MIT and TED Corpus

  2014cites this paper
• Multimodal Comparable Corpora for Machine Translation

  2014cites this paper
• English to Japanese spoken language translation system for classroom lectures

  2014cites this paper
• Extracting Persian-English Parallel Sentences from Document Level Aligned Comparable Corpus using Bi-Directional Translation

  2014cites this paper
• Cross-language information retrieval models based on latent topic models trained with document-aligned comparable corpora

  2013cites this paper