How Many Folders Do You Really Need?: Classifying Email into a Handful of Categories

Email classification is still a mostly manual task. Consequently, most Web mail users never define a single folder. Recently however, automatic classification offering the same categories to all users has started to appear in some Web mail clients, such as AOL or Gmail. We adopt this approach, rather than previous (unsuccessful) personalized approaches because of the change in the nature of consumer email traffic, which is now dominated by (non-spam) machine-generated email. We propose here a novel approach for (1) automatically distinguishing between personal and machine-generated email and (2) classifying messages into latent categories, without requiring users to have defined any folder. We report how we have discovered that a set of 6 "latent" categories (one for human- and the others for machine-generated messages) can explain a significant portion of email traffic. We describe in details the steps involved in building a Web-scale email categorization system, from the collection of ground-truth labels, the selection of features to the training of models. Experimental evaluation was performed on more than 500 billion messages received during a period of six months by users of Yahoo mail service, who elected to be part of such research studies. Our system achieved precision and recall rates close to 90% and the latent categories we discovered were shown to cover 70% of both email traffic and email search queries. We believe that these results pave the way for a change of approach in the Web mail industry, and could support the invention of new large-scale email discovery paradigms that had not been possible before.

• Publication year

  2014

• Venue

  International Conference on Information and Knowledge Management

• Publication date

  2014-11-03

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/2661829.2662018.arXiv 1606.09296
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Threading machine generated email

  2013cited by this paper
• Email pragmatics and automatic classification: A study in the organizational context

  2012cited by this paper
• Email classification with co-training

  2011cited by this paper
• Automatically tagging email by leveraging other users' folders

  2011influential reference
• Online Learning for Latent Dirichlet Allocation

  2010cited by this paper
• Latent Dirichlet Allocation

  2009cited by this paper
• Sparse Online Learning via Truncated Gradient

  2008cited by this paper
• MapReduce: simplified data processing on large clusters

  2008cited by this paper
• The Enron Corpus: A New Dataset for Email Classi(cid:12)cation Research

  2004cited by this paper
• Machine learning in automated text categorization

  2001cited by this paper
• Simple Mail Transfer Protocol

  2001cited by this paper
• Keystroke level analysis of email message organization

  2000cited by this paper
• Challenges of the Email Domain for Text Classification

  2000cited by this paper
• Na ive-Bayes vs. Rule-Learning in Classification of Email

  1999cited by this paper
• Scalable Internet resource discovery: research problems and approaches

  1994cited by this paper
• Simple Mail Transfer Protocol

  1981cited by this paper

• ‘It Never Occurred to Me That I Would Forget That’: Insights From a Personal Electronic Records Management Prototype Trial

  2025cites this paper
• Content-Based Email Classification at Scale

  2023cites this paper
• EmFore: Online Learning of Email Folder Classification Rules

  2023cites this paper
• Attributes of Personal Electronic Records

  2022cites this paper
• Classifying Emails into Human vs Machine Category

  2021influential citation
• Exploring Email-Prompted Information Needs

  2021cites this paper
• Effects of Past Interactions on User Experience with Recommended Documents

  2020cites this paper
• Rethinking Consumer Email: The Research Process for Yahoo Mail 6

  2020cites this paper
• Screening of Email Box in Portuguese with SVM at Banco do Brasil

  2020cites this paper
• Characterizing Reading Time on Enterprise Emails

  2020cites this paper
• SchedMail: Sender-Assisted Message Delivery Scheduling to Reduce Time-Fragmentation

  2019cites this paper
• Large-Scale Information Extraction from Emails with Data Constraints

  2019cites this paper
• Context-Aware Intent Identification in Email Conversations

  2019cites this paper
• Using Pre-trained Embeddings to Detect the Intent of an Email

  2019cites this paper
• Characterizing and Predicting Email Deferral Behavior

  2019cites this paper
• Exploring User Behavior in Email Re-Finding Tasks

  2019cites this paper
• RiSER: Learning Better Representations for Richly Structured Emails

  2019cites this paper
• Hidden in Plain Sight: Classifying Emails Using Embedded Image Contents

  2018cites this paper
• Characterizing and Supporting Question Answering in Human-to-Human Communication

  2018cites this paper
• Anatomy of a Privacy-Safe Large-Scale Information Extraction System Over Email

  2018cites this paper
• Template Trees: Extracting Actionable Information from Machine Generated Emails

  2018cites this paper
• To Read or To Do? That's The Task: Using Transfer Learning to Detect the Intent of an Email

  2018influential citation
• Evaluation of decentralized email architecture and social network analysis based on email attachment sharing

  2018cites this paper
• Mining information interaction behavior : Academic papers and enterprise emails

  2018cites this paper
• Automated Extractions for Machine Generated Mail

  2018cites this paper
• The Lifetime of Email Messages: A Large-Scale Analysis of Email Revisitation

  2018cites this paper
• Unsubscription: A Simple Way to Ease Overload in Email

  2018cites this paper
• Promoting Relevant Results in Time-Ranked Mail Search

  2017cites this paper
• My home is my post-office: evaluation of a decentralized email architecture on internet-of-things low-end device

  2017cites this paper
• Large-Scale Analysis of Email Search and Organizational Strategies

  2017cites this paper
• Characterizing and Predicting Enterprise Email Reply Behavior

  2017cites this paper
• Learning from User Interactions in Personal Search via Attribute Parameterization

  2017cites this paper
• Characterizing Email Search using Large-scale Behavioral Logs and Surveys

  2017cites this paper
• Learning User Embeddings from Emails

  2017cites this paper
• The Demographics of Mail Search and their Application to Query Suggestion

  2017cites this paper
• Email Category Prediction

  2017cites this paper
• Mailbox-Based vs. Log-Based Query Completion for Mail Search

  2017cites this paper
• Web Mail is not Dead!: It's Just Not Human Anymore

  2017cites this paper
• A Generalized Email Classification System for Workflow Analysis

  2017cites this paper
• RemindMe: Plugging a Reminder Manager into Email for Enhancing Workplace Responsiveness

  2017cites this paper
• Profiling, Modelling and Facilitating Online Activism

  2017cites this paper
• Assessing e-mail intent and tasks in e-mail messages

  2016cites this paper
• Activity Detection from Email Meta-Data Clustering

  2016cites this paper
• Hierarchical Label Propagation and Discovery for Machine Generated Email

  2016cites this paper
• Using Online Controlled Experiments to Examine Authority Effects on User Behavior in Email Campaigns

  2016cites this paper
• InLook: Revisiting Email Search Experience

  2016cites this paper
• You've got Mail, and Here is What you Could do With It!: Analyzing and Predicting Actions on Email Messages

  2016cites this paper
• Alecsa: Attentive Learning for Email Categorization using Structural Aspects

  2016cites this paper
• Structural Clustering of Machine-Generated Mail

  2016cites this paper
• Learning to Rank with Selection Bias in Personal Search

  2016cites this paper
• Enforcing k-anonymity in Web Mail Auditing

  2016cites this paper
• You Will Get Mail!Predicting the Arrival of Future Email

  2015cites this paper
• E-commerce in Your Inbox: Product Recommendations at Scale

  2015cites this paper
• Rank by Time or by Relevance?: Revisiting Email Search

  2015cites this paper
• Evolution of Conversations in the Age of Email Overload

  2015cites this paper
• Semantic Enriched Category Recommendation System for Large-Scale Emails Exploiting Big Data Processing Technologies

  2015cites this paper