MLlib: Machine Learning in Apache Spark

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.

• Publication year

  2015

• Venue

  Journal of machine learning research

• Publication date

  2015-05-26

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1505.06807
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Spark SQL: Relational Data Processing in Spark

  2015influential reference
• TuPAQ: An Efficient Planner for Large-scale Predictive Analytic Queries

  2015cited by this paper
• Automating model search for large scale machine learning

  2015cited by this paper
• GraphX: Graph Processing in a Distributed Dataflow Framework

  2014cited by this paper
• API design for machine learning software: experiences from the scikit-learn project

  2013cited by this paper
• Hidden factors and hidden topics: understanding rating dimensions with review text

  2013cited by this paper
• MLI: An API for Distributed Machine Learning

  2013cited by this paper
• MLbase: A Distributed Machine-learning System

  2013cited by this paper
• Discretized streams: fault-tolerant streaming computation at scale

  2013cited by this paper
• Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

  2012cited by this paper
• Scikit-learn: Machine Learning in Python

  2011cited by this paper
• PMML: An Open Standard for Sharing Models

  2009cited by this paper
• Matrix Factorization Techniques for Recommender Systems

  2009cited by this paper
• Latent Dirichlet Allocation

  2009cited by this paper
• PLANET: Massively Parallel Learning of Tree Ensembles with MapReduce

  2009cited by this paper
• MapReduce: simplified data processing on large clusters

  2008influential reference

• Optimising Image Feature Extraction and Selection: A Comprehensive Review With Spark Case Studies

  2026cites this paper
• bigMICE: Multiple Imputation of Big Data

  2026cites this paper
• cspray: Distributed Single Cell Transcriptome Analysis

  2026influential citation
• CACTUSDB: Unlock Co-Optimization Opportunities for SQL and AI/ML Inferences

  2026cites this paper
• The value of product-specific first-meet coupons: Enhancing customer acquisitions and sales through causal forest

  2026cites this paper
• Optimized Big Data Handling with Cloud-Based Parallel Divide and Conquer Techniques (A Review)

  2025cites this paper
• A Scalable Framework for Big Data Analytics in Psychological Research: Leveraging Distributed Systems and Cluster Management

  2025cites this paper
• Beginner-Friendly Review of Research on R-Based Energy Forecasting: Insights from Text Mining

  2025cites this paper
• Scalable Parallelization of Machine Learning Models Using Hybrid Data-Model Partitioning

  2025cites this paper
• Leveraging the supply chain costs and lead time impact of an innovative blockchain technology supply chain management application: the TrustyChain case study

  2025cites this paper
• Distributed Hierarchical Decomposition Framework for Multimodal Timeseries Prediction

  2025cites this paper
• Spatially inhomogeneous response of precipitation δ18O in China to ENSO cycles

  2025cites this paper
• Industrial brain: a human-like autonomous neuro-symbolic cognitive decision-making system

  2025cites this paper
• Enhancing performance of E-Government information systems with SSD-based Hadoop mapreduce

  2025cites this paper
• Unsupervised Learning for Intelligent Driver Profiling

  2025cites this paper
• Comparative Study of Clustering Algorithms for Geospatial Legal Data Analysis

  2025cites this paper
• Predicting time execution in fog computing using neural network and input evaluation with Interpretive Structural Modeling (ISM-ANN)

  2025cites this paper
• Machine Learning-Based News Classification in Big Data Environments

  2025cites this paper
• Synergistic Distributed CNN Model for Protein Classification With a Collaborative BSP Synchronization Based on LSTM Prediction

  2025cites this paper
• Advancing Trustworthy AI: A Comparative Evaluation of AI Robustness Toolboxes

  2025cites this paper
• Dynamic Workload Management System in the Public Sector: A Comparative Analysis

  2025cites this paper
• Spark-based Improved Ant Colony Algorithm for Solving TSP Problems

  2025cites this paper
• Mathematical and Artificial Intelligence Techniques in Modern Drug Discovery: A Review

  2025cites this paper
• Distribution-Aware Exploration for Adaptive HNSW Search

  2025cites this paper
• ScalaGBM: Memory Efficient GBDT Training for High-Dimensional Data on GPU

  2025cites this paper
• A faster heterogeneous parallel computing method for Tucker decomposition

  2025cites this paper
• Navigating Data Errors in Machine Learning Pipelines: Identify, Debug, and Learn

  2025cites this paper
• Data-Driven Mobility System for Vehicular Communication: A Step Towards Transport Resilience in Intelligent Transport Systems

  2025cites this paper
• Improved Isolated Forest Algorithm and Application

  2025cites this paper
• Architectural Requirements for Digital Twin Development Platform: A Review of Scientific Literature and Practical Considerations

  2025cites this paper
• AI-Driven F2N-ECC Framework for Secure, Real-Time E-Commerce Transactions

  2025cites this paper
• A segment anything model-based geological remote sensing interpretation method with a distributed data-parallel deep learning framework

  2025cites this paper
• A Survey on Parameter Server Architecture: Approaches for Optimizing Distributed Centralized Learning

  2025cites this paper
• Gpu-enhanced scalable methods for genome sequence feature extraction and clustering

  2025cites this paper
• High-Dimensional Data Processing: Benchmarking Machine Learning and Deep Learning Architectures in Local and Distributed Environments

  2025cites this paper
• A Comprehensive Review of Parallel Optimization Algorithms for High-Dimensional and Incomplete Matrix Factorization

  2025cites this paper
• Cost-Efficient Cloud Infrastructure with Hugepage-aware Memory Deduplication

  2025cites this paper
• Benchmarking Database Query Engines for Cross-Source Data Access

  2025cites this paper
• Data Systems Education: Curriculum Recommendations, Course Syllabi, and Industry Needs

  2025cites this paper
• Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-training

  2025cites this paper
• Artificial Intelligence-Assisted Microfluidic Bio-Imaging—From Sensors to Applications: A Review

  2025cites this paper
• MEMPHIS: Holistic Lineage-based Reuse and Memory Management for Multi-backend ML Systems

  2025cites this paper
• Predicting Performance Variability

  2025cites this paper
• Orchestrating Distributed Deep Learning atop Spark: A Systems-Centric Pathway for Scalable Big Data Training

  2025cites this paper
• SparseKmeans: Efficient K-means Clustering For Sparse Data

  2025cites this paper
• Pemanfaatan Machine Learning untuk Optimasi Big Data dalam Sistem Informatika Modern

  2025cites this paper
• Nime: a native in-memory compute framework for cluster computing

  2025cites this paper
• Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflation

  2025cites this paper
• Secure Data Analytics in Apache Spark with Fine-grained Policy Enforcement and Isolated Execution

  2025influential citation
• Scaling Up Optuna: P2P Distributed Hyperparameters Optimization

  2025cites this paper
• FreewayML: An Adaptive and Stable Streaming Learning Framework for Dynamic Data Streams

  2025cites this paper
• PIMDAL: Mitigating the Memory Bottleneck in Data Analytics using a Real Processing-in-Memory System

  2025cites this paper
• Approach to Deploying Batch File Data Products in a Big Data Environment

  2025cites this paper
• Performance Insights of Convolutional Neural Networks Operating on Distributed Computing Platforms

  2025cites this paper
• Live streaming recommendation based on multiple types of repeated behaviors

  2025cites this paper
• Patchwork: A Unified Framework for RAG Serving

  2025cites this paper
• Applicability Assessment of Technologies for Predictive and Prescriptive Analytics of Nephrology Big Data

  2025cites this paper
• Navigating the Edge-Cloud Continuum: A State-of-Practice Survey

  2025cites this paper
• A comprehensive survey of imputation methods in medical missing data analysis

  2025cites this paper
• Statistical Methods for Big Data Analysis in Technological Systems

  2025cites this paper
• Hierarchical Isomerism Distributed Equivalent Union Find for Billion-Scale Disjoint Sets: A Case Study

  2025cites this paper
• Bigsids: an efficient SDN-based network intrusion detection systems for big data environments

  2025cites this paper
• AgnoSVD: Dynamic resource allocation for serverless workloads using collaborative filtering

  2025cites this paper
• In-Memory Versus Disk-Based Computing with Random Forest for Stock Analysis: A Comparative Study

  2025cites this paper
• Towards Regaining Control Over Messy Machine Learning Pipelines

  2025cites this paper
• The Computational Efficiency of Task-Based Parallelism of Spark MLlib

  2025cites this paper
• FEVERLESS: Fast and Secure Vertical Federated Learning Based on XGBoost for Decentralized Labels

  2024cites this paper
• A Memory-Efficient Hybrid Parallel Framework for Deep Neural Network Training

  2024cites this paper
• Comparison of Fair and FIFO Schedulers on Cloud Cluster of Apache Spark

  2024cites this paper
• Toward Inference Delivery Networks: Distributing Machine Learning With Optimality Guarantees

  2024cites this paper
• A Survey on Scheduling Techniques in Computing and Network Convergence

  2024cites this paper
• Scalable and fault-tolerant algorithms for big data processing in distributed cloud architectures

  2024cites this paper
• Real-Time Visualization and Detection of Malicious Network Flows in IoT Devices using a Scalable Stream Processing Pipeline

  2024cites this paper
• ACID Support for Compute eXpress Link Memory Transactions

  2024cites this paper
• EmpireDB: Data System to Accelerate Computational Sciences

  2024cites this paper
• A Planner for Scalable Tensor Programs

  2024cites this paper
• Lactose prediction in dry milk with hyperspectral imaging: A data analysis competition at the “International Workshop on Spectroscopy and Chemometrics 2024”

  2024cites this paper
• Disaggregated RDDs: Extending and Analyzing Apache Spark for Memory Disaggregated Infrastructures

  2024cites this paper
• Intelligent Spark Agents: A Modular LangGraph Framework for Scalable, Visualized, and Enhanced Big Data Machine Learning Workflows

  2024cites this paper
• Large–Scale Knowledge Graph Embeddings in Apache Spark

  2024cites this paper
• ASTM :Autonomous Smart Traffic Management System Using Artificial Intelligence CNN and LSTM

  2024cites this paper
• Optimization of frequent item set mining parallelization algorithm based on spark platform

  2024cites this paper
• Artificial Intelligence-Driven Analysis of Telehealth Effectiveness in Youth Mental Health Services: Insights from SAMHSA Data

  2024cites this paper
• Automated Provenance-Based Screening of ML Data Preparation Pipelines

  2024cites this paper
• Network intrusion detection in big datasets using Spark environment and incremental learning

  2024cites this paper
• Spark Collaborative Filtering Based Food Recommendation System

  2024cites this paper
• Research on Vanadium and Titanium Culture Knowledge Recommendation Model Based on Collaborative Filtering Algorithm on the Spark Platform

  2024cites this paper
• Realizing the Potential of Big Data Analytics through Apache Spark MLlib

  2024cites this paper
• STRATA: Random Forests going Serverless

  2024cites this paper
• Analysis and Comparison of Dockerized and Standalone Apache Spark Configurations for Efficient Distributed Data Processing

  2024cites this paper
• Dealing With Class Imbalance in Uplift Modeling-Efficient Data Preprocessing via Oversampling and Matching

  2024cites this paper
• Big Data Analytics using Artificial Intelligence: Apache Spark for Scalable Batch Processing

  2024cites this paper
• MegaMmap: Blurring the Boundary Between Memory and Storage for Data-Intensive Workloads

  2024cites this paper
• Novel Collaborative Intrusion Detection for Enhancing Cloud Security

  2024cites this paper
• Distributed Deep Learning Based on Edge Computing Over Internet of Vehicles: Overview, Applications, and Challenges

  2024cites this paper
• Deep Learning Frontiers in 3D Object Detection: A Comprehensive Review for Autonomous Driving

  2024cites this paper
• Agile-Ant: Self-managing Distributed Cache Management for Cost Optimization of Big Data Applications

  2024cites this paper
• Big Data and Java are integrated with machine learning

  2024cites this paper
• Predicting Song Popularity Using Data Analysis

  2024cites this paper
• Enhancing KBQA Performance in Large-Scale Chinese Knowledge Graphs Using Apache Spark

  2024cites this paper