Generating Sequences With Recurrent Neural Networks

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

• Publication year

  2013

• Venue

  arXiv.org

• Publication date

  2013-08-04

• Fields of study

  Computer Science

• Identifiers
  arXiv 1308.0850
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• OF THE UNITED STATES

  2015cited by this paper
• Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets

  2013cited by this paper
• Speech recognition with deep recurrent neural networks

  2013cited by this paper
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

  2012cited by this paper
• A fast and simple algorithm for training neural probabilistic language models

  2012cited by this paper
• Sequence Transduction with Recurrent Neural Networks

  2012cited by this paper
• A Practical Guide to Training Restricted Boltzmann Machines

  2012cited by this paper
• Statistical Language Models Based on Neural Networks

  2012influential reference
• A Machine Learning Perspective on Predictive Coding with PAQ8

  2012cited by this paper
• Practical Variational Inference for Neural Networks

  2011influential reference
• A Machine Learning Perspective on Predictive Coding with PAQ

  2011influential reference
• SUBWORD LANGUAGE MODELING WITH NEURAL NETWORKS

  2011cited by this paper
• Generating Text with Recurrent Neural Networks

  2011cited by this paper
• Factored conditional restricted Boltzmann Machines for modeling motion style

  2009cited by this paper
• The Recurrent Temporal Restricted Boltzmann Machine

  2008cited by this paper
• A Scalable Hierarchical Distributed Language Model

  2008cited by this paper
• Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks

  2008cited by this paper
• The Minimum Description Length Principle (Adaptive Computation and Machine Learning)

  2007cited by this paper
• 2005 Special Issue: Framewise phoneme classification with bidirectional LSTM and other neural network architectures

  2005cited by this paper
• IAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard

  2005cited by this paper
• Learning Precise Timing with LSTM Recurrent Networks

  2003cited by this paper
• A First Look at Music Composition using LSTM Recurrent Neural Networks

  2002cited by this paper
• Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies

  2001cited by this paper
• Better Generative Models for Sequential Data Problems: Bidirectional Recurrent Mixture Density Networks

  1999cited by this paper
• Long Short-Term Memory

  1997influential reference
• An analysis of noise in recurrent neural networks: convergence and generalization

  1996influential reference
• Gradient-based learning algorithms for recurrent networks and their computational complexity

  1995cited by this paper
• Mixture Density Networks

  1994cited by this paper
• Learning long-term dependencies with gradient descent is difficult

  1994cited by this paper
• Building a Large Annotated Corpus of English: The Penn Treebank

  1993cited by this paper
• Neural Networks for Pattern Recognition

  1993cited by this paper
• The tagged LOB Corpus : user's manual

  1986cited by this paper
• Data Compression Using Adaptive Coding and Partial String Matching

  1984cited by this paper

• Controlling Repetition in Protein Language Models

  2026cites this paper
• Harnessing Selective State Space Models to Enhance Semianalytical Design of Fabrication-Ready Multilayered Huygens'Metasurfaces: Part II - Generative Inverse Design (MetaMamba)

  2026cites this paper
• Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory

  2026cites this paper
• GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization

  2026cites this paper
• Enhancing pedestrian trajectory prediction with dynamic interaction pruning and dual-path encoding

  2026cites this paper
• AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization

  2026cites this paper
• Rational Transductors

  2026cites this paper
• A Conditional GAN-based Framework for Generating Stealthy Adversarial Attacks on Autonomous Vehicle Trajectory Prediction Systems

  2026cites this paper
• Toward an Ultra-Lightweight Text Representation: Causal Convolutional Networks With Feature Self-Enhancement Mechanisms

  2026cites this paper
• A Transfer Learning CNN approach for automated Plant Growth Temporal Labelling: Addressing Class-Based Variability Paradox and introducing novel metrics

  2026cites this paper
• Effective document summarization: a hybrid clustering approach using transformer model

  2026cites this paper
• Unsupervised Anomaly Detection in Multi-Agent Trajectory Prediction via Transformer-Based Models

  2026cites this paper
• PWVFnet: A Short-Time Troposphere PWV Forecast Model Combined Empirical Models and Spatiotemporal ConvLSTM Network

  2026cites this paper
• Proxy Compression for Language Modeling

  2026cites this paper
• Gradient Regularized Natural Gradients

  2026cites this paper
• Enhancing Transformer-Based Language Models for Hungarian Handwritten Text Recognition

  2026cites this paper
• ScribeTokens: Fixed-Vocabulary Tokenization of Digital Ink

  2026influential citation
• Research on network security situation awareness based on graph neural networks

  2026cites this paper
• On the Convergence of HalpernSGD

  2026cites this paper
• A Hybrid Architecture Combining Physical Modeling and Neural Networks for Piano Sound Synthesis

  2026cites this paper
• Learning-Based Fault Identification and Fault-Tolerant Control for Quadruped Robots With Small Samples

  2026cites this paper
• One-Shot Identification with Different Neural Network Approaches

  2026cites this paper
• EyeLayer: Integrating Human Attention Patterns into LLM-Based Code Summarization

  2026cites this paper
• Privacy-Preserving EEG Data Generation: A Federated Split Learning Approach Using Privacy-Adaptive Autoencoders and Secure Aggregation with GFlowNet

  2025cites this paper
• Retrieval of Surface Solar Radiation through Implicit Albedo Recovery from Temporal Context

  2025cites this paper
• IoT-Enabled Traffic Management System Using Vehicle Count Prediction in a Semantic Communication Framework

  2025cites this paper
• Deep switching state space model for nonlinear time series forecasting with regime switching

  2025cites this paper
• PRO: Projection Domain Synthesis for CT Imaging

  2025cites this paper
• Developing a novel layer network structure for a LSTM model to predict mean monthly river streamflow

  2025cites this paper
• Transformer model with external token memories and attention for PersonaChat

  2025cites this paper
• Advances in conceptual process design: From conventional strategies to AI-assisted methods

  2025cites this paper
• Attention-Guided Deep Reinforcement Learning for Realistic Neural Painting

  2025cites this paper
• Where Is the Ball: 3D Ball Trajectory Estimation From 2D Monocular Tracking

  2025cites this paper
• Two-Stage Feature Generation with Transformer and Reinforcement Learning

  2025cites this paper
• WriteViT: Handwritten Text Generation with Vision Transformer

  2025cites this paper
• Distinguishing Between Formal and Colloquial: A Multilingual BERT Approach to Bengali Language Classification

  2025cites this paper
• Winner-takes-all for Multivariate Probabilistic Time Series Forecasting

  2025cites this paper
• Document-level relation extraction model guided by global information and external contextual knowledge

  2025cites this paper
• One-shot handwriting imitation via self-supervised cross spatial transformer networks

  2025cites this paper
• Diffusion Models and Generative Artificial Intelligence: Frameworks, Applications and Challenges

  2025cites this paper
• Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory

  2025cites this paper
• BMFNet: Bidirectional Multimodal Fusion Network for image captioning

  2025cites this paper
• Zero-Shot Styled Text Image Generation, but Make It Autoregressive

  2025cites this paper
• An adaptive hierarchical hybrid kernel ELM optimized by aquila optimizer algorithm for bearing fault diagnosis

  2025influential citation
• Agent-Q: Fine-Tuning Large Language Models for Quantum Circuit Generation and Optimization

  2025cites this paper
• Effect of pore water pressure on slope stability in a heavy-haul railway embankment using a deep learning approach

  2025cites this paper
• Hybrid CNN–GRU model for hourly flood forecasting index: case studies from the Fiji islands

  2025cites this paper
• Physiological Radar-Based Sedentary State Classification using Recurrent Neural Networks

  2025cites this paper
• Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking

  2025cites this paper
• MyanSpeech: Joint CTC-Attention and RNN Language Model for End-To-End Read Speech Recognition

  2025cites this paper
• Satellite Image Chronology Prediction for Sequence Modeling

  2025cites this paper
• Eco-Driving Decision Making Based on V2X Communication and Spatio-Temporal Prediction of Pedestrians

  2025cites this paper
• Prediction of yearly mean sunspot number using machine learning methods

  2025cites this paper
• From RNNs to Transformers and Beyond: a Deep Dive into Intent Detection in Goal-oriented Conversational Agents

  2025cites this paper
• IntTrajSim: Trajectory Prediction for Simulating Multi-Vehicle driving at Signalized Intersections

  2025cites this paper
• Tailoring orbital angular momentum entanglement through inversed design based on gradient descent

  2025cites this paper
• Tversky Neural Networks: Psychologically Plausible Deep Learning with Differentiable Tversky Similarity

  2025cites this paper
• Quantum-Inspired Differentiable Integral Neural Networks (QIDINNs): A Feynman-Based Architecture for Continuous Learning Over Streaming Data

  2025cites this paper
• Temporal attention to estimate continuous blood pressure using photoplethysmography signals

  2025cites this paper
• DRoLaS: Diffusion-Based Coarse-to-Fine Conditional Synthesis of Hierarchical Road Layouts

  2025cites this paper
• Improved storm surge prediction and extreme sea level future projections in the Indian Ocean using deep learning

  2025cites this paper
• Advancements in noise reduction for wheel speed sensing using enhanced LSTM models

  2025cites this paper
• QuGStep: Refining step size selection in gradient estimation for variational quantum algorithms

  2025cites this paper
• Leveraging artificial intelligence (AI) for data collection and analysis: the case of and application in corporate social responsibility (CSR) through sport research

  2025cites this paper
• Aligning to What? Limits to RLHF Based Alignment

  2025cites this paper
• Deep Learning Ensemble Approach for Predicting Expected and Confidence Levels of Signal Phase and Timing Information at Actuated Traffic Signals

  2025cites this paper
• Speech Re-Painting for Robust ASR

  2025cites this paper
• Predicting wheat yield in agricultural industry using deep learning techniques: a review

  2025influential citation
• Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages

  2025cites this paper
• Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs

  2025cites this paper
• A Spatial Attention-Based Transductive Long Short-Term Memory for Semantic Parsing Trained with a Semi-Supervised Method

  2025cites this paper
• Emergence of the Primacy Effect in Structured State-Space Models

  2025cites this paper
• A Pipeline of Augmentation and Sequence Embedding for Classification of Imbalanced Network Traffic

  2025cites this paper
• A deep learning-based system for automatic detection of emesis with high accuracy in Suncus murinus

  2025influential citation
• Gradient Correction in Federated Learning with Adaptive Optimization

  2025cites this paper
• On Self-Adaptive Perception Loss Function for Sequential Lossy Compression

  2025cites this paper
• Quantum Recurrent Neural Networks with Encoder-Decoder for Time-Dependent Partial Differential Equations

  2025cites this paper
• TPST: A Traffic Flow Prediction Model Based on Spatial–Temporal Identity

  2025cites this paper
• A dual adversarial structure of generative adversarial network for nature language generation

  2025cites this paper
• QSA-QConvLSTM: A Quantum Computing-Based Approach for Spatiotemporal Sequence Prediction

  2025cites this paper
• Invisible Strings: Revealing Latent Dancer-to-Dancer Interactions with Graph Neural Networks

  2025influential citation
• Evidential Uncertainty Estimation for Multi-Modal Trajectory Prediction

  2025cites this paper
• Combining Conv-LSTM and wind-wave data for enhanced sea wave forecasting in the Mediterranean Sea

  2025cites this paper
• Predicting the Temperature of a Permanent Magnet Synchronous Motor: A Comparative Study of Artificial Neural Network Algorithms

  2025influential citation
• A Personalized Data-Driven Generative Model of Human Motion

  2025cites this paper
• De-Confounding Feature Fusion Transformer Network for Image Captioning in Assistive Navigation Applications for the Visually Impaired

  2025cites this paper
• Bridging Language Models and Financial Analysis

  2025cites this paper
• Forecasting motion trajectories of elbow and knee joints during infant crawling based on long–short-term memory (LSTM) networks

  2025cites this paper
• Improving World Models using Deep Supervision with Linear Probes

  2025cites this paper
• Noisy Deep Ensemble: Accelerating Deep Ensemble Learning via Noise Injection

  2025cites this paper
• Investigation of the effect of climatic parameters in machine learning algorithms for streamflow predicting in Hamoon Helmand Catchment, Iran

  2025cites this paper
• HVAdam: A Full-Dimension Adaptive Optimizer

  2025cites this paper
• Preconditioning Natural and Second Order Gradient Descent in Quantum Optimization: A Performance Benchmark

  2025cites this paper
• Comparative Study on Curiosity with Attention, Memory and Empowerment

  2025cites this paper
• Short-Term Electricity Load Forecasting Based on T-CFSFDP Clustering and Stacking-BiGRU-CBAM

  2025cites this paper
• Bringing Suzhou Numerals into the Digital Age: A Dataset and Recognition Study on Ancient Chinese Trade Records

  2025cites this paper
• An Attention-Based CNN-LSTM Model to Diagnose Break Size in Loss-of-Coolant Accidents

  2025cites this paper
• FuMi: A Runtime Fuzz-based Machine Learning Precision Measurement and Testing Framework

  2025cites this paper
• Plausible Vehicle Trajectory Augmentation With Hybrid Machine Learning

  2025cites this paper
• Learning based lower limb joint kinematic estimation using open source IMU data

  2025cites this paper