Inkeraction: An Interaction Modality Powered by Ink Recognition and Synthesis

Ink is a powerful medium for note-taking and creativity tasks. Multi-touch devices and stylus input have enabled digital ink to be editable and searchable. To extend the capabilities of digital ink, we introduce Inkeraction, an interaction modality powered by ink recognition and synthesis. Inkeraction segments and classifies digital ink objects (e.g., handwriting and sketches), identifies relationships between them, and generates strokes in different writing styles. Inkeraction reshapes the design space for digital ink by enabling features that include: (1) assisting users to manipulate ink objects, (2) providing word-processor features such as spell checking, (3) automating repetitive writing tasks such as transcribing, and (4) bridging with generative models’ features such as brainstorming. Feedback from two user studies with a total of 22 participants demonstrated that Inkeraction supported writing activities by enabling participants to write faster with fewer steps and achieve better writing quality.

• Publication year

  2024

• Venue

  International Conference on Human Factors in Computing Systems

• Publication date

  2024-05-11

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3613904.3642498
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Gorilla: Large Language Model Connected with Massive APIs

  2023cited by this paper
• Character Queries: A Transformer-Based Approach to On-line Handwritten Character Segmentation

  2023influential reference
• Revisiting Relation Extraction in the era of Large Language Models

  2023cited by this paper
• Style Blink: Exploring Digital Inking of Structured Information via Handcrafted Styling as a First-Class Object

  2022cited by this paper
• Eliciting Gestures for Novel Note-taking Interactions

  2021cited by this paper
• Joint stroke classification and text line grouping in online handwritten documents with edge pooling attention networks

  2021cited by this paper
• Hierarchical Recurrent Neural Network for Handwritten Strokes Classification

  2021cited by this paper
• Contextual Stroke Classification in Online Handwritten Documents with Edge Graph Attention Networks

  2020cited by this paper
• Live Sketchnoting Across Platforms: Exploring the Potential and Limitations of Analogue and Digital Tools

  2020cited by this paper
• texSketch: Active Diagramming through Pen-and-Ink Annotations

  2020cited by this paper
• Inking Your Insights: Investigating Digital Externalization Behaviors During Data Analysis

  2019cited by this paper
• ActiveInk: (Th)Inking with Data

  2019cited by this paper
• Fast multi-language LSTM-based online handwriting recognition

  2019cited by this paper
• Flexible and Mindful Self-Tracking: Design Implications from Paper Bullet Journals

  2018cited by this paper
• "The pen is mightier than the keyboard: Advantages of longhand over laptop note taking": Corrigendum.

  2018cited by this paper
• DeepWriting: Making Digital Ink Editable via Deep Generative Modeling

  2018cited by this paper
• Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

  2018cited by this paper
• The Open Images Dataset V4

  2018cited by this paper
• As We May Ink?: Learning from Everyday Analog Pen Use to Improve Digital Ink Experiences

  2017cited by this paper
• Handwriting Text/Non-Text Classification on Mobile Device

  2017cited by this paper
• WritLarge: Ink Unleashed by Unified Scope, Action, & Zoom

  2017cited by this paper
• Freeform digital ink annotations in electronic documents: A systematic mapping study

  2016cited by this paper
• It's Just My History Isn't It?: Understanding Smart Journaling Practices

  2016cited by this paper
• cLuster: Smart Clustering of Free-Hand Sketches on Large Interactive Surfaces

  2015cited by this paper
• Barriers and Negative Nudges: Exploring Challenges in Food Journaling

  2015cited by this paper
• The Cognitive Design of Tools of Thought

  2015cited by this paper
• Designing a Physical Aid to Support Active Reading on Tablets

  2015cited by this paper
• InkAnchor: enhancing informal ink-based note taking on touchscreen mobile phones

  2014cited by this paper
• Constructing Grounded Theory

  2014cited by this paper
• RichReview: blending ink, speech, and gesture to support collaborative document review

  2014cited by this paper
• Handwriting synthesis: classifications and techniques

  2014cited by this paper
• The Pen Is Mightier Than the Keyboard

  2014cited by this paper
• TextTearing: opening white space for digital ink annotation

  2013cited by this paper
• Handwriting beautification using token means

  2013cited by this paper
• Generating Sequences With Recurrent Neural Networks

  2013influential reference
• Drawing the electric: storytelling with conductive ink

  2013cited by this paper
• Microanalysis of active reading behavior to inform design of interactive desktop workspaces

  2012cited by this paper
• Informal information gathering techniques for active reading

  2012cited by this paper
• "Yours is better!": participant response bias in HCI

  2012cited by this paper
• Designing a multi-slate reading environment to support active reading activities

  2012cited by this paper
• Visual Thinking In Action: Visualizations As Used On Whiteboards

  2011cited by this paper
• Active reading and its discontents: the situations, problems and ideas of readers

  2011cited by this paper
• Harpoon selection: efficient selections for ungrouped content on large pen-based surfaces

  2011cited by this paper
• NOOT: a tool for sharing moments of reflection during creative meetings

  2011cited by this paper
• NiCEBook: supporting natural note taking

  2010cited by this paper
• Thinking with external representations

  2010cited by this paper
• IAMonDo-database: an online handwritten document database with non-uniform contents

  2010cited by this paper
• Epistemic action: A measure for cognitive support in tangible user interfaces?

  2009cited by this paper
• Cognitive models of writing

  2009cited by this paper
• Gestures without Libraries , Toolkits or Training : A $ 1 Recognizer for User Interface Prototypes

  2009cited by this paper
• Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes

  2007cited by this paper
• Reading Revisited: Evaluating the Usability of Digital Display Surfaces for Active Reading Tasks

  2007cited by this paper
• InkSeine: In Situ search for active note taking

  2007cited by this paper
• A General Inductive Approach for Analyzing Qualitative Evaluation Data

  2006cited by this paper
• Ink normalization and beautification

  2005cited by this paper
• Cognitive effort during note taking

  2005cited by this paper
• Recognition and beautification of multi-stroke symbols in digital ink

  2005cited by this paper
• Design and analysis of delimiters for selection-action pen gesture phrases in scriboli

  2005cited by this paper
• Informing the design of computer-based environments to support creativity

  2005cited by this paper
• Reflowing digital ink annotations

  2003cited by this paper
• Perceptually-supported image editing of text and graphics

  2003cited by this paper
• Sketching Interfaces: Toward More Human Interface Design

  2001cited by this paper
• Fluid sketches: continuous recognition and morphing of simple hand-drawn shapes

  2000cited by this paper
• A Few Thoughts on Cognitive Overload

  2000cited by this paper
• NotePals: lightweight note sharing by the group, for the group

  1999cited by this paper
• NoteLook: taking notes in meetings with digital video and ink

  1999cited by this paper
• Using note-taking appliances for student to student collaboration

  1999cited by this paper
• Flatland: new dimensions in office whiteboards

  1999cited by this paper
• Beyond paper: supporting active reading with free form digital ink annotations

  1998cited by this paper
• XLibris: the active reading machine

  1998cited by this paper
• Pen-based interaction techniques for organizing material on an electronic whiteboard

  1997cited by this paper
• Annotation: from paper books to the digital library

  1997cited by this paper
• Interactive sketching for the early stages of user interface design

  1995cited by this paper
• Implicit structure for pen-based systems within a freeform interaction paradigm

  1995cited by this paper
• On Distinguishing Epistemic from Pragmatic Action

  1994cited by this paper
• Translucent patches—dissolving windows

  1994cited by this paper
• The cost structure of sensemaking

  1993cited by this paper
• Fitts' Law as a Research and Design Tool in Human-Computer Interaction

  1992cited by this paper
• Designing for error

  1987cited by this paper

• When Handwriting Goes Social: Creativity, Anonymity, and Communication in Graphonymous Online Spaces

  2026cites this paper
• InkFM: A Foundational Model for Full-Page Online Handwritten Note Understanding

  2025cites this paper
• EuterPen: Unleashing Creative Expression in Music Score Writing

  2025cites this paper
• ImaginationVellum: Generative-AI Ideation Canvas with Spatial Prompts, Generative Strokes, and Ideation History

  2025cites this paper
• Handwriting Enhancement: Recognition-Based and Recognition-Independent Approaches for On-Device Online Handwritten Text Alignment

  2024cites this paper