DrawTalking: Towards Building Interactive Worlds by Sketching and Speaking

We introduce DrawTalking, a prototype system enabling an approach that empowers users to build interactive worlds by sketching and speaking. The approach emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study shows the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.

• Publication year

  2024

• Venue

  CHI Extended Abstracts

• Publication date

  2024-05-11

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3613905.3651089
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Visual StoryCoder: A Multimodal Programming Environment for Children’s Creation of Stories

  2023cited by this paper
• CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and Collaboration

  2023cited by this paper
• Drawing as a versatile cognitive tool

  2023cited by this paper
• Visual Captions: Augmenting Verbal Communication with On-the-fly Visuals

  2023cited by this paper
• RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling

  2022cited by this paper
• Constructing Embodied Algebra by Sketching

  2021cited by this paper
• Understanding the Design Space for Animated Narratives Applied to Illustrations

  2021cited by this paper
• Rapido: Prototyping Interactive AR Experiences through Programming by Demonstration

  2021cited by this paper
• texSketch: Active Diagramming through Pen-and-Ink Annotations

  2020cited by this paper
• Scones: towards conversational authoring of sketches

  2020cited by this paper
• Crosspower: Bridging Graphics and Linguistics

  2020cited by this paper
• RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching

  2020cited by this paper
• Vocal Shortcuts for Creative Experts

  2019cited by this paper
• TakeToons: Script-driven Performance Animation

  2018cited by this paper
• Dynamic Brushes: Extending Manual Drawing Practices with Artist-Centric Programming Tools

  2018cited by this paper
• Chalktalk : A Visualization and Communication Language - As a Tool in the Domain of Computer Science Education

  2018influential reference
• User-Guided Synthesis of Interactive Diagrams

  2017cited by this paper
• Deixis in Modern Linguistics

  2017cited by this paper
• A Make-Believe Narrative for HCI

  2016cited by this paper
• Humane representation of thought: a trail map for the 21st century

  2014cited by this paper
• Draco: bringing life to illustrations with kinetic textures

  2014cited by this paper
• Kitty: sketching dynamic and interactive illustrations

  2014cited by this paper
• Put it there

  2013cited by this paper
• PixelTone: a multimodal interface for image editing

  2013cited by this paper
• SketchStory: Telling More Engaging Stories with Data through Freeform Sketching

  2013cited by this paper
• Physink: sketching physical behavior

  2013cited by this paper
• Playing with Genre: User-Generated Game Design in LittleBigPlanet 2

  2012cited by this paper
• Visualizing Thought

  2011cited by this paper
• The Scratch Programming Language and Environment

  2010cited by this paper
• Scratch

  2009cited by this paper
• K-sketch: a 'kinetic' sketch pad for novice animators

  2008cited by this paper
• Evaluating user interface systems research

  2007cited by this paper
• MathPad2: a system for the creation and exploration of mathematical sketches

  2004cited by this paper
• WordsEye: an automatic text-to-scene conversion system

  2001cited by this paper
• Improv: a system for scripting interactive actors in virtual worlds

  1996cited by this paper
• SILK: sketching interfaces like krazy

  1996cited by this paper
• Smalltalk-80: The Language and Its Implementation

  1983cited by this paper
• “Put-that-there”: Voice and gesture at the graphics interface

  1980cited by this paper
• Understanding natural language

  1972cited by this paper
• The on-line graphical specification of computer procedures

  1966cited by this paper
• Sketchpad: a man-machine graphical communication system

  1963cited by this paper
• Dreams

  1920influential reference
• Sketchpad a Man-Machine Graphical Communication System

  1899cited by this paper

• ImprovMate: Multimodal AI Assistant for Improv Actor Training

  2025cites this paper
• AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools

  2025cites this paper
• FusAIn: Composing Generative AI Visual Prompts Using Pen-based Interaction

  2025cites this paper
• Shape n’ Swarm: Hands-on, Shape-aware Generative Authoring with Swarm UI and LLMs

  2025cites this paper
• SketchGPT: A Sketch-based Multimodal Interface for Application-Agnostic LLM Interaction

  2025cites this paper
• DrawTalking: Building Interactive Worlds by Sketching and Speaking

  2024cites this paper
• Shape n’ Swarm: Hands-on, Shape-aware Generative Authoring for Swarm UI using LLMs

  year unknowncites this paper