Sikuli: using GUI screenshots for search and automation

We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.

• Publication year

  2009

• Venue

  ACM Symposium on User Interface Software and Technology

• Publication date

  2009-10-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/1622176.1622213
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• CoScripter: automating & sharing how-to knowledge in the enterprise

  2008cited by this paper
• Assieme: finding and leveraging implicit references in a web search interface for programmers

  2007cited by this paper
• Graphstract: minimal graphical help for computers

  2007influential reference
• Multiple Object Class Detection with a Generative Model

  2006cited by this paper
• An evaluation of using programming by demonstration and guided walkthrough techniques for authoring and utilizing documentation

  2006cited by this paper
• Stencils-based tutorials: design and evaluation

  2005cited by this paper
• DocWizards: a system for authoring follow-me documentation wizards

  2005cited by this paper
• Automation and customization of rendered web pages

  2005cited by this paper
• Robust wide-baseline stereo from maximally stable extremal regions

  2004cited by this paper
• Video Google: a text retrieval approach to object matching in videos

  2003cited by this paper
• Computer Vision: A Modern Approach

  2002cited by this paper
• Programming by example: visual generalization in programming by example

  2000cited by this paper
• Pixel data access: interprocess communication in the user interface for end-user programming and graphical macros

  1999cited by this paper
• Object recognition from local scale-invariant features

  1999influential reference
• A visual medium for programmatic control of interactive applications

  1999cited by this paper
• A comparison of still, animated, or nonillustrated on-line help with written or spoken instructions in a graphical user interface

  1995cited by this paper
• Automatic generation of task-oriented help

  1995cited by this paper
• Apple guide: a case study in user-aided design of online help

  1995cited by this paper
• Automatic generation of help from interface design models

  1994cited by this paper
• Coupling a UI framework with automatic generation of context-sensitive animated help

  1990cited by this paper

• An Empirical Study of Web Visual Flakiness: Characterisation and Fix Strategies

  2026cites this paper
• HIDAgent: A Toolkit Enabling"Personal Agents"on HID-Compatible Devices

  2026cites this paper
• Recognition of GUI Web Elements using the Deep Learning Approach and Yolo Architecture

  2026cites this paper
• A System-Level Testing Framework for Automated Assessment of Programming Assignments Allowing Students Object-Oriented Design Freedom

  2025cites this paper
• From Operation to Cognition: Automatic Modeling Cognitive Dependencies from User Demonstrations for GUI Task Automation

  2025cites this paper
• Agent for User: Testing Multi - User Interactive Features in TikTok

  2025cites this paper
• From Inquisitorial to Adversarial: Using Legal Theory to Redesign Online Reporting Systems

  2025cites this paper
• What Did My Users Experience? Discovering Visual Stimuli on Graphical User Interfaces of the Web

  2025cites this paper
• ImageR: Enhancing Bug Report Clarity by Screenshots

  2025cites this paper
• A Domain-Agnostic Framework for Visual Element Detection From High-Level Descriptions

  2025cites this paper
• SnapNCode: An Integrated Development Environment for Programming Physical Objects Interactions

  2025cites this paper
• Breaking Single-Tester Limits: Multi-Agent LLMs for Multi-User Feature Testing

  2025cites this paper
• Detecting Outdated Screenshot from GUI Document

  2025cites this paper
• Model-based GUI automation

  2025cites this paper
• GUIPilot: A Consistency-Based Mobile GUI Testing Approach for Detecting Application-Specific Bugs

  2025cites this paper
• GenGUI: A Dataset for Automatic Generation of Web User Interfaces Using ChatGPT

  2025cites this paper
• Automated Monitoring of Web User Interfaces

  2025cites this paper
• Explorer: Robust Collection of Interactable GUI Elements

  2025cites this paper
• Fit prediction for filtering facepiece respirator using 3D face shape.

  2025cites this paper
• Enhanced Crowdsourced Test Report Prioritization via Image-and-Text Semantic Understanding and Feature Integration

  2025cites this paper
• TVTAT - A Real Time Camera Imaging Testing Tool for Smart TVs: Preliminary Results

  2024cites this paper
• Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention Mechanism

  2024cites this paper
• Context-Dependent Interactable Graphical User Interface Element Detection for Spatial Computing Applications

  2024cites this paper
• AQuA: Automated Question-Answering in Software Tutorial Videos with Visual Anchors

  2024cites this paper
• Ranking approaches for similarity-based web element location

  2024cites this paper
• MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

  2024cites this paper
• Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM

  2024cites this paper
• On Effectiveness and Efficiency of Gamified Exploratory GUI Testing

  2024cites this paper
• UIClip: A Data-driven Model for Assessing User Interface Design

  2024cites this paper
• Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps

  2024cites this paper
• Investigating the robustness of locators in template-based Web application testing using a GUI change classification model

  2023cites this paper
• A Comprehensive Survey on Program Synthesis With Evolutionary Algorithms

  2023cites this paper
• OmniFill: Domain-Agnostic Form Filling Suggestions Using Multi-Faceted Context

  2023cites this paper
• Clonemator: Composing Spatiotemporal Clones to Create Interactive Automators in Virtual Reality

  2023cites this paper
• DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models

  2023cites this paper
• Having Difficulty Understanding Manuals? Automatically Converting User Manuals into Instructional Videos

  2023cites this paper
• Evaluating a Large Language Model on Searching for GUI Layouts

  2023cites this paper
• Challenges of End-to-End Testing with Selenium WebDriver and How to Face Them: A Survey

  2023cites this paper
• Task Automation Intelligent Agents: A Review

  2023cites this paper
• IoTeach:Learning Support System for IoT Programming by Integrating Real Devices and Sequential Contents

  2023cites this paper
• Colaroid: A Literate Programming Approach for Authoring Explorable Multi-Stage Tutorials

  2023cites this paper
• NiCro: Purely Vision-based, Non-intrusive Cross-Device and Cross-Platform GUI Testing

  2023cites this paper
• WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics

  2023cites this paper
• A Pairwise Dataset for GUI Conversion and Retrieval between Android Phones and Tablets

  2023cites this paper
• Why do people take Screenshots on their Smartphones?

  2023cites this paper
• Widget Detection-based Testing for Industrial Mobile Games

  2023cites this paper
• DiLogics: Creating Web Automation Programs with Diverse Logics

  2023cites this paper
• ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

  2023cites this paper
• Robust web element identification for evolving applications by considering visual overlaps

  2023cites this paper
• Idle Strategy of Smart Monkey to Enhance Testing Operable GUI Regions

  2023cites this paper
• Video2Action: Reducing Human Interactions in Action Annotation of App Tutorial Videos

  2023cites this paper
• Vision-Based Mobile App GUI Testing: A Survey

  2023cites this paper
• Relation-enhanced DETR for Component Detection in Graphic Design Reverse Engineering

  2023cites this paper
• ReSPlay: Improving Cross-Platform Record-and-Replay with GUI Sequence Matching

  2023cites this paper
• SemanticOn: Specifying Content-Based Semantic Conditions for Web Automation Programs

  2022cites this paper
• Synapse

  2022cites this paper
• Task and Process Capturing Toolkit using GUI Automation

  2022cites this paper
• Visual Identification of Mobile App GUI Elements for Automated Robotic Testing

  2022cites this paper
• Describing UI Screenshots in Natural Language

  2022cites this paper
• Automated Visual Testing for Mobile Apps in an Industrial Setting

  2022cites this paper
• Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinements

  2022cites this paper
• Web Element Identification by Combining NLP and Heuristic Search for Web Testing

  2022cites this paper
• Sensing Hand Interactions with Everyday Objects by Profiling Wrist Topography

  2022cites this paper
• PSDoodle: Searching for App Screens via Interactive Sketching

  2022cites this paper
• MIT Open Access Articles Associating the visual representation of user interfaces with their internal structures and metadata

  2022influential citation
• Real Time Detection of Mobile Graphical User Interface Elements Using Convolutional Neural Networks

  2022cites this paper
• PSDoodle: Fast App Screen Search via Partial Screen Doodle

  2022cites this paper
• GUI-Guided Test Script Repair for Mobile Apps

  2022cites this paper
• Programming cobots by voice: a pragmatic, web-based approach

  2022cites this paper
• Towards Better Semantic Understanding of Mobile Interfaces

  2022cites this paper
• PSDoodle

  2022cites this paper
• A Smart Automatic Agent Operating System Using Screen Analysis And Remote Control

  2022cites this paper
• Intelligent Software Testing based on Visual Feedback

  2022cites this paper
• Sidereal: Statistical adaptive generation of robust locators for web testing

  2021cites this paper
• HapticSeer: A Multi-channel, Black-box, Platform-agnostic Approach to Detecting Video Game Events for Real-time Haptic Feedback

  2021influential citation
• Screen2Vec: Semantic Embedding of GUI Screens and GUI Components

  2021cites this paper
• DIY assistant: a multi-modal end-user programmable virtual assistant

  2021cites this paper
• Model-Based Ideal Testing of GUI Programs–Approach and Case Studies

  2021cites this paper
• VINS: Visual Search for Mobile User Interface Design

  2021cites this paper
• Demonstration + Natural Language: Multimodal Interfaces for GUI-Based Interactive Task Learning Agents

  2021cites this paper
• Can the User Help? Leveraging User Actions for Network Profiling

  2021cites this paper
• NLP-assisted Web Element Identification Toward Script-free Testing

  2021cites this paper
• Realistic simulation of users for IT systems in cyber ranges

  2021cites this paper
• Etna: Harvesting Action Graphs from Websites

  2021cites this paper
• RePlay: Touchscreen Interaction Substitution Method for Accessible Gaming

  2021cites this paper
• Translation from layout-based to visual android test scripts: An empirical evaluation

  2021cites this paper
• Feature Matching-based Approaches to Improve the Robustness of Android Visual GUI Testing

  2021cites this paper
• Advanced UI test automation (AUTA) for BIOS validation using OpenCV and OCR

  2021cites this paper
• Testing cockpit display systems of aircraft using a model-based approach

  2021influential citation
• Understanding the Challenges and Needs of Programmers Writing Web Automation Scripts

  2021cites this paper
• Recent Developments in Program Synthesis with Evolutionary Algorithms

  2021cites this paper
• Glider: A Reinforcement Learning Approach to Extract UI Scripts from Websites

  2021cites this paper
• WebEvo: taming web application evolution via detecting semantic structure changes

  2021cites this paper
• Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots

  2021cites this paper
• Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

  2021cites this paper
• A Failed attempt at creating Guidelines for Visual GUI Testing: An industrial case study

  2021cites this paper
• Software Integrity and Validation Using Cryptographic Composability and Computer Vision

  2021cites this paper
• Help through demonstration and automation for interactive computing systems: A survey of recent works

  2021cites this paper
• UI Test Migration Across Mobile Platforms

  2021cites this paper
• Live Replay of Screen Videos: Automatically Executing Real Applications as Shown in Recordings

  2020cites this paper