Pic2Translator: A tool to extract mobile GUI components for machine learning and tasks

Mobile apps are becoming more popular in society whether professional, entertainment or simply communication fields since they can provide a collection of useful features in daily activities. One of the essential requirements for an application to be successful in the market is the Graphical User Interface which encompasses most of the user interaction. Thus, the GUI test is an important practice to control the issues that affect the interaction and user experience. However, those tests face some challenges like high-cost maintenance in UI changes. This paper presents a tool that can extract information from mobile GUI interfaces. The information extracted includes the GUI components type, layout hierarchy, and belonging scenario, as well as component position in the screen, component image, and pixel map. Such a pixel map provides the component class of each image pixel to enable semantic segmentation training schemes. Such a dataset may support the research on machine applications associated, for instance, with the identification of changes employing the user interface.

• Publication year

  2021

• Venue

  ICCBD

• Publication date

  2021-11-27

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3507524.3507541
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• GUI-Guided Test Script Repair for Mobile Apps

  2022cited by this paper
• Test Transfer Across Mobile Apps Through Semantic Mapping

  2019cited by this paper
• Automated Cross-Platform GUI Code Generation for Mobile Apps

  2019cited by this paper
• Mobile GUI Testing Fragility: A Study on Open-Source Android Applications

  2019cited by this paper
• StoryDroid: Automated Generation of Storyboard for Android Apps

  2019cited by this paper
• Deep Learning UI Design Patterns of Mobile Apps

  2018cited by this paper
• Change-Based Test Script Maintenance for Android Apps

  2018cited by this paper
• Exploring the integration of user feedback in automated testing of Android applications

  2018cited by this paper
• Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

  2018cited by this paper
• Detecting and Summarizing GUI Changes in Evolving Mobile Apps

  2018cited by this paper
• pix2code: Generating Code from a Graphical User Interface Screenshot

  2017cited by this paper
• Rico: A Mobile App Dataset for Building Data-Driven Design Applications

  2017cited by this paper
• Mining Android App Usages for Generating Actionable GUI-Based Execution Scenarios

  2015cited by this paper
• A Combined Technique of GUI Ripping and Input Perturbation Testing for Android Apps

  2015cited by this paper
• Efficiently, effectively detecting mobile app bugs with AppDoctor

  2014cited by this paper
• API change and fault proneness: a threat to the success of Android apps

  2013cited by this paper
• Experimental assessment of manual versus tool-based maintenance of GUI-directed test scripts

  2009cited by this paper

• No citing papers are available for this paper.