A Smart Automatic Agent Operating System Using Screen Analysis And Remote Control

In this century, more automated production has been driven by high demand. As a result, more automated production machines are required to meet market demands. Unfortunately, most machine screen activities still need humans to control manually, and most of the operations are simple and repeated. The human eye must constantly assess the production situation and control the keyboard and mouse to operate machinery to do excessively repetitive tasks. We proposed a smart automated agent operating system called “iScript” to upgrade the old machine to a more automated machine. “iScript” is a software solution that uses image recognition techniques to automate agent operation by automatically controlling the keyboard and mouse to perform agent operations using preset scripts. The agent operation system is also integrated with a modular software design. The essential modules include the Steps Control Module (SCM), Function Plug-in Module (FPM), Variable Module (VM), Condition Control Module (CCM), and Operator Control Module (OCM). The modular design makes the script setting highly expandable. It can set customized scripts for varied purposes and applications dynamically, making production process management more accessible and efficient and accomplishing many tiresome tasks necessary for industrial automation upgrades. The proposed smart automatic agent operating system has been applied to several industrial applications, and the cases of application are shared in this research.

• Publication year

  2022

• Venue

  2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME)

• Publication date

  2022-11-16

• Fields of study

  Not labeled

• Identifiers
  DOI 10.1109/ICECCME55909.2022.9988239
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• RecurBot: Learn to Auto-complete GUI Tasks From Human Demonstrations

  2018cited by this paper
• A Novel Automated GUI Testing Echnology Based on Image Recognition

  2016cited by this paper
• GUI testing using computer vision

  2010cited by this paper
• Sikuli: using GUI screenshots for search and automation

  2009cited by this paper

• No citing papers are available for this paper.