フレクトのクラウドblog re:newal

http://blog.flect.co.jp/cloud/からさらに引っ越しています

Remote monitoring system for operating and monitoring robots based on simple user interface

研究開発室の馮 志聖(マイク)です。

Content

  • Introduction
  • Remote control on ROS(Robot Operating System)
  • Conclusion
  • Future work
  • Other
  • Reference

Introduction

With the rapid development of science and technology and the popularization of the Internet, people are pursuing a better quality of life, and relaxed and happy work has become the first choice. At the same time, many labor-intensive and overtime jobs have gradually appeared in the labor gap. To make up for the labor gap, it is an inevitable trend for robots to replace humans. In our daily life, we often benefit from services provided by robots, such as sweeping robots, restaurant front desk services, home delivery services, etc., Many products we use are also made by robots in factories. To facilitate production, robots have gradually become indispensable.

In recent years, due to the spread of the epidemic and for security reasons, most people choose to work at home. Some software and tools based on remote collaboration have gradually gained attention. Among them, video chat services, which can increase the chances of discussions, especially have grown dramatically. The fluency of meetings allows members to quickly reach a consensus. For this reason, remote work is becoming no longer a barrier, but remote collaboration also has many limitations. Many researchers and developers have discussed how to break through these limitations.

What is the correlation between the above two? For example, in the following two situations, remotely controlling the robot would be a better choice. First, there are some areas in the world that are hazardous to humans or living things. The remote control can allow robots to perform tasks in these dangerous areas. Second, some emergency tasks require the assistance of senior personnel from other countries. Based on the epidemic, time, and distance considerations, remote control robots can be used to perform tasks.

In this blog, we will discuss a fast and effective way to establish a simple interface operation and monitoring robot system, and verify the feasibility of remote execution of tasks.

Remote control on ROS(Robot Operating System)

Goal

This is to demonstrate collaboration remotely with remote robots through a simple user interface. The figure below is a simple conceptual explanation. The left side is the remote operator, and the right side is the robot in a remote warehouse. The operator uses a simple user interface, and the interface monitors the robot and reaches the target location for remote monitoring. For example, the worker needs to remotely access the factory because he cannot go to the warehouse due to coronavirus.

f:id:fengchihsheng:20210708031000p:plain
Diagram Use case

Why choose an easy-to-use user interface?

If you don't use an easy-to-operate interface, you need additional cost to train the operator, and as the version is updated, the operator must be continuously trained, which is bound to be a huge cost. Through the easy-to-operate user interface, the time cost of training operators can be reduced, unnecessary cost waste (like human labor cost) can be avoided, and the staff can be quickly put into work.

Why do remote collaborations with remote robots?

In the above background, it has been mentioned that how to break through the limitations of remote collaboration will be a research and development issue. Under certain severe conditions, I believe that robots can effectively help humans. For example, some areas that cannot be reached by humans, overtime work cannot be carried by humans, and tasks that require a short time to perform long distance tasks, etc.

General Concepts of ROS

What is OS?

To understand ROS, you must first know what OS is. OS is short for operating systems. What is the operating system? Operating system (English: Operating System, abbreviation: OS) is a group of interrelated system software programs that supervise and control computer operations, use and execute hardware, software resources, and provide public services to organize user interaction. It is also a computer system. The core and cornerstone of OS I will not go into details anymore. Those who want to know can learn more about the content and knowledge in this link.

https://en.wikipedia.org/wiki/Operating_system

What is ROS?

ROS (Robot Operating System), as its name implies, is an operating system that is only used for robots. Because the function of the robot is somewhat different from that of an ordinary personal computer, an operating system is required for additional customization of the robot. For detailed content, please refer to the following link:

https://en.wikipedia.org/wiki/Robot_Operating_System

ROS (Robot Operating System) is a BSD(Berkeley Software Distribution)-licensed system for controlling robotic components from a PC. A ROS system consists of many independent nodes, each of which communicates with the other nodes using a publish/subscribe messaging model. For example, a particular sensor’s driver might be implemented as a node, which publishes sensor data in a stream of messages. These messages could be consumed by any number of other nodes, including filters, loggers, and also higher-level systems such as guidance, pathfinding, etc.

BSD licenses are a family of permissive free software licenses, imposing minimal restrictions on the use and distribution of covered software.

https://en.wikipedia.org/wiki/BSD_licenses

Why ROS? Or OS?

Note that nodes in ROS do not have to be on the same system (multiple computers) or even of the same architecture! You could have Arduino publishing messages, a laptop subscribing to them, and an Android phone driving motors. This makes ROS really flexible and adaptable to the needs of the user. ROS is also open source, maintained by many people. If you use an ordinary personal computer operating system, you cannot have these functions, and it will take more time to set up and develop.

https://www.clearpathrobotics.com/assets/guides/kinetic/ros/Intro%20to%20the%20Robot%20Operating%20System.html

One of the most important concepts in ROS is the "node". The use of nodes based on the publish/subscribe model allows developers to develop low-coupling functional modules in parallel and facilitate secondary reuse.

First, we explain the general structure in ROS. The following figure shows the relationship architecture diagram between the main node and multiple sub-nodes in the robot operating system. The sub-nodes can be increased according to demand, and each sub-node can transmit messages to each other.

f:id:fengchihsheng:20210708031512p:plain
The general structure in ROS

Target Robot

We use turtlebot3 as a test robot. I will discuss why we choose turtlebot.

What is turtlebot?

TurtleBot is a low-cost, personal robot kit with open-source software. The TurtleBot kit consists of a mobile base, 2D/3D distance sensor, SBC(Single Board Computer), and the TurtleBot mounting hardware kit. TurtleBot is designed to easily buy, build, and assemble, using off-the-shelf consumer products and parts that can easily be created from standard materials. As an entry-level mobile robotics platform, TurtleBot has many of the same capabilities as the company’s larger robotics platforms, like PR2.

f:id:fengchihsheng:20210727232443p:plain
Turtlebot3 Burger

http://wiki.ros.org/Robots/TurtleBot

Why choose turtlebot3 as the test robot?

As mentioned above, characteristics are very important for the inexperienced and zero-based stage: Low cost, open-source, easy to assemble, with many development documents and information. Because turtlebot can meet the above characteristics, we chose it. We choose this robot for future malleability and plan to extend the application based on this robot.

System Overview

The following content will discuss the structure and process of this attempt.

First, I will start to introduce the cloud service structure. The figure below shows a diagram of the cloud service structure. The remote pc publishes all the data to cloud service by API for the robot side. And subscribe to the robot command data from cloud service. The Client-side website subscribes all data to cloud service by API for the website. And publish the command data to cloud service.

f:id:fengchihsheng:20210730172014p:plain
the cloud service structure

Second, I will start to introduce the network relationship structure. The figure below is a flowchart of the network structure. The left side is the robot and the remote computer. The remote computer is actually the network. The robot console on the local end of the road is responsible for monitoring all the robots on the local end. The upper right is the cloud service, and the remote computer stores and transmits various information through the cloud service. The lower right is the client's web browser, and the browser is a web page set up through the cloud. The server obtains various information from the remote computer and displays it in the user interface.

f:id:fengchihsheng:20210708032104p:plain
the network relationship structure

Third, we explain each node in the robot and the remote computer. The figure below is a diagram of the connection relationship between each node of the robot and each node of the remote computer. The left is the robot side, the name of the function that will be used in the middle, and the remote computer on the right. The green line is from the "publish" of the robot on the left to the "subscription" on the right via the API. Only one blue line is the opposite. At the same time, this figure is also the node and function that will be used in this attempt.

f:id:fengchihsheng:20210708032028p:plain
each node in the robot and the remote computer

Fourth, after introducing the relationship structure between the robot and the remote computer, we explain the connection between the node and each sensor in the robot. The following figure shows the detailed structure diagram of the relationship between the sub-nodes and the sensors. It is also the sensor and node that will be used in this attempt. The following is an explanation of some sensors and chips:

1.OpenCR means Open-source Control Module for ROS. OpenCR is developed for ROS embedded systems to provide completely open-source hardware and software.

http://wiki.ros.org/opencr

2.LiDAR is a method for determining ranges (variable distance) by targeting an object with a laser and measuring the time for the reflected light to return to the receiver.

https://en.wikipedia.org/wiki/Lidar

3.An inertial measurement unit (IMU) is an electronic device that measures and reports a body's specific force, angular rate, and sometimes the orientation of the body, using a combination of accelerometers, gyroscopes, and sometimes magnetometers.

https://en.wikipedia.org/wiki/Inertial_measurement_unit

f:id:fengchihsheng:20210708031927p:plain
the connection between the node and each sensor in the robot

Demo

The figure below is the diagram of the experiment. First of all, we first set the starting point and ending point (the fork part made of white tape). There will be many obstacles between these two points, and the operator will reach the end through a simple user interface operation. The operator can view the robot's camera screen through the user interface and control the robot to the destination at the same time. The goal of this demo is to be able to successful to connect without fail.

f:id:fengchihsheng:20210727235329p:plain
The diagram of the experiment.

In the video, the upper left is the environment of the actual robot, and the right is the user interface for remote control of the robot in different network environments. Through this operation interface, you can obtain the information of each sensor in the robot, and you can also control the movement of the robot. As shown in the structure and process in the diagram in the previous chapter, all operations in the operation interface are updated in real-time, including the map and camera screen.

f:id:fengchihsheng:20210708194531g:plain
Demo video

Conclusion

When we first started the development, because we had no experience in robot development, it was undoubtedly a challenge for us. Everything started from scratch. The first step is to search for all robots on the market and compare and screen them. The second step is to study ROS. Versions are compared and screened in detail, and the third step is to develop your own functions and systems.

Finally, in this attempt, we successfully developed our own execution library, and obtained the information of the sensors in the robot and the remote control of the robot's movement, although a lot of time was spent matching the remote computer and the robot during the development process. The version of the operating system repeated testing and screening and also spent a lot of time studying the system structure and functions of ROS, but this knowledge and experience will bring us growth in the field of robotics research and development.

Future work

After we are able to build the above system, more applications can be extended in the future. For example, the application of relevant elements of artificial intelligence, virtual reality, and augmented reality. In the near future, I believe that robots can help people do more work.

As everyone knows, the field of artificial intelligence has grown rapidly in recent years, especially in the field of image recognition and autonomous driving.

Other

Tell us what do you think about our result or anything else that comes to mind.

We welcome all of your comments and suggestions.

Reference

TurtleBot3 e-Manual
ROS Documentation
ROS 2 Documentation
TurtleBot3 GitHub
Black Dashboard GitHub