WO2022021739A1

WO2022021739A1 - Humanoid inspection operation method and system for semantic intelligent substation robot

Info

Publication number: WO2022021739A1
Application number: PCT/CN2020/135608
Authority: WO
Inventors: 李建祥; 王振利; 王万国; 慕世友; 王克南; 肖鹏; 周大洲; 黄锐; 郝永鑫; 吕俊涛; 郭锐; 王海鹏
Original assignee: 国网智能科技股份有限公司
Priority date: 2020-07-30
Filing date: 2020-12-11
Publication date: 2022-02-03
Also published as: CN111897332B; CN111897332A

Abstract

A humanoid inspection operation method and system for a semantic intelligent substation robot. The humanoid inspection operation method for a semantic intelligent substation robot comprises: autonomously constructing a three-dimensional semantic map of an unknown substation environment (S101); autonomously planning a walking path of a robot on the basis of the three-dimensional semantic map and in combination with an inspection/operation task and the current position of the robot (S102); controlling the robot to move according to the planned walking path, and carrying out the inspection/operation task during an advancing process (S103); and during the process of carrying out the inspection/operation task, adjusting, in real time, the posture of a mechanical arm (2) that carries an inspection/operation tool (3), so as to automatically collect and recognize, at the optimal angle, an image of a device to be inspected or automatically execute the operation task at the optimal angle, and to complete a fully autonomous inspection/operation task of the substation environment (S104).

Description

A humanoid patrol operation method and system for a semantic intelligent substation robot

technical field

The invention belongs to the field of robots, and in particular relates to a humanoid patrol operation method and system of a semantic intelligent substation robot.

Background technique

The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

Existing inspection robots generally use a docking-preset operation mode, and the deployment and implementation process is divided into two stages: configuration and operation. In the configuration stage, for a new substation with unknown environmental information, a large amount of manual work is required. The inspection point of the inspection robot is usually manually set by the on-site personnel according to the inspection task. When setting, the on-site personnel firstly control the robot to run along the inspection route, and then stop when it runs to the surrounding of the electric equipment to be inspected; then it remotely adjusts the robot. The gimbal posture enables the gimbal to drive the visible light camera, infrared thermal imager and other non-contact detection sensors to sequentially align each device to be inspected around the robot and record the corresponding preset position of the gimbal, thereby completing the setting of a detection point. Repeat the above process to complete the setting of all inspection points of the equipment to be inspected included in the inspection task. In the running phase, after all the inspection points are set, the inspection robot runs along the inspection route, and stops at the inspection points one by one, and then calls the pre-position of the PTZ to complete the inspection of the equipment.

The inventor found that the existing substation robots have the following problems in the process of equipment inspection operations:

(1) When the robot is deployed on-site, the process of setting up inspection points is cumbersome, requiring a lot of manual work, and the on-site configuration of personnel is labor-intensive and inefficient; the setting of inspection and inspection points is more affected by the subjective judgment of the field configuration personnel. The quality of the inspection points cannot be guaranteed due to the inconsistent setting standards. The robot adopts the docking operation method, and each inspection point needs to be stopped for operation. The inspection efficiency is low, and frequent start and stop will cause hidden dangers to the stable operation of the robot.

(2) In the process of robot inspection, traditional robots use a single laser navigation method, which has the problem of sparse laser point cloud leading to navigation failure, and the accuracy of navigation cannot be guaranteed.

(3) In terms of inspection data analysis, the existing substation inspection robots have weak video and image processing capabilities at the front end of the robot. At present, most of the image and video data are analyzed and processed by the network back-end server, which is affected by the transmission network bandwidth. There is a delay in the analysis, which cannot meet the needs of application scenarios with high real-time requirements such as robot navigation, visual servoing, and timely defect detection.

(4) The staff cannot truly understand the substation environment and equipment status in the control room.

SUMMARY OF THE INVENTION

In order to solve the above problems, the present invention provides a human-like inspection operation method and system for a semantic intelligent substation robot, which breaks the "stop-preset" operation mode of the traditional substation inspection robot, and realizes the fully autonomous operation of the robot inspection. .

In order to achieve the above object, the present invention adopts the following technical solutions:

A first aspect of the present invention provides a humanoid patrol operation method for a semantic intelligent substation robot.

A humanoid patrol operation method for a semantic intelligent substation robot, comprising:

Independently build a 3D semantic map of the unknown substation environment;

Based on the 3D semantic map, combined with inspection/job tasks and the current position of the robot, the robot autonomously plans the walking path;

Control the robot to move according to the planned walking path, and carry out inspection/operation tasks during the traveling process;

In the process of carrying out inspection/operation tasks, adjust the pose of the robotic arm equipped with inspection/operation tools in real time, so as to automatically collect and identify images of the equipment to be inspected at the best angle or automatically perform operations at the best angle task, complete the fully autonomous inspection/operation task of the substation environment.

Further, based on the prior knowledge of the substation, the location information of the equipment in the station is automatically obtained, so that the robot can independently construct a three-dimensional semantic map of the substation without configuration-free information injection.

Further, the specific process of constructing the 3D semantic map of the unknown substation environment is as follows:

Real-time acquisition of binocular image data, inspection image data and 3D point cloud data of the current environment;

Based on the binocular image data and 3D point cloud data, the spatial distribution of the current environmental objects is obtained, and the inspection image data is used for real-time analysis to identify the device identification code in the image, locate the target area of the device, and simultaneously obtain the device identification and location in the spatial information. ;

According to the spatial distribution of objects in the current environment, the automatic identification of the unknown area around the robot is realized, the local path planning method is used to realize the robot motion planning in the unknown area, and the map construction of the unknown environment is performed until the whole station environment semantic map is completed. Construct.

Further, the process of performing the map construction of the unknown environment includes:

Obtain the spatial distribution of objects in the current environment based on binocular image data and 3D laser data;

Based on the binocular image data and inspection image data, the semantic information of the road, equipment and obstacle objects in the current environment is obtained, and the spatial position coordinate transformation is used to project the spatial information of the road, equipment and obstacles to the 3D point cloud data to establish a semantic map. .

Further, according to the positional relationship between the robot and the equipment to be inspected, the robot arm is driven to move, so that the end of the robot arm faces the position of the device and moves to the local range of the target device;

Real-time acquisition of inspection camera image data, automatic identification, tracking and positioning of the equipment to be inspected, driving the position of the robotic arm for precise adjustment, so that the image acquisition device at the end of the robotic arm is at the best shooting angle, and driving the image acquisition device to adjust the focal length , compensating for the influence of the robot motion on the image, obtaining the image of the target inspection equipment, and realizing the accurate shooting of the target image;

Based on the acquired fine images of the equipment, target recognition is automatically performed at the front end of the robot, automatic analysis of image data at the front end is realized, and status information of the equipment is obtained in real time.

Further, in the process of moving the end of the robot arm toward the device and moving to the local range of the target device, the robot arm is controlled to adjust the pose and always aim at the device to be inspected, so that the robot is always connected to the device to be inspected during data collection. Maintain the best relative pose relationship;

When the robot reaches the optimal observation pose of the device to be inspected and enters the range of the inspection data acquisition device, the deep learning algorithm is used to identify and obtain the position of the device in the image, and the relative pose relationship between the robot and the device to be inspected is used to determine Realize the spatial pose control of the acquisition device carried at the end of the robotic arm;

The quality of the collected data is evaluated and optimized, so as to realize the optimal collection of the inspection data of the equipment to be inspected.

Further, in the process of quality evaluation and optimization of the collected data, the relationship model of the optimal image collection points for inspection based on historical data changes with time is used to realize the independent optimal selection of inspection points in different seasons and different time periods. .

Further, in the process of collecting data quality evaluation and optimization, the confidence level of the inspection data at different locations and under different lighting conditions is evaluated. Check status data.

Further, a panoramic three-dimensional model of the substation is constructed based on the digital twin method, and the immersive inspection operation of the substation based on virtual reality technology is realized through the real-time reproduction of image, sound and tactile information.

A second aspect of the present invention provides a robot.

A robot, which adopts the above-mentioned semantic intelligent substation robot humanoid patrol operation method for patrol inspection.

A third aspect of the present invention provides a humanoid patrol operation system for a semantic intelligent substation robot.

The present invention provides a humanoid patrol operation system for a semantic intelligent substation robot, which includes at least one robot as described above.

Another semantic intelligent substation robot humanoid patrol operation system provided by the present invention includes:

control center;

at least one robot; the robot is deployed in various areas in the substation;

Each robot includes a robot body, the robot body is provided with a robotic arm, and the end of the robotic arm is equipped with an inspection/work tool;

A computer program is stored in the control center, and when the program is executed by the processor, the steps in the above-mentioned semantic intelligent substation robot humanoid patrol operation method are realized.

A fourth aspect of the present invention provides a computer-readable storage medium.

A computer-readable storage medium on which a computer program is stored, when the program is executed by a processor, implements the steps in the above-mentioned humanoid patrol operation method of a semantic intelligent substation robot.

Compared with the prior art, the beneficial effects of the present invention are:

(1) The present invention creatively invents a human-like patrol operation method of semantic intelligent substation robots, and proposes an autonomous construction method for a three-dimensional semantic map of substations, which realizes active autonomous perception of road and equipment information in an unknown substation environment, and completely gets rid of traditional The robot's reliance on manual stops and preset configuration breaks the traditional "stop-preset" inspection operation mode of the traditional substation inspection robot, and solves the lack of intelligent level of traditional robot inspection and high dependence on manual configuration. The main problem of inefficient docking operations.

(2) The present invention proposes an automatic construction method of a robot semantic map, which realizes the automatic construction of the robot semantic map, adopts the visual laser fusion navigation method, realizes the three-dimensional autonomous navigation of the robot, and solves the ineffectiveness of the traditional robot single navigation method and intelligent perception. The problem of insufficient capacity.

(3) The present invention proposes an AI front-end recognition method for substation inspection video, which uses the deep learning model quantization and cutting method to reduce the computational complexity of the algorithm, improve the real-time performance of the system, and develop a low-power and high-performance substation. The inspection video real-time analysis hardware system reduces the network transmission pressure and improves the real-time performance of data processing.

(4) The present invention proposes an immersive operation method for robots, which combines image, video, sound and other multimodal information, and reconstructs the panoramic information of the robot operating environment through the deep fusion of multi-source and multimodal information, so that the staff can The control room can truly understand the substation environment and equipment conditions, and realize the immersive inspection operation of the substation robot.

Description of drawings

The accompanying drawings forming a part of the present invention are used to provide further understanding of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation of the present invention.

Fig. 1 is the flow chart of the humanoid patrol operation method of the semantic intelligent substation robot according to the embodiment of the present invention;

2 is a schematic diagram of an optimal data collection process for substation inspection equipment according to an embodiment of the present invention;

Fig. 3 is the structure diagram of the humanoid patrol operation system of the substation robot according to the embodiment of the present invention;

4 is a flowchart of autonomous construction of a positioning and navigation map of an inspection robot according to an embodiment of the present invention;

5 is a flowchart of a three-dimensional electronic map semantic analysis according to an embodiment of the present invention;

6 is a frame diagram of a real-time identification frame diagram of a substation inspection video according to an embodiment of the present invention;

7 is a flowchart of real-time identification of substation inspection video according to an embodiment of the present invention;

8 is a schematic structural diagram of a robot according to an embodiment of the present invention;

Figure 9 (a) is a schematic structural diagram of a main arm of an embodiment of the present invention;

FIG. 9(b) is a schematic diagram of the slave arm structure according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of a quick replacement structure according to an embodiment of the present invention.

detailed description

The present invention will be further described below with reference to the accompanying drawings and embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

In the present invention, terms such as "upper", "lower", "left", "right", "front", "rear", "vertical", "horizontal", "side", "bottom", etc. The orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, and is only a relational word determined for the convenience of describing the structural relationship of each component or element of the present invention, and does not specifically refer to any component or element in the present invention, and should not be construed as a reference to the present invention. Invention limitations.

In the present invention, terms such as "fixed connection", "connected", "connected", etc. should be understood in a broad sense, indicating that it can be a fixed connection, an integral connection or a detachable connection; it can be directly connected, or through the middle media are indirectly connected. For the relevant scientific research or technical personnel in the field, the specific meanings of the above terms in the present invention can be determined according to the specific situation, and should not be construed as a limitation of the present invention.

Example 1

Referring to FIG. 1, a humanoid patrol operation method of a semantic intelligent substation robot of the present embodiment is provided, including:

S101: Build a three-dimensional semantic map of the unknown substation environment independently;

S102: Based on the three-dimensional semantic map, combined with the inspection/job task and the current position of the robot, autonomously plan the walking path of the robot;

S103: Control the robot to move according to the planned walking path, and carry out inspection/operation tasks during the traveling process;

S104: In the process of carrying out inspection/operation tasks, adjust the pose of the robotic arm equipped with inspection/operation tools in real time, so as to automatically collect and identify images of the equipment to be inspected at the best angle or automatically at the best angle Perform operation tasks and complete fully autonomous inspection/operation tasks of the substation environment.

In the specific implementation of steps S101 and S102, based on the prior knowledge of the substation, the location information of the equipment in the station is automatically obtained, and the robot can independently construct a three-dimensional semantic map of the substation without configuration-free information injection.

In the specific implementation, the specific process of constructing the three-dimensional semantic map of the unknown substation environment is as follows:

Based on the binocular image data and 3D point cloud data, the spatial distribution of the current environmental objects is obtained. Through real-time analysis of the inspection image data, the identification code of the equipment in the image is identified, and the target area of the equipment is located to realize the simultaneous identification and location of the equipment in the spatial information. Obtain;

According to the spatial distribution of objects in the current environment, the automatic identification of the unknown area around the robot is realized, the local path planning method is used to realize the motion planning of the robot in the unknown area, and the map construction of the unknown environment is performed until the semantic map of the environment in the entire station is completed. Construct.

Among them, the process of performing the map construction of the unknown environment includes:

The three-dimensional semantic map is a pre-stored semantic map, wherein the method for formulating an inspection/operation path includes:

Receive inspection/operation tasks, the inspection/operation tasks include designated inspection/operation areas or designated inspection/operation equipment;

Equipment to be inspected/worked according to inspection/job tasks;

The three-dimensional space projection coordinates of all equipment to be inspected/operated in the semantic map are used as points on the robot's walking route, and the inspection/operation route is planned based on the current location of the robot.

The semantic map includes a three-dimensional map of a substation and semantic information of equipment on the three-dimensional map. The construction method, referring to FIG. 4 , includes:

Obtain prior knowledge data such as substation drawings, electrical design drawings, etc., use knowledge graph and knowledge understanding technology, and form a coarse-precision semantic map based on the prior knowledge data, and automatically construct a task path for the robot to construct the semantic map; The robot motion is controlled by the task path described above. During the motion process, the construction of a roaming semantic map is realized by executing the following steps, as shown in Figure 5:

(1) Obtain binocular images, inspection images and 3D point cloud data of the current environment from binocular vision cameras, inspection cameras and 3D laser sensors;

(2) Identify objects such as roads, equipment, and obstacles in the current environment according to the inspection images; the embedded AI analysis module pre-stores deep learning models for identifying roads, equipment, and various obstacles. Target detection; that is, the semantic information of roads, equipment and obstacles in the current environment is obtained; according to the binocular image and 3D point cloud data, the spatial position distribution of roads, equipment, and obstacles in the current environment is obtained; The image and 3D point cloud data can obtain the distance information of the robot's peripheral equipment or obstacles from the robot body (the binocular image is used to identify close-range obstacles, and the 3D point cloud data is used to identify long-distance obstacles), and then combined with the robot in the inspection task. From the running direction information, the spatial distribution of obstacles centered on the robot body can be obtained.

(3) According to the spatial distribution of objects in the current environment, the automatic identification of the passable unknown area around the robot is realized. If there is an unknown passable area, the local path planning method is used to realize the motion planning of the robot in the unknown area, and to the industrial control of the robot. The robot sends motion commands to make the robot move to the passable unknown area, and then go to step (4); if there is no passable unknown area, it means that all the unknown areas have been explored and the map construction is over;

(4) Build a 3D SLAM map according to the binocular image and the 3D point cloud data, and return to step (1).

In the step (4), the three-dimensional SLAM map is constructed according to the binocular image and the three-dimensional point cloud data, which specifically includes:

Step (4.1): read the binocular image obtained by the binocular camera, the inspection image obtained by the inspection camera and the 3D laser sensor data;

Step (4.2): based on binocular image data and three-dimensional laser data, obtain the spatial position distribution of equipment, equipment and obstacles, and construct a three-dimensional point cloud map based on three-dimensional laser sensor data;

Step (4.3): obtain semantic information of equipment, equipment and obstacles in the current environment based on binocular image data and inspection image data;

Step (4.4): using the spatial position coordinate change, according to the binocular image, and according to the spatial position of the device, the spatial position of the device is projected to the three-dimensional point cloud map, and the mapping from two-dimensional to three-dimensional point cloud map is realized. Combined with step (2) ) Semantic information of roads, equipment and obstacles in the current environment to build a semantic map. By projecting the device identified by the binocular camera to the 3D point cloud map, and then combining the point cloud density distribution of the 3D point cloud map, the 3D position of the device to be inspected in the 3D navigation map and the accurate clustering and semantics of the point cloud can be achieved, and the roaming can be obtained. Semantic map. The roaming semantic map includes the three-dimensional space position of the equipment in the substation and its semantics.

Through the mapping from 2D to 3D point cloud, semantic information such as passable roads, towers, meters, etc. identified by the 2D image can be assigned to the 3D point cloud. Combined with the positioning based on the 2D image, the 3D point can be more accurately identified. Clouds are clustered, making the constructed map closer to reality.

After the robot has established a 3D navigation semantic map, it can use the 3D navigation map and the ROS navigation module to realize the robot's motion navigation in the substation. For the non-stop detection of the inspection equipment specified by the task, the robot adopts a combination of static map and dynamic map: the static map method uses the roaming semantic map to project the three-dimensional space coordinates of the equipment on the walking route, and the space of the equipment to be inspected is projected. The vertical fan-shaped area of the position is used as the task navigation point; the dynamic map method is that the robot obtains the current three-dimensional coordinates of the device after dynamically identifying the device concerned by the task during the movement process, realizes the dynamic identification of the device, and updates the map information in real time.

This embodiment proposes an autonomous construction method for a robot inspection, positioning and navigation map, which realizes the roaming construction of a three-dimensional visual semantic map, and proposes a task-oriented inspection and navigation control method integrating binocular vision and three-dimensional laser. The laser vision fusion navigation planning of the robot is realized, and the navigation failure problem caused by the sparse laser point cloud of the traditional robot is solved.

In the specific implementation of step S104, according to the positional relationship between the robot and the equipment to be inspected, the robot arm is driven to move, so that the end of the robot arm faces the position of the device and moves to the local range of the target device;

Referring to Fig. 2, the robot arm is controlled to adjust the pose and always align with the device to be inspected, so that the robot always maintains the best relative pose relationship with the device to be inspected during data collection;

Among them, in the process of quality evaluation and optimization of the collected data, the relationship model of the optimal image collection points for inspection over time based on historical data is used to realize the autonomous optimal selection of inspection points in different seasons and different time periods.

In the process of collecting data quality evaluation and optimization, the confidence level of the inspection data at different locations and under different lighting conditions is evaluated. During the robot inspection process, the inspection data with the highest confidence is selected as the inspection status data of the equipment to be inspected , to improve the effectiveness of inspection data.

R=0.5*R _position +0.5*R _l

R _position = cos(C _dx )

R _l =1-(LL _x )/L _x ; L>L _x

R _l = 1; L<L _x

where R is the execution degree of the robot's current inspection data, R _position is the position confidence, C _dx is the angle between the current robot end position and the surface normal vector of the device to be detected, cos is the cosine calculation function; R _l is the illumination confidence , Install the light intensity sensor at the end of the robot arm and the inspection camera coaxially to realize the calculation of the current light direction and intensity, L is the current light intensity, L _x is the standard light intensity, and the value is the light intensity under normal light conditions, generally 100000Lux.

In the specific implementation, based on the three-dimensional semantic map, the real-time positions of the equipment to be inspected and the robot in the task inspection are obtained, the robot is controlled to move to the work point based on the inspection path or the operation path, and the end of the robot arm is driven to face the position of the equipment to be inspected. Location.

According to the current position of the robot, the inspection path or working path and the set inspection speed, the relative motion relationship between the robot and the equipment to be inspected is calculated, and the robot arm is controlled to adjust the pose to always align with the equipment to be inspected, so that the The sensor module mounted on the end of the robot arm collects the inspection data of the equipment to be inspected.

According to the three-dimensional semantic map, determine the best inspection pose of the robot for each device to be inspected, and when it arrives at each device to be inspected according to the inspection route, it will perform detection according to the best inspection pose. Among them, the detection according to the optimal inspection pose includes: determining the current actual pose of the robot based on the 3D semantic map and binocular vision and 3D laser sensor data; calculating the relative pose according to the actual pose and the optimal inspection pose Deviation; control the robot to adjust the pose according to the relative pose deviation, and perform detection.

During the inspection process, binocular vision and 3D laser sensor data are obtained in real time to determine whether the layout of the equipment on the walking route is inconsistent with the 3D semantic map, and if so, the 3D semantic map is updated.

Specifically, during the inspection process, the equipment images are also collected in a refined manner, and the process is as follows:

1): During the inspection process, the image data is acquired in real time, and the equipment to be detected in the image is identified.

The substation environment is complex, and the collected images may contain multiple types of equipment at the same time. The deep learning device recognition algorithm library is built here, including mainstream target recognition algorithms such as faster-rcnn, ssd, and yolo. The algorithm library is based on the fully convolutional deep neural network, combined with the equipment information contained in the inspection task, to extract the target detection features and semantic features, and then classify and detect the fused features to realize the accurate identification of the equipment in the inspection images. .

2): Calculate the optimal relative pose relationship between the robot arm and the device to be inspected according to the position of the device in the semantic map in advance; during the inspection process, according to the corresponding relative position relationship, as well as the current position of the robot, inspection route and equipment. The inspection speed is fixed, and the robot arm is controlled to adjust the pose, so that the inspection camera is always aimed at the device to be inspected, so as to collect the image of the device to be inspected from the best angle, perform inspection, and improve the accuracy of device inspection.

This embodiment designs a target detection algorithm (not limited to faster-rcnn algorithm, SSD, yolo, etc.) that combines the characteristics of the spatial position relationship of power equipment, constructs an automatic scheduling method for high-performance computing resources, and proposes a The equipment target detection and tracking method realizes real-time and efficient identification of inspection videos and improves the accuracy of identification of substation equipment.

Specifically, according to the three-dimensional semantic electronic map of the substation and the robot pose calculation, the optimal relative positional relationship between the inspection camera at the end of the robot arm and the equipment to be inspected during data collection is calculated, according to the current position of the robot, inspection route and setting. The inspection speed of the robot arm is calculated at the next moment in the non-stop state, so that the inspection camera at the end of the robot arm and the device to be inspected can maintain the best relative position relationship, that is, to align the device to be inspected. .

Among them, the best relative pose relationship between the robotic arm and the equipment to be inspected is:

max[|n _x (xx _r )+n _y (yy _r )+n _z (zz _r )|+|n _x *n _xr +n _y *n _yr +n _z *n _zr |]

In the formula: n _x , n _y , n _z are the normal vectors of the inspection surface of the device to be inspected (such as the dial surface of the marked reading), x, y, z are the spatial coordinates of the device to be inspected, and x _r , y _r , z _r and n _xr , n _yr , n _zr are the robot space pose vectors. When the robot's running pose makes the above formula get the maximum value, the best relative pose between the robot and the device to be detected can be obtained.

The spatial pose of the end of the robotic arm is:

max[|n _x *n _xa +n _y *n _ya +n _z *n _za |]

In the formula: n _x , n _y , n _z are the normal vectors of the detection surface of the equipment to be inspected (such as the dial surface of the marked reading), n _xa , n _ya , and n _za are the spatial attitude vectors of the robotic arm. Detect the best data collection attitude of the equipment, and control the manipulator so that the above formula can achieve the maximum value.

During the attitude adjustment process of the robot arm, the distance information between the device to be detected and the end of the robot arm is used to automatically calculate the configuration focal length of the inspection camera to ensure that the information of the device to be detected is clearly visible in the image.

At the same time, the image data collected by the binocular vision camera is acquired in real time, the equipment to be detected in the image is identified based on the deep learning method, and the posture of the robotic arm is fine-tuned to ensure that the area of the equipment to be detected is always in the central area of the image.

In a specific implementation, a deep learning algorithm is used to perform device identification on each frame of images in the inspection video, and when a target device is identified, a binocular stereo algorithm is used to obtain the three-dimensional space position coordinates of the target device. This paper proposes a local self-adjustment method for inspection camera pose, using DeblurGAN motion video deblurring algorithm.

A motion compensation algorithm for robot image acquisition is proposed. Robot motion compensation is used to improve the stability of inspection image acquisition during motion and ensure the validity of inspection images. Since the robot needs to keep the device to be detected in the central area of the image during the process of traveling, to achieve accurate acquisition of the device to be detected, the robot motion needs to be compensated for this. This embodiment proposes a motion compensation algorithm for the robot to capture images. The formula as follows:

Control_x=Kpx*delta_x+Vx*Kbx*D

Control_y=Kpy*delta_y+Vy*Kby*D

Among them: Control_x and Control_y are the control adjustments of the robot end posture in the X and Y directions, delta_x and delta_y are the coordinate deviations in the X and Y directions between the center of the device area and the center of the image in the image captured by the robot at a certain moment, and Kpx and Kpy are The proportional coefficient of the control adjustment amount of the robot end posture in the X and Y directions, Vx and Vy are the movement speeds of the robot end in the X and Y directions respectively, Kbx and Kby are the control amount compensation coefficients of the robot end posture in the X and Y directions, D is the distance between the end of the robot and the device to be detected. The non-stop inspection robot of this embodiment can be used on a substation inspection robot, and can also be used in operations.

When the posture of the robot arm and the focal length of the robot inspection camera are adjusted in place, the refined acquisition of the device image is completed.

In the specific implementation, in the process of equipment identification, the equipment area is calibrated in a small number of images of the object to be identified collected by the inspection; The picture is transformed to simulate the situation of shooting the device from different angles and different distances; the background picture is updated to obtain images of the equipment to be inspected in different backgrounds, thereby generating a large number of calibrated pictures.

Through the above method, the expansion of various sample image data and sample annotation files is realized, and the image data of the sample is enriched, which is convenient for the subsequent realization of image training by using artificial intelligence deep learning algorithm, so as to realize the recognition of image equipment status more accurately.

In order to better illustrate the above technical solutions, the following takes an instrument in a substation as an example to illustrate the generation process from a few samples to a large number of samples:

First, use the inspection robot to collect several images of the front, side and back of a certain instrument equipment in the substation.

Preprocess a small number of images of objects to be identified collected by power inspection to enhance the quality of the images. A small number of images are preprocessed, including image preprocessing such as deblurring and debounce.

A small amount of collected images are calibrated to realize the calibration of the device area in the image. Calibration is performed in this step, and the number of calibrations to be performed is small.

The calibrated image is processed to remove the background to obtain a real picture with a transparent background. In this step, background-removing processing is performed to obtain a physical image with a transparent background, so that the background can be replaced later to realize physical images under different backgrounds.

Transform the real picture after the background is removed, specifically: randomly scaling, rotating, and radiating the image with the transparent background. Simulate shooting the device from different angles and distances.

For a small number of images obtained, because some parts are inconvenient to collect or cannot be collected, the image of the corresponding angle cannot be obtained. Through the transformation processing of the above steps, a relatively comprehensive physical image can be obtained, which is more helpful to show the real object. Structural state.

After transforming the real picture after the background is removed, it also includes:

Import the image into the Blender software, add different lighting rendering to the image, simulate the situation under different lighting conditions, and obtain image data under different lighting conditions.

Since a small number of images obtained are images of a certain illumination at a certain time, they cannot meet the needs of images under different lighting conditions. Therefore, performing the above operations to obtain image data under different lighting conditions can make the types of images meet the training requirements. time requirements.

Update the texture background or background environment, and obtain the images of the object to be recognized in different texture backgrounds or background environments, thereby generating a large number of calibrated pictures, realizing the expansion of various sample image data and sample annotation files, and enriching the sample image data.

After generating a large number of calibrated pictures, there are large and small samples in the obtained massive images, and the number of samples is unbalanced. Therefore, the SMOTE (Synthetic Minority Over-sampling Technique) method is used to solve the problem of unbalanced samples, thereby further improving the performance of the classifier.

In a specific embodiment, when the multi-sample data is enhanced, it is specifically:

Define the feature space, correspond each sample to a certain point in the feature space, and determine the sampling ratio according to the sample imbalance ratio;

For each small sample class sample, find the nearest neighbor sample according to the Euclidean distance, randomly select a sample point from it, and randomly select a point on the line segment connecting the sample point and the nearest neighbor sample point in the feature space as a new sample point. until the large and small sample sizes are balanced.

Class imbalance is common and refers to the fact that the number of classes in a dataset is not approximately equal. If the sample categories are very different, it will affect the classification effect of the classifier. Assuming that the number of small sample data is very small, such as only 1% of the population, even if all small samples are mistakenly identified as large samples, the accuracy of classifier recognition under the empirical risk minimization strategy can still reach 99%, but due to Without learning the features of small samples, the actual classification effect will be very poor.

The SMOTE method is an interpolation-based method, which can synthesize new samples for small sample classes. The main process is as follows:

The first step is to define the feature space, correspond each sample to a certain point in the feature space, and determine a sampling ratio N according to the sample imbalance ratio;

In the second step, for each small sample class sample (x, y), find the K nearest neighbor samples according to the Euclidean distance, and randomly select a sample point from them, assuming that the selected nearest neighbor point is (x _n , y _n ). A point is randomly selected as a new sample point on the line segment connecting the sample point and the nearest neighbor sample point in the feature space, which satisfies the following formula:

(x _new , y _new )=(x, y)+rand(0-1)*((x _n -x), (y _n -y))

The third step is to repeat the above steps until the number of large and small samples is balanced.

In a specific embodiment, the background image is a background image captured in reality or a background image in an open source texture library, and the two images are in a certain ratio so that the training image takes into account both virtual and real data.

Background images captured in random reality, collected through fixation, and reused.

Some open source texture libraries on the Internet are collected and reused through fixed collections.

The selection of the combination ratio of the two textures (50%, 50%, realizes the effective fusion of virtual and real background images, makes the training images take into account both virtual and real data, and better improves the recognition accuracy of the training model).

This embodiment proposes an autonomous analysis method for robot inspection image data, designs an automatic identification algorithm for substation equipment based on few-sample images, realizes automatic analysis and screening of inspection equipment status information, and improves inspection image data. Analysis quality. The method for enhancing image data with few samples for power inspection in this embodiment can be applied to common inspection robots, UAV inspections, etc., and processes the collected images to obtain a large number of calibrated images.

Specifically, based on the position data of the equipment to be inspected, the coordinates of the robotic arm are adjusted so that the equipment to be inspected is located in the center of the image, so as to realize real-time adjustment of the state of the equipment to be inspected.

After identifying the location of the equipment to be inspected, the location of the equipment to be inspected is also tracked, and the real-time location information of the equipment to be inspected is sent to the robotic arm control module.

This embodiment also performs real-time identification of substation inspection videos based on AI front-end, as shown in Figure 7, and the process includes:

A) Sample and model construction steps: collect the image data of the equipment and various states of the equipment in the station, mark it, form the image sample library of the substation equipment, and use the deep learning target detection algorithm to train the sample images to form the substation equipment model. And the status recognition model of substation equipment, the substation equipment model is used for the identification and positioning of the equipment in the inspection video; the substation equipment status recognition model is used for the identification of the equipment status in the inspection video;

B) Identification model initialization step: AI analysis module loads substation equipment model and substation equipment state identification model; wherein, as shown in Figure 6, the components involved in the real-time identification process of substation inspection video include at least one fixed-point camera , at least one robot camera and AI analysis module;

The robot camera is installed on the substation inspection robot to collect equipment and environmental video information in the substation inspection robot's inspection route coverage area; fixed-point cameras are distributed in the substation equipment area to collect robot inspection in the substation equipment area. Equipment and environmental video information in the area; the AI analysis module processes the substation inspection video collected by fixed-point cameras and robot cameras in real time, identifies and outputs equipment location information, and analyzes and processes equipment image information in the collected video. Real-time tracking of equipment status.

C) Device identification step: The AI analysis module starts the device identification service function, detects the device target of the fixed-point surveillance camera and the robot inspection video, realizes the real-time identification and positioning of the device to be detected in the video, and outputs the inspection image of the target device. The detection frame in , including the center position of the target device and the length and width of the device area;

D) Device target tracking step: After the AI analysis module realizes the identification of the target device, in order to ensure the real-time and accuracy of target acquisition, the target device is tracked, and the KCF method is used to track the target device. Because the target tracking algorithm has serious prospects The problem of tracking target loss under changing circumstances,

(X _t , Y _t , W _t , H _t )=KCF(R(t-Floor((td _t )/d _t )*d _t ))

(X _t , Y _t , W _t , H _t ) is the coordinate output tracked by the KCF algorithm at time t, R(t) is the coordinate of the target device output by the target detection algorithm at time t, Floor is the rounding function; every d _t The time interval is calculated for the target detection and recognition algorithm, and it is used as the input coordinates of the KCF algorithm. The target detection algorithm is used to regularly update the input coordinates of the KCF algorithm to eliminate the problem of false tracking, improve the accuracy of target tracking, and improve the real-time performance of the algorithm.

E) Step of fine image acquisition: During the target tracking process, by sending the real-time position information of the device to the robotic arm control module, adjusting the coordinates of the end of the robotic arm so that the device is located in the center of the image, and adjusting the focal length of the camera to obtain the image capture device Detailed image information.

F) Equipment status identification step: The AI analysis module starts the substation equipment status identification service, realizes the intelligent analysis of the detailed image of the equipment, completes the real-time acquisition of the identification status, and sends back the substation inspection video background.

The target recognition algorithm uses the YOLOV3 algorithm, and the target tracking algorithm uses the KCF target tracking algorithm.

The target tracking algorithm builds a device target detection framework that interacts with key frame target detection and non-key frame target tracking, and uses deep learning model quantization and clipping technology to reduce the computational complexity of the algorithm and improve the real-time performance of the system.

The AI analysis module adopts the automatic scheduling method of high-performance computing resources to realize the analysis function of multi-channel video of substation robots and fixed point inspection.

When there are multiple channels of video analysis and processing, to ensure the real-time analysis, an automatic scheduling method of high-performance computing resources is required. This aspect is described as follows:

(1) Dynamically monitor the current number of videos to be identified;

(2) Check the current usage of graphics card resources;

(3) When an idle graphics card is found, assign the identification task to the idle graphics card;

(4) When it is found that there is no free graphics card, start the rotation analysis mode (multi-channel video frames alternately use graphics card resources), and realize the processing of multi-channel video alternately to improve the real-time and effectiveness of video analysis.

In other embodiments, a panoramic three-dimensional model of the substation is constructed based on the digital twin method, and the immersive inspection operation of the substation based on the virtual reality technology is realized through the real-time reproduction of image, sound, and tactile information.

For example, the virtual reality module on the robot body can be used to construct the virtual environment of the substation operation site. The VR virtual reality module includes a VR camera, which can collect the on-site environment and build a virtual environment on the job site. Through this module, the operation and maintenance personnel can remotely perceive the on-site working environment virtually, so as to realize the precise operation of the on-site equipment. Under normal circumstances, the robot can conduct autonomous inspection; when the robot finds equipment defects or problems, it will send the information to the operation and maintenance personnel in time, and give the corresponding problem category and corresponding solution for the operation and maintenance personnel to refer to.

This embodiment proposes a panoramic immersive inspection and operation method of a robot, which can combine multi-modal information such as images, videos, and sounds to construct panoramic 3D information of substations based on digital twin technology (which can be 3D images, 3D lasers, virtual models, etc. ), proposes an immersive robot operation method (which can be heterogeneous or isomorphic, master-slave or automated), and reconstructs robot operations through the deep fusion of multi-source and multi-modal information The panoramic environment information enables the staff to truly understand the substation environment and equipment conditions in the control room, and realizes the immersive inspection operation of the substation robot.

Embodiment 2

This embodiment provides a robot, which adopts the humanoid patrol operation method of a semantic intelligent substation robot as described in Embodiment 1 to perform patrol inspection.

As shown in FIG. 8 , the robot includes a robot body 1 with a multi-degree-of-freedom mechanical arm 2 , and the end of the multi-degree-of-freedom mechanical arm is equipped with an inspection device 3 .

Specifically, the inspection equipment carried at the end of the multi-degree-of-freedom robotic arm includes: visible light camera, infrared camera, hand grasp, suction cup, partial discharge detector, etc.

Referring to Figure 9(a) and Figure 9(b), the multi-degree-of-freedom manipulator arm on the robot body is used as the slave arm 4, and a control master arm 5 is additionally set. The master arm is a portable operating system suitable for human operation. After wearing the main arm 5, the centralized control operation and maintenance personnel can remotely control the slave arm 4 through 5G communication from the centralized control room to realize the inspection operation of the equipment in the substation.

In addition, a VR virtual reality module 3 is also set on the robot body, which is used to construct a virtual environment of the substation operation site. The VR virtual reality module 3 includes a VR camera, which can collect the on-site environment and build a virtual environment on the job site. Through this module, the operation and maintenance personnel can remotely perceive the on-site working environment virtually, so as to realize the precise operation of the on-site equipment.

Using the structure of this embodiment, under normal circumstances, the robot can conduct autonomous inspection; when the robot finds equipment defects or problems, it will send the information to the operation and maintenance personnel in time, and give the corresponding problem category and corresponding solution. For the reference of operation and maintenance personnel.

If the discovered equipment problems can be solved through remote operation, the operation and maintenance personnel can issue online operation orders, and the robot will automatically switch to the remote operation mode after receiving the order.

During operation maintenance, the robot will come to the equipment that needs to be repaired by itself, turn on the robot VR virtual reality module, and remotely construct the on-site virtual environment in the centralized control center through the 5G communication module;

Through the main arm of the centralized control center, the operation and maintenance personnel use the 5G communication module to remotely control the slave arm on the substation on-site robot; through the VR virtual reality, the operation and maintenance personnel can perceive the environment of the equipment to be repaired in the substation in real time, and use the slave arm to carry out refined operations. The remote maintenance of the substation equipment by the operation and maintenance personnel is realized, the timeliness of the maintenance of the substation is improved, and the personal safety of the operation and maintenance personnel is ensured.

As an optional implementation manner, the front end of the robot body is provided with an AI front-end data processing module, and the AI front-end data processing module is configured to realize front-end recognition of images of substation inspection equipment. The process of target recognition based on images is carried out at the front end of the robot, which avoids the problem of untimely video analysis due to the time delay of data transmission in the process of returning massive data to the background; at the same time, it reduces the bandwidth requirements.

In addition, since the current substation inspection robots mainly focus on visible light and infrared temperature measurement, most of them fix the visible light camera and the infrared camera on both sides of the gimbal, and then directly fix the gimbal directly on the gimbal with screws or bolts. On the robot body, due to factors such as appearance and IP protection, it is difficult to replace the gimbal fixed on the robot body, so it is impossible to quickly replace the detection equipment.

In this embodiment, referring to FIG. 10 , a quick-change joint is provided at the end of the multi-degree-of-freedom manipulator, which can achieve a breakthrough in that a single robot can complete various detection operations in the substation, and solves the problem that the detection function of the traditional inspection robot is single and cannot be arbitrarily changed. equipment problem.

Specifically, the connection sleeve 7 is used to realize quick connection and replacement; threaded connectors are provided at the end 8 of the multi-degree-of-freedom manipulator arm and the end of the detection device 6. First, the end of the manipulator 8 and the detection device 6 are aligned, and then rotated. The prefabricated connecting sleeve 7 at the end of the manipulator uses the threaded threads shared by the end of the manipulator and the detection equipment, so that the connecting sleeve gradually fixes the manipulator and the terminal equipment together.

In this embodiment, the equipment that can be converted includes four modules: visible light camera, infrared temperature measurement, partial discharge detection, manipulator grasping, and electric suction head.

Embodiment 3

This embodiment provides a humanoid patrol operation system of a semantic intelligent substation robot, which includes at least one robot according to the second embodiment.

The humanoid inspection operation system of the semantic intelligent substation robot in this embodiment includes: an embedded AI analysis module, a multi-degree-of-freedom robotic arm, an inspection camera, a binocular vision camera, and a three-dimensional laser radar connected to the embedded AI analysis module , Inertial Navigation Sensor, Robot IPC, and Robot Arm; wherein, the binocular vision camera is located at the front end of the robot, and the inspection camera is located at the end of the robotic arm through the robotic arm. The robotic robotic IPC is connected to the robot motion platform, which can realize multiple Vision, laser, GPS, inertial navigation and other sensor data access and synchronous acquisition, so as to realize the panoramic perception of the robot itself and the surrounding environment, as shown in Figure 3. Among them, the binocular vision camera is used to construct a semantic map; the inspection data acquisition camera is used to collect fine images of the equipment to perform detection.

All devices are connected with network switches through the network to form a robot ROS control network. Among them, the embedded AI analysis module is the key node of system data analysis and processing. As the operation node of ROS-Core, it is responsible for the information collection of each sensor of the robot, the realization of the ROS interface driven by the robot chassis, the three-dimensional information analysis and fusion of laser/vision, the robot navigation control, and control of the robotic arm. The system adopts the storage ROS interface, and the laser, vision, and driver all use the standard ROS interface. The system design mainly includes 11 node function packages, and the security functions are classified into roaming semantic map building module, inspection and navigation control module, and equipment image refinement. Acquisition module and device status identification module.

The walkthrough semantic map building block is configured to:

The roaming semantic map includes a three-dimensional map of a substation and semantic information of devices on the three-dimensional map, and the construction method includes:

Obtain prior knowledge data such as substation drawings, electrical design drawings, etc., use knowledge graph and knowledge understanding technology, and form a coarse-precision semantic map based on the prior knowledge data, and automatically construct a task path for the robot to construct the semantic map; The above task path controls the robot movement. During the movement process, the construction of the roaming semantic map is realized by executing the following steps:

(2) Identify objects such as roads, equipment, and obstacles in the current environment based on inspection images; the embedded AI analysis module pre-stores deep learning models for identifying roads, equipment, and various obstacles. Based on these models Perform target detection; that is, obtain the semantic information of roads, equipment, and obstacles in the current environment; obtain the spatial position distribution of roads, equipment, and obstacles in the current environment according to the binocular image and 3D point cloud data; Specifically, Binocular image and 3D point cloud data can obtain the distance information of robot peripheral equipment or obstacles from the robot body (binocular image is used to identify short-range obstacles, and 3D point cloud data is used to identify long-distance obstacles), and then combined with inspection tasks The spatial distribution of obstacles centered on the robot body can be obtained by using the information of the robot's running direction.

(3) According to the spatial distribution of objects in the current environment, the automatic identification of the passable unknown area around the robot is realized. If there is an unknown passable area, the local path planning method is used to realize the motion planning of the robot in the unknown area, and report to the robot industrial computer. Send a motion command to make the robot move to a passable unknown area, and go to step (4); if there is no passable unknown area, it means that all unknown areas have been explored and the map construction is over;

Step (4.3): obtain semantic information of objects such as equipment, equipment and obstacles in the current environment based on the binocular image data and the inspection image data;

Step (4.4): using the spatial position coordinate change, according to the binocular image, and according to the spatial position of the device, the spatial position of the device is projected to the three-dimensional point cloud map, and the mapping from two-dimensional to three-dimensional point cloud map is realized. Combined with step (2) ) Semantic information of roads, equipment and obstacles in the current environment to build a semantic map. By projecting the equipment identified according to the binocular camera to the 3D point cloud map, and then combining the point cloud density distribution of the 3D point cloud map, the accurate clustering and semantics of the 3D position of the device to be inspected and the point cloud in the 3D navigation map can be achieved, and we get Roaming Semantic Maps. The roaming semantic map includes the three-dimensional space position of the equipment in the substation and its semantics.

The inspection and navigation control module is configured as:

Step 1: Receive an inspection task, where the inspection task includes a designated inspection area or designated inspection equipment;

Step 2: Determine the detectable area information of the equipment to be inspected corresponding to the inspection task according to the semantic map;

Step 3: Integrate the detectable area information of all the devices to be detected in the robot's current inspection task, combine the robot's current location, and plan the inspection route based on the inspection road information in the semantic map; The three-dimensional space projection coordinates of the equipment to be inspected are used as points on the robot's walking route, and the inspection route is planned based on the current location of the robot;

Further, according to the roaming semantic map, the optimal inspection pose of the robot for each device to be inspected is determined, and when reaching each device to be inspected according to the inspection route, the detection is performed according to the optimal inspection pose;

Step 4: Perform the inspection according to the inspection route, and if the optimal inspection pose is obtained, the inspection is performed according to the optimal inspection pose.

During the inspection process, the binocular vision and 3D laser sensor data are obtained in real time, and it is judged whether the layout of the equipment on the walking route is inconsistent with the roaming semantic map. If there is, the roaming semantic map is updated.

The device image refinement acquisition module is configured as:

Step 1: During the inspection process, image data is acquired in real time, and the equipment to be detected in the image is identified.

The substation environment is complex, and the collected images may contain multiple types of equipment at the same time. A deep learning device recognition algorithm library is built here, including mainstream target recognition algorithms such as faster-rcnn, ssd, and yolo. The algorithm library is based on the fully convolutional deep neural network, combined with the equipment information contained in the inspection task, to extract the target detection features and semantic features, and then classify and detect the fused features to realize the accurate identification of the equipment in the inspection images. .

Step 2: Calculate the optimal relative pose relationship between the robotic arm and the device to be inspected according to the position of the device in the semantic map in advance; during the inspection process, according to the corresponding relative position relationship, as well as the current position of the robot, inspection route and settings It controls the robot arm to adjust the pose, so that the inspection camera is always aimed at the device to be inspected, so as to collect the image of the device to be inspected from the best angle, perform inspection, and improve the accuracy of device inspection.

Specifically, in the step 2, according to the three-dimensional semantic electronic map of the substation and the robot pose calculation, the optimal relative positional relationship between the inspection camera at the end of the robot arm and the equipment to be inspected during data collection is calculated. The inspection route and the set inspection speed are used to calculate the robot arm pose control parameters at the next moment in the non-stop state, so that the inspection camera at the end of the robot arm and the device to be inspected can maintain the best relative position relationship, that is, Aim at the device to be tested.

Specifically, the optimal relative pose relationship between the robotic arm and the device to be inspected is:

The spatial pose of the end of the robotic arm is:

max[|n _x *n _xa +n _y *n _ya +n _z *n _za |]

The embedded AI analysis module further includes a device state identification module configured to:

After the robot completes the refined snapshot of the equipment to be tested, it uses the front-end technology of the deep learning algorithm, and relies on the front-end computing power provided by the embedded AI analysis module to realize the front-end real-time analysis of the equipment status, and discover the equipment to be tested in time. The operation defects of the equipment are improved, and the operation safety of the equipment is improved.

In a specific implementation, the robotic industrial computer is further configured to perform the following steps:

Based on the digital twin method, a panoramic three-dimensional model of the substation is constructed to realize the immersive inspection of the substation based on the virtual reality technology.

In other embodiments, the robotic industrial computer is further configured to perform the following steps:

Use the deep learning algorithm to identify the equipment in each frame of the inspection video. When the equipment to be inspected is identified, the binocular stereo algorithm is used to obtain the three-dimensional space position coordinates of the equipment to be inspected;

In the process of equipment identification, the equipment area is calibrated in a small number of images of the object to be identified collected by the inspection; Shoot the device from different angles and distances; update the background image to obtain images of the device to be inspected in different backgrounds, thereby generating a large number of calibrated images;

Before calibration, a small number of images of the object to be recognized collected by the power inspection are preprocessed to enhance the quality of the images.

Add different lighting rendering to the image, simulate the situation under different lighting conditions, and obtain image data under different lighting conditions.

Control the robot arm to adjust the pose and always aim at the device to be inspected, so that the robot always maintains the best relative pose relationship with the device to be inspected during data collection;

When the robot reaches the optimal observation pose of the equipment to be inspected and enters the range of the inspection data acquisition device, the deep learning algorithm is used to identify and obtain the position of the equipment in the image, and the relative pose relationship between the robot and the equipment to be inspected is used to obtain Realize the spatial pose control of the acquisition device carried at the end of the robotic arm;

In the process of quality evaluation and optimization of the collected data, the relationship model of the optimal image collection points for inspection over time based on historical data is used to realize the autonomous optimal selection of inspection points in different seasons and different time periods.

In other embodiments, confidence evaluation is performed on inspection data at different locations and under different lighting conditions, and during the robot inspection process, the detection data with the highest confidence is selected as the inspection status data of the device to be inspected.

Embodiment 4

This embodiment provides a semantic intelligent substation robot humanoid patrol operation system, including:

control center;

at least one robot; the robot is deployed in various areas in the substation;

A computer program is stored in the control center, and when the program is executed by the processor, the steps in the humanoid patrol operation method of the semantic intelligent substation robot described in the first embodiment are implemented.

Embodiment 5

This embodiment provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the steps in the humanoid patrol operation method for a semantic intelligent substation robot described in the first embodiment.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

A humanoid patrol operation method for a semantic intelligent substation robot, characterized by comprising:

Build a 3D semantic map of the unknown substation environment;

Based on the 3D semantic map, combined with inspection/job tasks and the current position of the robot, plan the walking path of the robot;

Control the robot to move according to the planned walking path, and carry out inspection/operation tasks during the traveling process;

In the process of carrying out inspection/operation tasks, adjust the pose of the robot arm equipped with inspection/operation tools in real time, so as to automatically collect and identify the images of the equipment to be inspected at the best angle or automatically execute at the best angle. Operation tasks, complete the fully autonomous inspection/operation tasks of the substation environment.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 1, wherein the constructing a three-dimensional semantic map of the unknown substation environment comprises:

The panoramic three-dimensional model of the substation is constructed based on the digital twin method, and the immersive inspection operation of the substation based on virtual reality technology is realized through the real-time reproduction method of image, sound and tactile information.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 1, wherein the constructing a three-dimensional semantic map of the unknown substation environment comprises:

Based on the prior knowledge of the substation, the location information of the equipment in the station is automatically obtained, and the 3D semantic map of the unknown substation environment can be constructed without the configuration-free information injection of the robot.
The semantic intelligent substation robot humanoid patrol operation method according to claim 3, wherein the specific process of constructing the three-dimensional semantic map of the unknown substation environment is:

Real-time acquisition of binocular image data, inspection image data and three-dimensional point cloud data of the current environment; the robot is provided with a binocular vision camera, an inspection camera and a three-dimensional laser sensor, and the binocular vision camera captures a binocular image The inspection camera captures the inspection image data, and the three-dimensional laser sensor detects the three-dimensional point cloud data and the three-dimensional laser data;

Based on the binocular image data and 3D point cloud data, the spatial distribution of the current environmental objects is obtained. Through real-time analysis of the inspection image data, the identification code of the equipment in the image is identified, and the target area of the equipment is located to realize the simultaneous identification and location of the equipment in the spatial information. Obtain;

According to the spatial distribution of objects in the current environment, the automatic identification of the unknown area around the robot is realized, the local path planning method is used to realize the robot motion planning in the unknown area, and the map construction of the unknown environment is performed until the whole station environment semantic map is completed. Construct.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 4, wherein the process of executing the map construction of the unknown environment comprises:

Obtain the spatial distribution of objects in the current environment based on binocular image data and 3D laser data;

Based on the binocular image data and inspection image data, the semantic information of the road, equipment and obstacle objects in the current environment is obtained, and the spatial position coordinate transformation is used to project the spatial information of the road, equipment and obstacle to the 3D point cloud data to establish a semantic map. .
The humanoid inspection operation method of a semantic intelligent substation robot according to claim 4, wherein the inspection/operation tool comprises an image acquisition device;

In the process of carrying out inspection/operation tasks, the real-time adjustment of the pose of the robot arm equipped with inspection/operation tools includes:

According to the positional relationship between the robot and the equipment to be inspected, drive the robot arm to move, so that the end of the robot arm faces the position of the equipment and moves to the local range of the target equipment;

Real-time acquisition of the image data captured by the inspection camera, automatic identification, tracking and positioning of the position of the equipment to be inspected, and precise adjustment of the position of the robotic arm, so that the image acquisition device at the end of the robotic arm is at the best shooting angle and drives the image acquisition The equipment adjusts the focal length, obtains the image of the target inspection equipment, and realizes the accurate shooting of the target image;

Based on the obtained equipment target image, target recognition is automatically performed at the front end of the robot, which realizes the automatic analysis of image data at the front end, and obtains the status information of the equipment in real time.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 4, characterized in that, in the process of performing patrol inspection/operation tasks, the pose of the robot arm equipped with inspection/operation tools is adjusted in real time ,include:

Calculate the relative motion relationship between the robot and the equipment to be inspected according to the current position of the robot, the inspection/work path and the set inspection speed, and control the robot arm to adjust the pose to always align with the equipment to be inspected, so that the robot can The sensor mounted on the end of the robotic arm collects the inspection data of the equipment to be inspected.
The human-like patrol operation method of a semantic intelligent substation robot according to claim 4, wherein the planning of the robot walking path based on the three-dimensional semantic map, combined with the patrol inspection/operation task and the current position of the robot, includes:

According to the three-dimensional semantic map, determine the optimal inspection pose of the robot for each device to be inspected, and when it arrives at each device to be inspected according to the inspection route, it will perform detection according to the optimal inspection pose.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 8, wherein the detecting according to the best patrol posture comprises: based on a three-dimensional semantic map and the binocular vision and binocular vision of the binocular vision camera. The three-dimensional laser sensor data is used to determine the current actual pose of the robot; the relative pose deviation is calculated according to the actual pose and the optimal inspection pose; the robot is controlled to adjust the pose according to the relative pose deviation to perform detection.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 4, characterized in that, in the process of patrol inspection, the binocular vision and three-dimensional laser data of the binocular vision camera are acquired in real time, and it is judged whether there is equipment on the walking path. The layout of the 3D semantic map is inconsistent with the 3D semantic map. If it exists, the 3D semantic map is updated.
The human-like inspection operation method of a semantic intelligent substation robot according to claim 1, wherein the robot is provided with an inspection camera, and the inspection camera captures an inspection video, and each frame in the inspection video The image is the inspection image data;

The deep learning algorithm is used to identify the equipment in each frame of the inspection video. When the equipment to be inspected is identified, the binocular stereo algorithm is used to obtain the three-dimensional space position coordinates of the equipment to be inspected.
The human-like patrol operation method of a semantic intelligent substation robot according to claim 11, characterized in that, in the process of equipment identification, the equipment area in a small number of images of the object to be identified collected by the inspection is calibrated; The image is subjected to background removal processing, and the background image of the equipment to be inspected is transformed to simulate the situation of shooting the equipment from different angles and distances; the background image is updated to obtain images of the equipment to be inspected in different backgrounds, thereby Generate a large number of calibrated pictures.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 12, characterized in that, before the calibration, a small number of images of the objects to be recognized collected by the power patrol are preprocessed to enhance the quality of the images.
The humanoid patrol operation method of a semantic intelligent substation robot as claimed in claim 12, characterized in that, after transforming the real picture after the background is removed, the method further comprises:

Add different lighting rendering to the image, simulate the situation under different lighting conditions, and obtain image data under different lighting conditions.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 12, wherein after generating a large number of calibrated pictures, multi-sample data enhancement is performed, specifically:

Define the feature space, correspond each sample to a certain point in the feature space, and determine the sampling ratio according to the sample imbalance ratio; each of the samples is each of the calibrated pictures;

For each small sample class sample, find the nearest neighbor sample according to the Euclidean distance, randomly select a sample point from it, and randomly select a point on the line segment connecting the sample point and the nearest neighbor sample point in the feature space as a new sample point. until the large and small sample sizes are balanced.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 1, characterized in that, in the process of performing patrol inspection/operation tasks, the pose of the robot arm equipped with inspection/operation tools is adjusted in real time ,include:

Based on the position data of the equipment to be inspected, the posture of the end of the robotic arm is adjusted so that the equipment to be inspected is located in the center of the image, and the state of the equipment to be inspected can be adjusted in real time.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 1, characterized in that after identifying the position of the equipment to be inspected, the position of the equipment to be inspected is also tracked, and the real-time location information of the equipment to be inspected is sent. Give the robot arm control module.
The humanoid patrol operation method of a semantic intelligent substation robot according to claim 1, characterized in that, in the process of performing patrol inspection/operation tasks, the pose of the robot arm equipped with inspection/operation tools is adjusted in real time ,include:

Control the robot arm to adjust the pose and always aim at the device to be inspected, so that the robot always maintains the best relative pose relationship with the device to be inspected during data collection;

When the robot reaches the optimal observation pose of the device to be inspected and enters the range of the inspection data acquisition device, the deep learning algorithm is used to identify and obtain the position of the device in the image, and the relative pose relationship between the robot and the device to be inspected is used to determine Realize the spatial pose control of the acquisition device carried at the end of the robotic arm;

The quality of the collected data is evaluated and optimized, so as to realize the optimal collection of the inspection data of the equipment to be inspected.
The human-like patrol operation method of a semantic intelligent substation robot according to claim 18, characterized in that, in the process of quality evaluation and optimization of the collected data, a relationship model of the optimal image acquisition points for patrol inspection based on historical data changes with time is adopted. , in order to realize the independent optimal selection of inspection points in different seasons and different time periods.
The human-like patrol operation method of a semantic intelligent substation robot according to claim 18, characterized in that, in the process of collecting data quality evaluation and optimization, a confidence evaluation is performed on the patrol data at different locations and under different lighting conditions, and the robot During the inspection process, the detection data with the highest confidence is selected as the inspection status data of the equipment to be inspected.
A robot, characterized in that the patrol inspection is carried out by adopting the humanoid patrol operation method of a semantic intelligent substation robot according to any one of claims 1-20.
A semantic intelligent substation robot humanoid patrol operation system, characterized by comprising:

control center;

at least one robot; the robot is deployed in various areas in the substation;

Each robot includes a robot body, the robot body is provided with a robotic arm, and the end of the robotic arm is equipped with an inspection/work tool;

A computer program is stored in the control center, and when the program is executed by the processor, the steps in the humanoid patrol operation method of a semantic intelligent substation robot according to any one of claims 1-20 are implemented.
A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the humanoid patrol operation method of a semantic intelligent substation robot according to any one of claims 1-20 is realized. A step of.