CN115526897A

CN115526897A - Flame positioning method and system for fire-fighting robot of extra-high voltage converter station

Info

Publication number: CN115526897A
Application number: CN202211144984.1A
Authority: CN
Inventors: 张佳庆; 刘睿; 励刚; 王刘芳; 罗沙; 谢佳; 朱太云; 过羿; 尚峰举; 孙韬; 何灵欣
Original assignee: State Grid Corp of China SGCC; Electric Power Research Institute of State Grid Anhui Electric Power Co Ltd
Current assignee: State Grid Corp of China SGCC; Electric Power Research Institute of State Grid Anhui Electric Power Co Ltd
Priority date: 2022-09-20
Filing date: 2022-09-20
Publication date: 2022-12-27

Abstract

The invention discloses a flame positioning method and a system for a fire-fighting robot of an extra-high voltage converter station, wherein the method comprises the steps of collecting left and right visible light images and left and right infrared images of a flame area; performing flame identification detection on the left and right eye infrared images, and taking the maximum flame outline region in the left and right eye infrared images as an ROI (region of interest); based on the ROI area, performing stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images to obtain a flame three-dimensional coordinate; and converting the flame three-dimensional coordinate into the coordinate system of the fire-fighting robot to obtain the flame coordinate in the coordinate system of the fire-fighting robot. According to the invention, the infrared image is used as the ROI area, and image segmentation is not carried out on the visible light image independently, so that the positioning precision and speed are obviously improved; specifically, the infrared images of the left eye and the right eye or the visible light images of the left eye and the right eye are selected for stereo matching and parallax calculation, so that the accuracy of flame positioning is improved.

Description

Extra-high voltage converter station fire-fighting robot flame positioning method and system

Technical Field

The invention relates to the technical field of fire detection and positioning, in particular to a flame positioning method and a flame positioning system for a fire-fighting robot of an extra-high voltage converter station.

Background

At present, a relatively complete fixed fire extinguishing system application scheme is formed in an extra-high voltage converter transformer area, a fire-fighting robot is arranged in a station, a single spectrum camera or a binocular camera is usually arranged in the robot, one binocular camera is used for collecting visible light, the other binocular camera is used for collecting infrared spectrum, and the mode is difficult to handle the condition that flame is shielded by smoke or other buildings. And the calculated amount is large when flame detection and three-dimensional positioning are carried out, and the real-time performance and the precision are insufficient, so that the accurate release fire extinguishing of the fire-fighting robot is restricted.

In the related art, chinese patent publication No. CN108665487A describes a method for positioning an operation object and a target of a substation based on infrared and visible light fusion, and the technical solution is: the method comprises the steps of collecting field images in real time by using a visible light and infrared sensor system deployed on a transformer substation site, carrying out denoising, fusion and enhancement preprocessing by using a distributed image processing platform, further carrying out segmentation and target feature extraction on a target and a background so as to detect a field invasion target, and then carrying out identification, positioning and tracking on a dynamic target.

The fire source positioning method based on the thermal infrared imager is implemented in fire control science and technology, engineering science and technology II, 3 months and 15 days in 2019, and the area of a suspected fire source is divided from the background by using the temperature characteristics sensed by the thermal infrared imager; obtaining a pixel set of each target area by utilizing a Blob extraction algorithm on the binarized image; on the basis, extracting dynamic features of the image by using an optical flow method; a method for segmenting the mean value is provided, and whether the target area is a fire source is judged by combining other characteristics of the optical flow direction.

Based on infrared and visible light image fire positioning research, ship electronic engineering, engineering science and technology II, 20 d 3/2022, aiming at the problem of infrared and visible light camera combined calibration, a calibration plate capable of calibrating an infrared camera and a visible light camera simultaneously is designed by the article by utilizing the heating difference of different materials, the infrared camera and the visible light camera are calibrated by utilizing a camera module in Matlab, fire videos at different positions are collected, and a fire positioning experiment is carried out.

In the related art, a single-spectrum camera or a binocular camera is adopted, one binocular camera is used for collecting visible light, the other binocular camera is used for collecting infrared spectrum, registration of visible light images and infrared images is needed, and when flames are shielded by smoke or other buildings, accurate positioning of the flames cannot be achieved.

In the related art, chinese patent publication No. CN114694341A describes a visible-near infrared optical mechanical structure for fire monitoring, which includes a camera support, a lens, an optical filter switching mechanism, a detector, and a motor, where the optical filter switching mechanism includes a sector turntable, an optical filter, and a fixing device, the sector turntable is provided with two symmetrical mounting holes, the optical filter is fixedly mounted in the optical filter and fixed in the mounting hole by an optical filter pressing ring, the fixing device is used to fasten the optical filter switching mechanism and a motor rotating shaft, the optical filter includes a visible optical filter and a near infrared optical filter, and the optical filter switching system is used to realize smoke identification in a visible band and optical fog penetration in a near infrared band.

However, the monocular camera is arranged in the scheme, the visible light filtering or near infrared light filtering piece is switched through the light filtering piece switching mechanism, the smoke recognition function of the visible wave band and the optical fog penetration function of the near infrared wave band are realized, and the functions are still the collected monocular visible light image and the monocular near infrared image.

Disclosure of Invention

The invention aims to solve the technical problem of how to improve the accuracy of flame identification and positioning.

The invention solves the technical problems through the following technical means:

the invention provides a flame positioning method for a fire-fighting robot of an extra-high voltage converter station, which comprises the following steps:

collecting left and right eye visible light images and left and right eye infrared images of a flame area;

carrying out flame identification detection on the left and right eye infrared images, marking flame outer frames, and taking the maximum flame outer frame area in the left and right eye infrared images as an ROI (region of interest);

based on the ROI, performing stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images to obtain flame three-dimensional coordinates;

and converting the flame three-dimensional coordinate into a coordinate system of the fire-fighting robot to obtain the position coordinate of the flame in the coordinate system of the fire-fighting robot.

According to the invention, the binocular visible light image and the binocular infrared image are acquired, the flame detection processing is carried out on the acquired infrared image, the ROI area is determined, the infrared image is used as the basic image for ROI area detection, and the image segmentation is not carried out on the visible light image alone, so that the matching efficiency is improved as much as possible, the matching calculation amount is reduced, and the positioning precision and speed are obviously improved; and the visible light imaging has abundant contrast, color and shape information, so that a detected target can be rapidly identified and measured, but the flame surrounded by smoke cannot be imaged, the infrared image can sense the temperature information of the detected object, and the infrared imaging is obvious compared with the visible light imaging under the special environmental conditions of weak light, smoke and the like.

Further, the fire-fighting robot installs dual-band image acquisition module, dual-band image acquisition module includes two visible light cameras and two infrared filter, two infrared filter can remove respectively and leave or cover two visible light cameras, utilize fire-fighting robot to gather the regional visible light image of the left and right eyes infrared image of flame and control the mesh infrared image, include:

calibrating the two visible light cameras to obtain internal reference, external reference and distortion coefficients of the left camera and the right camera;

driving the infrared filter to leave the visible light camera, and collecting left and right visible light images of a flame area by using the visible light camera;

and driving the infrared filter to cover the visible light camera, and acquiring left and right infrared images of the flame area by using the visible light camera.

Further, the performing flame identification detection on the left and right eye infrared images, marking flame outer frames, and taking the maximum flame outer frame region in the left and right eye infrared images as an ROI region includes:

performing flame identification detection on the left and right eye infrared images by using a convolutional neural network model;

judging whether flames meeting a set threshold value are detected in the binocular infrared images or not;

if so, performing flame identification detection results based on the left and right eye infrared images, and taking the maximum flame outer frame region in the left and right eye infrared images as an ROI region;

and if not, moving the fire-fighting robot, and reusing the fire-fighting robot to acquire the binocular visible light image and the binocular infrared image of the flame area.

Further, the performing stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images based on the ROI area to obtain a flame three-dimensional coordinate includes:

performing flame identification detection on the left and right eye visible light images by using a convolutional neural network model, and judging whether flames are completely detected in the left and right eye visible light images or not;

if yes, performing stereo matching and parallax calculation on the left and right visible light images based on the ROI to obtain a flame three-dimensional coordinate;

and if not, performing stereo matching and parallax calculation on the left and right eye infrared images based on the ROI area to obtain the flame three-dimensional coordinate.

Further, the convolutional neural network model adopts an SSD target identification network, a trunk feature extraction network of the SSD target identification network is a lightweight network MobileNet V2, and an attention mechanism Squeeze-and-Excitation is added into the lightweight network MobileNet V2.

on the basis of the ROI, matching all feature points in corresponding ROI areas in the left and right eye infrared images or the left and right eye visible light images to correspondingly obtain a first feature matching set or a second feature matching set;

taking the absolute value of the coordinate difference of each characteristic point pair in the first characteristic matching set or the second characteristic matching set in the horizontal axis direction as the parallax of the characteristic point pair;

filtering matching costs of the parallaxes of all the feature point pairs in the first feature matching set or the second feature matching set, and then obtaining a parallax average value D of all the feature point pairs as a final parallax;

and calculating the final parallax based on a triangulation distance measuring principle to obtain the flame three-dimensional coordinate.

Further, based on the principle of triangulation, the final parallax is calculated, and a formula for obtaining the flame three-dimensional coordinate is represented as:

in the formula: (u) _L ,v _L ) The coordinate of a target flame point on a left eye camera is shown, f is the focal length of the camera, b is the distance between the optical centers of the left eye camera and the right eye camera, D is the final parallax, x is the coordinate in the horizontal direction, y is the coordinate in the vertical direction, and z is the distance.

Further, before the matching all feature points in the corresponding ROI regions in the left-eye and right-eye infrared images or the left-eye and right-eye visible light images based on the ROI regions and obtaining the first feature matching set or the second feature matching set correspondingly, the method further includes:

calculating the similarity of corresponding ROI areas in the left and right eye infrared images or the left and right eye visible light images by adopting a mean square error method to obtain a similarity value;

judging whether the similarity value is smaller than a set threshold value or not;

if so, executing the matching of all feature points in the corresponding ROI areas in the left and right eye infrared images or the left and right eye visible light images based on the ROI areas to correspondingly obtain a first feature matching set or a second feature matching set;

and if not, carrying out flame identification detection on the binocular infrared image again to determine the ROI area.

In addition, the invention also provides a flame positioning system of the fire-fighting robot of the extra-high voltage converter station, and the system comprises:

the image acquisition module is used for acquiring left and right eye visible light images and left and right eye infrared images of the flame area;

the identification detection module is used for carrying out flame identification detection on the left and right eye infrared images, marking flame outer frames and taking the maximum flame outer frame area in the left and right eye infrared images as an ROI (region of interest);

the coordinate calculation module is used for carrying out stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images based on the ROI area to obtain flame three-dimensional coordinates;

and the coordinate conversion module is used for converting the flame three-dimensional coordinate into a coordinate system of the fire-fighting robot to obtain the position coordinate of the flame in the coordinate system of the fire-fighting robot.

Furthermore, the image acquisition module comprises a fixed base, and a left eye visible light camera, a right eye visible light camera, a left eye optical filter, a right eye optical filter, a left eye optical filter swing motor and a right eye optical filter swing motor are arranged on the fixed base;

the driving end of the left eye optical filter swing motor is connected with the left eye optical filter through a left eye optical filter swing rod; the driving end of the right eye optical filter swing motor is connected with the right eye optical filter through a right eye optical filter swing rod;

the left eye visible light camera, the right eye visible light camera, the left eye optical filter swing motor and the right eye optical filter swing motor are all connected with a controller.

The invention has the advantages that:

(1) According to the invention, the binocular visible light image and the binocular infrared image are collected, the flame detection processing is carried out on the collected infrared image, the ROI area is determined, the infrared image is adopted as the basic image for ROI area detection, and the image segmentation is not carried out on the visible light image alone, so that the matching efficiency is improved as much as possible, the matching calculation amount is reduced, and the positioning precision and speed are obviously improved; and the visible light imaging has abundant contrast, color and shape information, so that a detected target can be rapidly identified and measured, but the flame surrounded by smoke cannot be imaged, the infrared image can sense the temperature information of the detected object, and the infrared imaging is obvious compared with the visible light imaging under the special environmental conditions of weak light, smoke and the like.

(2) The binocular visible light camera is adopted to configure the structure of the infrared filter, the effect of the two visible light cameras and the two infrared cameras is achieved, registration calculation of the infrared images and the visible light images is not needed, image processing time is shortened, working efficiency is improved, requirements for processing calculation force are lowered, power consumption is reduced, working time of the fire-fighting robot powered by a battery is increased, cost is saved, and system complexity is lowered.

(3) The method comprises the steps of adopting a lightweight network MobilNet V2 with an attention mechanism of Squeeze-and-Excitation to carry out flame recognition on a left-eye visible light image, a right-eye visible light image and a left-eye infrared image, using the lightweight network MobilNet V2 to achieve balance between operation speed and detection precision, relieving difficulty in actual deployment through less parameter quantity and calculation quantity, and adding the Squeeze-and-Excitation attention mechanism on the basis of the lightweight network MobilNet V2 to increase precision through extremely small calculation quantity.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

FIG. 1 is a schematic flow chart of a method for positioning flames of a fire-fighting robot of an extra-high voltage converter station according to an embodiment of the invention;

FIG. 2 is a schematic structural diagram of a dual-band image acquisition module according to an embodiment of the present invention;

FIG. 3 is a functional block diagram of a dual band image capture module in accordance with an embodiment of the present invention;

FIG. 4 is a schematic diagram illustrating a state between an infrared filter and a visible light camera according to an embodiment of the present invention;

FIG. 5 is a flow chart illustrating the subdivision step of step S10 according to an embodiment of the present invention;

FIG. 6 is a flow chart illustrating the subdivision step of step S20 according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of the ROI area target frame in the IR image of the left eye and the right eye according to an embodiment of the present invention, wherein (a) is the flame target frame marked in the IR image of the left eye; (b) A flame target outer frame marked in the right eye infrared image;

FIG. 8 is a flow chart illustrating the subdivision step of step S30 according to an embodiment of the present invention;

FIG. 9 is a schematic overall flow chart of a flame positioning method for a fire-fighting robot in an extra-high voltage converter station according to an embodiment of the invention;

fig. 10 is a schematic flow chart illustrating stereo matching and positioning of the left-eye and right-eye infrared images or the left-eye and right-eye visible light images according to an embodiment of the present invention;

FIG. 11 is a flowchart illustrating the subdivision step of step S320 according to an embodiment of the present invention;

FIG. 12 is a schematic structural diagram of a fire-fighting robot flame positioning system of an extra-high voltage converter station according to an embodiment of the invention;

FIG. 13 is a block diagram of an identification detection module according to an embodiment of the invention;

FIG. 14 is a block diagram of a coordinate calculation module according to an embodiment of the present invention;

FIG. 15 is a fire extinguishing schematic diagram of a flame localization system of a fire-fighting robot of the extra-high voltage converter station in an embodiment of the invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention, and it is obvious that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.

As shown in fig. 1, a first embodiment of the present invention provides a method for positioning flames of a fire-fighting robot in an extra-high voltage converter station, where the method includes the following steps:

s10, collecting left and right eye visible light images and left and right eye infrared images of a flame area;

specifically, in this embodiment, two visible light cameras and two infrared spectrum cameras can be used to acquire the left and right visible light images and the left and right infrared images of the flame region.

It should be understood that, persons skilled in the art may also combine factors such as cost, and use other methods to collect the left and right eye visible light images and the left and right eye infrared images of the flame region, and this embodiment is not limited in particular.

It should be noted that, unlike the monocular visible light image and the monocular infrared image collected by the conventional binocular camera, the binocular visible light image and the binocular infrared image are collected for the flame region in the present embodiment, and the registration of the visible light image and the infrared image is not required.

S20, carrying out flame identification detection on the left and right eye infrared images, marking flame outer frames, and taking the maximum flame outer frame area in the left and right eye infrared images as an ROI (region of interest);

it should be noted that the Region Of Interest (ROI) for flame localization is a flame Region, and other regions do not need to be matched, and there are many cases Of mismatching, and calculation and elimination are also needed; if only the ROI is subjected to matching calculation, the required feature points are less relative to global matching, the calculation amount of matching is greatly reduced, and the positioning precision and speed are remarkably improved.

It should be noted that, after the flame regions in the left and right eye infrared images are identified and detected, the positions of the target frames are marked for the flame regions in the left and right eye infrared images, and then the position regions of the target frames are calculated, and the maximum region in the position regions of the target frames in the binocular infrared images is taken as the final ROI region, so as to ensure that the most feature points are extracted as far as possible, and improve the accuracy of flame positioning and positioning.

S30, performing stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images based on the ROI to obtain a flame three-dimensional coordinate;

specifically, in the embodiment, if flames can be detected in the binocular visible light image, the judgment is performed by using the visible light image, and because the visible light image has texture details with high spatial resolution and definition, the positioning accuracy can be effectively improved; if the flame can not be detected or can not be detected completely in the visible light image due to the smoke shielding reason, the flame positioning calculation is carried out by directly utilizing the left and right eye infrared images.

It should be noted that, in the present embodiment, the right and left eye infrared images are used as the ROI region detection base image, and image segmentation is not performed on the visible light image alone, so as to improve matching efficiency as much as possible.

And S40, converting the flame three-dimensional coordinate into the coordinate system of the fire-fighting robot to obtain the position coordinate of the flame in the coordinate system of the fire-fighting robot.

In the embodiment, the method comprises the steps of acquiring a binocular visible light image and a binocular infrared image, determining an ROI (region of interest) by adopting flame detection processing based on a deep learning algorithm on the acquired infrared image, and detecting a basic image by adopting the infrared image as the ROI, without independently carrying out image segmentation on the visible light image, so that the matching efficiency is improved as much as possible, the matching calculation amount is reduced, and the positioning precision and speed are obviously improved; and the visible light imaging has abundant contrast, color and shape information, so that a detected target can be rapidly identified and measured, but the flame surrounded by smoke cannot be imaged, the infrared image can sense the temperature information of the detected object, and the infrared imaging is obvious compared with the visible light imaging under the special environmental conditions of weak light, smoke and the like.

In one embodiment, the fire-fighting robot is equipped with a dual-band image acquisition module, as shown in fig. 2 to 4, the dual-band image acquisition module comprises a fixed base 7, and a left eye visible light camera 8, a right eye visible light camera 9, a left eye optical filter 10, a right eye optical filter 11, a left eye optical filter swing motor 12 and a right eye optical filter swing motor 13 are arranged on the fixed base 7;

the driving end of the left eye optical filter swing motor 12 is connected with the left eye optical filter 10 through a left eye optical filter swing rod 16; the driving end of the right eye optical filter swinging motor 13 is connected with the right eye optical filter 11 through a right eye optical filter swinging rod 17;

the left eye visible light camera 8, the right eye visible light camera 9, the left eye filter swing motor 12 and the right eye filter swing motor 13 are all connected with a controller.

Thereby this embodiment drives left and right mesh light filter pendulum rod through left and right mesh light filter swing motor respectively and drives left and right mesh light filter and remove, realizes leaving or covering left and right mesh visible light camera, realizes two wave band binocular image acquisition. Specifically, as shown in fig. 3, during normal operation, the moving motor drives the infrared filter to synchronously leave the visible light camera, and at this time, the same image is formed as a visible light image, and then the moving motor drives the infrared filter to synchronously cover the front of the camera, and at this time, the image is formed as an infrared image.

Accordingly, as shown in fig. 5 and 9, the step S10: the method for collecting the left and right eye visible light images and the left and right eye infrared images of the flame area comprises the following steps:

s110, calibrating the two visible light cameras to obtain internal parameters, external parameters and distortion coefficients of the left camera and the right camera;

it should be noted that, in this embodiment, the binocular visible light camera is calibrated to obtain the internal reference K of the left and right cameras _L 、K _R And an external reference T and a distortion coefficient, which are not repeated herein.

S120, driving the infrared filter to leave the visible light camera, and collecting left and right visible light images of a flame area by using the visible light camera;

and S130, driving the infrared filter to cover the visible light camera, and collecting left and right infrared images of the flame area by using the visible light camera.

The embodiment specifically adopts the dual-waveband binocular image acquisition module to acquire images, does not need to perform registration operation of visible light images and infrared images, can greatly reduce the difficulty of image matching and fusion in the later period, shortens the image processing time, improves the working efficiency, reduces the requirement on processing calculation force, reduces the power consumption, and is favorable for increasing the working time of a fire-fighting robot powered by a battery; and the method that the binocular visible light camera is configured with the infrared filter is adopted, the effects of two visible light cameras and two infrared cameras are realized, the cost is saved, and the complexity of the system is reduced.

In one embodiment, as shown in fig. 6, the step S20: carrying out flame identification detection on the left and right eye infrared images, marking a flame outer frame, and taking the maximum flame outer frame area in the left and right eye infrared images as an ROI (region of interest), wherein the method specifically comprises the following steps:

s210, performing flame identification detection on the left and right eye infrared images by using a convolutional neural network model;

s220, judging whether flames meeting a set threshold value are detected in the binocular infrared images, if so, executing a step S230, otherwise, executing a step S240;

s230, performing flame identification detection results based on the left and right eye infrared images, and taking the maximum flame outline region in the left and right eye infrared images as an ROI (region of interest);

specifically, in this embodiment, a convolutional neural network model is used to perform flame identification and detection on the left and right eye infrared images, and a target outer frame in a flame region is marked, where the formula is as follows:

S _l ＝{x _l ,y _l ,w _l ,h _l },x _l ,y _l ,w _l ,h _l ∈R；S _r ＝{x _r ,y _r ,w _r ,h _r },x _r ,y _r ,w _r ,h _r ∈R

in the formula: x is a radical of a fluorine atom _l ,y _l Coordinates of the upper left corner of the outer frame of the target in the infrared image of the left eye, w _l ,h _l The width and height of the target frame. x is a radical of a fluorine atom _r ,y _r Is infrared to the right eyeCoordinates of upper left corner of target outer frame in image, w _r ,h _r R is a real number field for the width and height of the target outline.

Note that, as shown in fig. 7, the target frame marked on the flame region recognized in the right-left eye infrared image is shown, fig. 7- (a) shows the flame target frame marked in the left eye infrared image (see a gray frame), and fig. 7- (b) shows the flame target frame marked in the right eye infrared image (see a gray frame).

Further, according to the positions of flame target frames of the left and right eye infrared images, the areas of the two target frames are calculated, the largest target frame area in the binocular ROI area is used as an interesting ROI area, and the formula is as follows:

S _ROI ＝{x,y,w,h},x＝min(x _l ,x _r ),y＝min(y _l ,y _r ),w＝max(w _l ,w _r ),h＝max(h _l ,h _r )。

it should be noted that, in the present embodiment, the target frame with the largest region is used as the ROI region of interest, so as to ensure that as many feature points as possible are extracted, and ensure the accuracy of flame positioning.

S240, moving the fire-fighting robot, and recycling the fire-fighting robot to acquire the binocular visible light image and the binocular infrared image of the flame area.

Specifically, if only the monocular imaging is performed, if only the flame meeting the set threshold is detected in the monocular infrared image, it is indicated that other buildings are shielded, and if only the monocular imaging is performed, if only the flame is detected in the left eye infrared image, it is indicated that the fire-fighting robot is not aligned with the flame, and the fire-fighting robot needs to move left until the flame can be detected in the binocular infrared image; if only flames are detected in the right eye infrared image, it is indicated that the fire fighting robot needs to move to the right to aim at the flames.

It should be noted that, by performing flame identification detection on the left and right eye infrared images, whether and what kind of shielding conditions occur can be judged according to the flame identification result, if the shielding conditions occur, a motion instruction is sent to the fire-fighting robot according to the processing result, the fire-fighting robot is controlled to move to a proper position to re-collect the visible light image and the infrared image of the flame region, and after the binocular image is subjected to stereo correction, stereo matching processing is performed according to the determined ROI region, so that the accuracy of the flame positioning result is improved.

Further, in this embodiment, the images of the left and right eye cameras are subjected to stereo correction according to the calibration data of the binocular cameras, and then stereo matching is performed, when stereo matching is performed, the matching precision directly affects the ranging precision, and the stereo matching speed directly affects the ranging efficiency. The general global matching calculation amount is large, and directly influences the real-time performance of the system, so the embodiment considers reducing unnecessary global pixel processing to reduce the calculation amount to improve the matching speed and precision, for flame positioning, a region of interest (ROI) is a flame region, other regions are not necessary for matching, and many situations of wrong matching exist, and calculation and elimination are needed. In the embodiment, only the ROI is subjected to matching calculation, the required feature points are fewer compared with global matching, the calculation amount of matching is greatly reduced, and the positioning precision and speed are obviously improved.

In an embodiment, as shown in fig. 8 and 10, the step S30: and performing stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images based on the ROI area to obtain a flame three-dimensional coordinate, which specifically comprises the following steps:

s310, performing flame identification detection on the left and right eye visible light images by using a convolutional neural network model, judging whether flames are completely detected in the left and right eye visible light images, if so, executing a step S320, and if not, executing a step S330;

s320, performing stereo matching and parallax calculation on the left and right visible light images based on the ROI to obtain three-dimensional coordinates of the flame;

s330, performing stereo matching and parallax calculation on the left and right eye infrared images based on the ROI to obtain the flame three-dimensional coordinates.

It should be noted that, in this embodiment, the convolutional neural network model is used to perform flame identification detection on the left and right eye visible light images and the left and right eye infrared images, and if flames can be detected in the binocular visible light images, the visible light images are used for determination, because the visible light images have texture details with high spatial resolution and definition, the positioning accuracy can be effectively improved, and therefore, the visible light images are used as much as possible in the following processing; and if the visible light image cannot be detected or the flame cannot be completely detected due to the smoke shielding reason, directly judging by using the infrared image.

In an embodiment, the convolutional neural network model adopts an SSD target recognition network, a trunk feature extraction network of the SSD target recognition network is a lightweight network MobileNetV2, and the lightweight network MobileNetV2 is added with an attention system Squeeze-and-Excitation.

It should be noted that, in the present embodiment, a balance between an operation speed and a detection accuracy is achieved by using the lightweight network MobilNetV2, and a small amount of parameters and calculation amount alleviate difficulty in actual deployment, and based on the lightweight network MobilNetV2, an Squeeze-and-Excitation attention mechanism is added, so that an extremely small amount of calculation is increased, and the accuracy is increased.

Specifically, in the present embodiment, the attention mechanism Squeeze-and-Excitation is added to each Bottlenck operation in the lightweight network MobileNetV2, all spatial information is compressed into one channel by using the attention mechanism, and a dependency relationship between channels is explicitly constructed by performing channel-level activation on a feature map, so as to enhance the characterization capability of the convolutional neural network.

In one embodiment, as shown in fig. 11, the step S320: based on the ROI, stereo matching and parallax calculation are carried out on the left and right eye visible light images to obtain the flame three-dimensional coordinate, and the method comprises the following steps of:

s321, matching all feature points in the corresponding ROI region in the left and right eye visible light images based on the ROI region to obtain a second feature matching set;

it should be noted that, stereo matching is performed through the ROI in the binocular visible light image, and first, feature extraction is performed on corresponding ROI in the left and right eye visible light images, in this embodiment, an ORB feature method is specifically adopted, and then, matching of all feature points is performed on feature points in the left and right eye visible light images by using a brute force matching method.

S322, taking an absolute value of a coordinate difference of each feature point pair in the second feature matching set in the horizontal axis direction as a disparity of the feature point pair;

s323, after filtering processing of matching cost is carried out on the parallaxes of all the feature point pairs in the second feature matching set, solving a parallax average value D of all the feature point pairs as a final parallax;

it should be noted that, in this embodiment, the coordinate difference d of each pair of feature points in the left and right eye visible light images in the horizontal axis direction is solved, and the absolute value of the coordinate difference d is used as the parallax of the calculated distance; and after acquiring the parallaxes of all the feature points in the ROI area, performing filtering processing of matching cost to remove the maximum and minimum values, and then solving the final parallaxes taking the average parallaxes D in the ROI area as targets.

And S324, calculating the final parallax based on a triangular distance measurement principle to obtain the flame three-dimensional coordinate.

It should be understood that, based on the ROI region, the step of performing stereo matching and parallax calculation on the left-eye and right-eye infrared images to obtain the three-dimensional coordinates of flames is similar to the process of performing stereo matching and parallax calculation on the left-eye and right-eye visible light images, specifically:

on the basis of the ROI, matching all feature points in the corresponding ROI area in the left and right eye infrared images to correspondingly obtain a first feature matching set;

taking the absolute value of the coordinate difference of each characteristic point pair in the first characteristic matching set in the horizontal axis direction as the parallax of the characteristic point pair;

filtering matching cost of the parallaxes of all characteristic point pairs in the first characteristic matching set, and then obtaining a parallax average value D of all characteristic point pairs as a final parallax;

In an embodiment, in step S324, based on a principle of triangulation, the final parallax is calculated, and a formula of the flame three-dimensional coordinates is expressed as:

In one embodiment, in the step S321: based on the ROI area, matching all feature points in the corresponding ROI areas in the left-eye and right-eye visible light images to obtain a second feature matching set, or based on the ROI area, matching all feature points in the corresponding ROI areas in the left-eye and right-eye infrared images to obtain a first feature matching set, and the method further comprises the following steps:

calculating the similarity of corresponding ROI (region of interest) areas in the left and right eye infrared images or the left and right eye visible light images by adopting a mean square error method to obtain a similarity value;

if yes, matching all feature points in corresponding ROI areas in the left and right eye infrared images or the left and right eye visible light images based on the ROI areas to correspondingly obtain a first feature matching set or a second feature matching set;

Specifically, a Mean Square Error (MSE) method is used to calculate the similarity of ROI regions of left and right infrared images or left and right visible light images, and the formula is as follows:

wherein: n is a radical of hydrogen _u ,N _v Number of pixels in width and height of ROI area, f _l (u,v),f _r (u, v) are gray values at points (u, v) on the left eye infrared image and the right eye infrared image respectively, as long as the left and right eye images can be regarded as the same target when the MSE value is smaller than a set threshold (usually 0.01-0.06, set according to actual conditions), and if the MSE value is not in accordance with the threshold, the left and right eye images are reprocessed to B1.

In the embodiment, the similarity of the ROI areas of the left infrared image and the right infrared image or the similarity of the ROI areas of the left visible light image and the right visible light image is calculated to confirm whether the flame targets in the left infrared image, the right infrared image and the left visible light image are the same target or not, so that the accuracy of flame positioning is ensured.

In the embodiment, the acquired infrared image is subjected to flame detection processing based on a deep learning algorithm to determine the ROI (region of interest), whether shielding conditions occur or not and what shielding conditions occur can be judged according to flame identification results, and if shielding conditions occur, a motion instruction is sent to the fire-fighting robot to move to a proper position according to the processing results; and performing stereo matching processing on the binocular image according to the determined ROI after stereo correction, calculating the three-dimensional coordinate of the flame by adopting a binocular ranging algorithm, converting the three-dimensional coordinate into the three-dimensional coordinate of a robot coordinate system according to the transformation relation between a camera coordinate system and a fire-fighting robot coordinate system, and driving the fire-fighting monitor head to adjust to be suitable for posture accurate release.

As shown in fig. 12, a second embodiment of the present invention provides a fire-fighting robot flame positioning system for an extra-high voltage converter station, where the system includes:

the image acquisition module 100 is used for acquiring left and right eye visible light images and left and right eye infrared images of a flame area;

the identification detection module 200 is used for performing flame identification detection on the left and right eye infrared images, marking flame outer frames, and taking the maximum flame outer frame area in the left and right eye infrared images as an ROI (region of interest);

the coordinate calculation module 300 is configured to perform stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images based on the ROI region to obtain a flame three-dimensional coordinate;

and the coordinate conversion module 400 is used for converting the flame three-dimensional coordinate into a coordinate system of the fire-fighting robot to obtain the position coordinate of the flame in the coordinate system of the fire-fighting robot.

3. In the embodiment, the method comprises the steps of acquiring binocular visible light images and binocular infrared images, determining the ROI (region of interest) by adopting flame detection processing based on a deep learning algorithm on the acquired infrared images, and using the infrared images as the basic images for ROI area detection without independently carrying out image segmentation on the visible light images, so that the matching efficiency is improved as much as possible, the matching calculation amount is reduced, and the positioning precision and speed are obviously improved; and the visible light imaging has abundant contrast, color and shape information, so that a detected target can be rapidly identified and measured, but the flame surrounded by smoke cannot be imaged, the infrared image can sense the temperature information of the detected object, and the infrared imaging is obvious compared with the visible light imaging under the special environmental conditions of weak light, smoke and the like.

In an embodiment, the image capturing module 100 is installed on a fire-fighting robot, and specifically includes a fixed base 7, as shown in fig. 2, a left eye visible light camera 8, a right eye visible light camera 9, a left eye optical filter 10, a right eye optical filter 11, a left eye optical filter swinging motor 12, and a right eye optical filter swinging motor 13 are arranged on the fixed base 7;

the driving end of the left eye optical filter swinging motor 12 is connected with the left eye optical filter 10 through a left eye optical filter swinging rod 16; the driving end of the right eye optical filter swinging motor 13 is connected with the right eye optical filter 11 through a right eye optical filter swinging rod 17;

the left eye visible light camera 8, the right eye visible light camera 9, the left eye filter swing motor 12 and the right eye filter swing motor 13 are all connected with a controller 18.

Thereby this embodiment drives left and right mesh light filter pendulum rod through left and right mesh light filter swing motor respectively and removes to leave or cover left and right mesh visible light camera, realizes two wave band binocular image acquisition. Specifically, as shown in fig. 3, during normal operation, the moving motor drives the infrared filter to synchronously leave the visible light camera, at this time, the infrared filter forms a visible light image, and then the moving motor drives the infrared filter to synchronously cover the front of the camera, at this time, the infrared image is formed.

Further, a left eye camera component mounting slide seat 14 and a right eye camera component mounting slide seat 15 are mounted on the fixing base, the left eye visible light camera 8 and the left eye optical filter swing rod 16 are mounted on the left eye camera component mounting slide seat 14, and the right eye visible light camera 9 and the right eye optical filter swing rod 17 are mounted on the right eye camera component mounting slide seat 15.

In the embodiment, a structure that the binocular visible light camera is configured with the infrared filter is adopted, so that the effects of two visible light cameras and two infrared cameras are realized, registration calculation of infrared images and visible light images is not needed, the cost is saved, and the complexity of the system is reduced; but the degree of difficulty of later stage image matching fusion shortens image processing time, improves work efficiency, reduces the requirement to handling the power of calculating, reduces the consumption, is favorable to increasing battery powered fire-fighting robot's operating time.

Further, in this embodiment, the controller 18 is composed of Jetson XavierNX of NVIDIA, the recognition detecting module 200, the coordinate calculating module 300, and the coordinate converting module 400 are integrated as an image processor and installed on the fire-fighting robot, and wifi and CAN bus communication is adopted between the controller and the image processor.

Specifically, as shown in fig. 4, the image output ends of the left eye visible light camera 8 and the right eye visible light camera 9 are both connected with the input of the controller 18, the driving end of the controller 18 is connected with the left eye optical filter 10 and the right eye optical filter 11 through the left eye optical filter swing motor 12 and the right eye optical filter swing motor 13, respectively, the image output end of the controller 18 is connected with an image processor, and the image processor is used for processing the left eye infrared image, the right eye infrared image, the left eye visible light image and the right eye visible light image.

The controller 18 is used for driving the motor to rotate so as to drive the left and right eye optical filters to move away from or cover the left and right eye visible light cameras, and dual-waveband binocular image acquisition is achieved.

In one embodiment, as shown in fig. 13, the recognition detection module 200 includes:

the first identification detection unit 210 is configured to perform flame identification detection on the left and right eye infrared images by using a convolutional neural network model;

the first judging unit 220 is configured to judge whether flames meeting a set threshold are detected in the binocular infrared images;

an ROI region determining unit 230, configured to perform flame identification detection based on the left and right eye infrared images when an output result of the first determining unit is yes, and use a region with a largest flame outline region in the left and right eye infrared images as an ROI region;

the image acquisition module 100 is configured to acquire the binocular visible light image and the binocular infrared image of the flame region again when the output result of the first determination unit is negative.

In one embodiment, as shown in fig. 14, the coordinate calculation module 300 includes:

the second identification detection unit 310 is configured to perform flame identification detection on the left and right eye visible light images by using a convolutional neural network model;

a second judging unit 320, configured to judge whether flames are completely detected in the left and right eye visible light images;

a coordinate calculation unit 330, configured to perform stereo matching and parallax calculation on the left and right eye visible light images based on the ROI region to obtain the flame three-dimensional coordinates when the output result of the second determination unit is yes; and the second judgment unit is used for performing stereo matching and parallax calculation on the left and right eye infrared images based on the ROI area to obtain the flame three-dimensional coordinate when the output result of the second judgment unit is negative.

In an embodiment, the convolutional neural network model adopts an SSD target identification network, a trunk feature extraction network of the SSD target identification network is a lightweight network MobileNetV2, and an attention mechanism Squeeze-and-Excitation is added to the lightweight network MobileNetV 2.

In an embodiment, the coordinate calculation unit 330 includes:

the matching subunit is used for matching all the feature points in the corresponding ROI areas in the left-eye infrared image, the right-eye infrared image or the left-eye visible light image and the right-eye visible light image based on the ROI areas to correspondingly obtain a first feature matching set or a second feature matching set;

an absolute value operator unit, configured to use an absolute value of a coordinate difference of each feature point pair in the first feature matching set or the second feature matching set in the horizontal axis direction as a disparity of the feature point pair;

a disparity calculating subunit, configured to perform filtering processing on the disparity of all feature point pairs in the first feature matching set or the second feature matching set according to matching cost, and then obtain a disparity average value D of all feature point pairs as a final disparity;

and the coordinate calculating subunit is used for calculating the final parallax based on a triangular distance measuring principle to obtain the flame three-dimensional coordinate.

It should be noted that, in the present embodiment, stereo matching is performed through the ROI in the binocular visible light image, first, feature extraction is performed on corresponding ROI in the left and right eye visible light images, in the present embodiment, an ORB feature method is specifically adopted, then, matching of all feature points is performed on feature points in the left and right eye visible light images by using a brute force matching method, and since only the ROI region image needs to be matched in the matching process, the calculation amount is small, and the real-time performance is excellent. Then solving the coordinate difference d of each pair of characteristic points in the left and right eye visible light images in the direction of the horizontal axis, and taking the absolute value of the coordinate difference d as the parallax of the calculated distance; and after acquiring parallaxes of all the feature points in the ROI area, performing filtering processing of matching cost, eliminating the maximum and minimum values, and then solving the final parallaxes taking the average parallaxes D in the ROI area as targets.

In one embodiment, the coordinate calculation subunit calculates the formula of the flame three-dimensional coordinate as:

In an embodiment, the coordinate calculation unit 330 further includes:

the similarity operator unit is used for calculating the similarity of the corresponding ROI areas in the left and right eye infrared images or the left and right eye visible light images by adopting a mean square error method to obtain a similarity value;

a judging subunit, configured to judge whether the similarity value is smaller than a set threshold;

the matching subunit is configured to, when the output result of the judging subunit is yes, execute the matching on all feature points in the corresponding ROI regions in the left-eye and right-eye infrared images or the left-eye and right-eye visible light images based on the ROI region, and obtain a first feature matching set or a second feature matching set correspondingly;

and the identification detection module is used for carrying out flame identification detection on the binocular infrared image again to determine the ROI area when the output result of the judgment subunit is negative.

Specifically, a Mean Square Error (MSE) is used to calculate the similarity of the ROI regions of the left and right infrared images or the left and right visible light images, and the formula is as follows:

wherein: n is a radical of _u ,N _v Number of pixels in width and height of ROI area, f _l (u,v),f _r (u, v) are infrared images of the left and right eyes, respectivelyLike the gray values at the points (u, v), the left and right target images can be regarded as the same target as long as the value of MSE is smaller than the set threshold (usually 0.01-0.06, set according to the actual situation), and if the MSE does not meet the threshold, the processing is resumed at B1.

It should be noted that, as shown in fig. 15, a working schematic diagram of fire extinguishing by using the flame positioning system of the ultra-high voltage converter station fire-fighting robot provided in this embodiment is shown, a fire monitor 3 and a flame positioning system 2 are installed on a body of the fire-fighting robot 4, converter transformers 1 are arranged in front of a valve hall 5, and a firewall 6 is arranged between the converter transformers 1. When the converter transformer catches fire, the fire-fighting robot 4 is used for identifying and positioning the flame, and then the fire monitor is used for releasing accurately to extinguish the fire.

It should be noted that, other embodiments or implementation methods of the flame locating system of the fire-fighting robot in the extra-high voltage converter station according to the present invention may refer to the above-mentioned embodiments, and no redundancy is provided here.

It should be noted that the logic and/or steps represented in the flowcharts or otherwise described herein, such as an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Further, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or to implicitly indicate the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims

1. A flame positioning method for a fire-fighting robot of an extra-high voltage converter station is characterized by comprising the following steps:

based on the ROI area, performing stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images to obtain a flame three-dimensional coordinate;

2. An extra-high voltage converter station fire-fighting robot flame positioning method according to claim 1, wherein the fire-fighting robot is provided with a dual-band image acquisition module, the dual-band image acquisition module comprises two visible light cameras and two infrared filters, the two infrared filters can respectively move away from or cover the two visible light cameras, and the method for acquiring left and right eye visible light images and left and right eye infrared images of a flame area comprises:

calibrating the two visible light cameras to obtain the internal reference, the external reference and the distortion coefficient of the left camera and the right camera;

3. The method for positioning flames of a fire-fighting robot in an extra-high voltage converter station according to claim 1, wherein the steps of identifying and detecting flames of the left and right eye infrared images, marking outer frames of the flames, and taking the largest outer frame region of the flames in the left and right eye infrared images as an ROI (region of interest) comprise:

if so, performing flame identification detection results based on the left and right eye infrared images, and taking the maximum flame outline region in the left and right eye infrared images as an ROI region;

4. The method for positioning flames of a fire-fighting robot in an extra-high voltage converter station according to claim 1, wherein the step of performing stereo matching and parallax calculation on the left-eye infrared image and the right-eye visible light image based on the ROI area to obtain three-dimensional coordinates of flames comprises the steps of:

if yes, performing stereo matching and parallax calculation on the left and right visible light images based on the ROI area to obtain a flame three-dimensional coordinate;

5. The method for positioning flames of a fire-fighting robot in an extra-high voltage converter station according to claim 3 or 4, wherein the convolutional neural network model adopts an SSD target identification network, a trunk feature extraction network of the SSD target identification network is a lightweight network MobileNet V2, and an attention mechanism Squeeze-and-Excitation is added into the lightweight network MobileNet V2.

6. The method for positioning flames of an extra-high voltage converter station fire-fighting robot according to claim 1, wherein the step of performing stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images based on the ROI area to obtain three-dimensional coordinates of flames comprises the steps of:

filtering matching costs of the parallaxes of all feature point pairs in the first feature matching set or the second feature matching set, and then obtaining a parallax average value D of all feature point pairs as a final parallax;

and calculating the final parallax based on a triangular distance measurement principle to obtain the flame three-dimensional coordinate.

7. The method for positioning the flame of the fire-fighting robot at the extra-high voltage converter station as recited in claim 6, wherein the final parallax is calculated based on a triangulation theory, and a formula for obtaining the three-dimensional coordinate of the flame is expressed as:

in the formula: (u) _L ,v _L ) The coordinate of a target flame point on a left eye camera is taken as the pixel coordinate, f is the focal length of the camera, b is the optical center distance between the left eye camera and a right eye camera, D is the final parallax, x is the coordinate in the horizontal direction, y is the coordinate in the vertical direction, and z is the distance.

8. The method of claim 6, wherein before the matching of all feature points in corresponding ROI regions in the left-eye infrared image, the right-eye infrared image or the left-eye visible light image based on the ROI region and the corresponding obtaining of the first feature matching set or the second feature matching set, the method further comprises:

9. The utility model provides an extra-high voltage convertor station fire-fighting robot flame positioning system which characterized in that, the system includes:

the image acquisition module is used for acquiring left and right eye visible light images and left and right eye infrared images of a flame area;

the identification detection module is used for carrying out flame identification detection on the left and right eye infrared images, marking flame outer frames and taking the largest flame outer frame area in the left and right eye infrared images as an ROI (region of interest);

the coordinate calculation module is used for carrying out stereo matching and parallax calculation on the left and right eye infrared images or the left and right eye visible light images based on the ROI area to obtain a flame three-dimensional coordinate;

10. The system as claimed in claim 9, wherein the image capturing module comprises a fixed base, and a left eye visible light camera, a right eye visible light camera, a left eye filter, a right eye filter, a left eye filter swing motor, and a right eye filter swing motor are disposed on the fixed base;