CN117177027B - Double-spectrum fusion visual perception system and method based on crow's eyes layout - Google Patents
Double-spectrum fusion visual perception system and method based on crow's eyes layout Download PDFInfo
- Publication number
- CN117177027B CN117177027B CN202311444601.7A CN202311444601A CN117177027B CN 117177027 B CN117177027 B CN 117177027B CN 202311444601 A CN202311444601 A CN 202311444601A CN 117177027 B CN117177027 B CN 117177027B
- Authority
- CN
- China
- Prior art keywords
- visible light
- infrared
- shell
- explosion
- fixed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 31
- 238000001228 spectrum Methods 0.000 title claims abstract description 23
- 230000016776 visual perception Effects 0.000 title claims abstract description 22
- 238000000034 method Methods 0.000 title claims abstract description 17
- 239000000428 dust Substances 0.000 claims abstract description 76
- 238000004880 explosion Methods 0.000 claims description 15
- 238000005286 illumination Methods 0.000 claims description 13
- 238000003384 imaging method Methods 0.000 claims description 12
- 230000003287 optical effect Effects 0.000 claims description 7
- 150000001875 compounds Chemical class 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 6
- 230000008602 contraction Effects 0.000 claims description 3
- 230000009977 dual effect Effects 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 3
- 239000000853 adhesive Substances 0.000 claims description 2
- 230000001070 adhesive effect Effects 0.000 claims description 2
- 238000003672 processing method Methods 0.000 claims description 2
- 239000000779 smoke Substances 0.000 claims description 2
- 238000011065 in-situ storage Methods 0.000 claims 1
- 230000008685 targeting Effects 0.000 claims 1
- 238000012544 monitoring process Methods 0.000 abstract description 16
- 238000005065 mining Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 8
- 230000008878 coupling Effects 0.000 description 6
- 238000010168 coupling process Methods 0.000 description 6
- 238000005859 coupling reaction Methods 0.000 description 6
- 230000005641 tunneling Effects 0.000 description 5
- 229910000831 Steel Inorganic materials 0.000 description 4
- 210000000988 bone and bone Anatomy 0.000 description 4
- 239000010959 steel Substances 0.000 description 4
- 239000003245 coal Substances 0.000 description 3
- 238000009434 installation Methods 0.000 description 3
- 239000004033 plastic Substances 0.000 description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Chemical compound O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 3
- 241000271532 Crotalus Species 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 239000003292 glue Substances 0.000 description 2
- 210000003128 head Anatomy 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000011148 porous material Substances 0.000 description 2
- 241001137251 Corvidae Species 0.000 description 1
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 239000002390 adhesive tape Substances 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007797 corrosion Effects 0.000 description 1
- 238000005260 corrosion Methods 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000035515 penetration Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 239000004417 polycarbonate Substances 0.000 description 1
- 229920000515 polycarbonate Polymers 0.000 description 1
- 238000005507 spraying Methods 0.000 description 1
- 229910001220 stainless steel Inorganic materials 0.000 description 1
- 239000010935 stainless steel Substances 0.000 description 1
Landscapes
- Lenses (AREA)
Abstract
The invention discloses a double-spectrum fusion visual perception system and a double-spectrum fusion visual perception method based on a crow's eyes layout. The two visible light machine core modules are symmetrically arranged at the upper part in the shell, the two infrared machine core modules can be driven by the electric push rod to horizontally rotate by a certain angle, and the two infrared machine core modules are symmetrically arranged at the lower part in the shell, and can be driven by the electric push rod to horizontally rotate by a certain angle. The system can monitor on-site operators and equipment such as mining under the environments of low visibility and high dust of complex scenes such as mines through independent and joint work of the two movement lenses, and the monitoring capability is improved.
Description
Technical Field
The invention relates to the technical field of mining camera shooting and image processing, in particular to a double-spectrum fusion visual perception system and method based on a crow's tail compound eye layout.
Background
The underground operation working condition environment of the coal mine is severe, the tunneling operation is the most complex and dangerous front-end production link in the existing coal mining links, the space-time staggering of equipment, personnel and environment is complex, the potential safety hazard is prominent, in particular to the vicinity of the tunneling end, the dust concentration is higher, the typical illuminance value is lower, and the intervention of water vapor spraying causes the environmental factors to be more complex. In the tunneling environment with low illumination, high dust and high water vapor, the common mining imaging equipment cannot meet the strict monitoring requirement of the tunneling site, so that the problems of narrow imaging field range, incapability of completely monitoring the mining site, unclear image blurring and the like occur. In view of the foregoing, it is necessary to design a sensing system and method with scene detail representation capability and environment penetration.
Disclosure of Invention
The invention aims to provide a double-spectrum fusion visual perception system and method based on a crow's eyes layout, which can realize large-view-field detection by utilizing infrared light and visible light to cooperatively image, can acquire image detail texture information in a severe environment, and has important value in pushing the intellectualization of a tunneling working face and guaranteeing the safe production of a coal mine.
In order to achieve the aim, the invention discloses a double-spectrum fusion visual perception system based on the compound eye layout of the crow, which comprises an explosion-proof shell, a visible light core module, an infrared core module, an electric control device and a dust removal device,
the explosion-proof shell comprises a shell and an explosion-proof lens, wherein the middle of the front end face of the shell is outwards protruded, and the lower part of the front end face is inwards retracted; two sides of the middle line of the front end face of the shell are symmetrically provided with windows, two of each side are arranged up and down; the anti-explosion lens comprises two visible light anti-explosion lenses and two infrared light anti-explosion lenses, wherein the visible light anti-explosion lenses are fixed on the two upper windows, and the infrared light anti-explosion lenses are fixed on the two lower windows; the visible light movement module, the infrared movement module and the electric control device are positioned in the shell, and the dust removing device is arranged on the front end face of the shell;
the two visible light machine core modules are symmetrically arranged at the upper part in the shell, each visible light machine core module comprises a visible light machine core, a fixed seat I and a fixed plate I, the fixed plates I are horizontally fixed in the shell, the fixed seats I are rotatably connected with the fixed plates I, and the visible light machine core is arranged in the fixed seats I; the visible light engine core lens faces the visible light explosion-proof lens;
the two infrared core modules are symmetrically arranged at the lower part in the shell, each infrared core module comprises an infrared core, a fixed base II and a fixed flat plate II, the fixed flat plate II is horizontally fixed in the shell, the fixed base II is rotatably connected with the fixed flat plate II, and the infrared cores are arranged in the fixed base II; the infrared movement lens faces the infrared light explosion-proof lens;
the electric control device comprises an electric push rod I and an electric push rod II which are arranged in the shell, the electric push rod I is positioned between the two visible light machine core modules, the front end of the electric push rod I is connected with two long connecting rods, and the head part of each long connecting rod is hinged with one fixing seat I; the electric push rod II is positioned between the two infrared core modules, the front end of the electric push rod II is symmetrically connected with two short connecting rods, and each short connecting rod is hinged with one fixed seat II respectively;
the dust collector is used for removing dust to the explosion-proof lens, dust collector includes dust removal motor, shaft coupling, dust removal brush installation piece and boneless dust removal brush, dust removal motor install in the casing, the shaft coupling with the dust removal motor shaft allies oneself with, dust removal brush installation piece install in the shaft coupling tip, boneless dust removal brush with dust removal brush installation piece fixed connection.
Further, the lower parts of the left end surface and the right end surface of the shell are retracted; the upper end face of the shell is provided with an upper cover, the rear end face of the shell is provided with a cable connector, and the bottom face of the shell is provided with a bracket connecting piece.
Further, right-angle limiting blocks are arranged at four corners of the fixed seat I; the fixing base I comprises a main body plate and a telescopic plate, wherein the main body plate is connected with shaft holes at two ends of the telescopic plate, and the middle of the main body plate is connected through a screw rod to form a telescopic structure.
Further, the inside of right angle stopper is equipped with the soft adhesive paster of bradyseism.
Further, the electric putter I further comprises a fixing support, and the fixing support is installed on the fixing flat plate I and used for fixing the electric putter I.
Further, a rotating shaft and a limiting rod are arranged at the bottom of the fixed seat I, a rotating shaft hole and an arc-shaped notch are formed in the fixed plate I, the rotating shaft is arranged in the rotating shaft hole, and the limiting rod is arranged in the arc-shaped notch; the fixed seat I is provided with a connecting ring, and the front end of the long connecting rod is hinged with the connecting ring; the electric push rod I drives the fixing seat I to rotate on the fixing flat plate I by a certain angle through expansion and contraction, and the limiting rod and the arc notch are used for limiting the rotation angle.
On the other hand, the invention also discloses a double-spectrum fusion visual perception method, which is based on the double-spectrum fusion visual perception system based on the crow's tail compound eye layout and comprises the following steps:
s1, detecting the dust concentration and illumination condition of the site environment through a sensor;
s2, under the environment with good illumination condition and no dust, the visible light machine core module works, the two visible light machine cores are driven by the electric push rod I to horizontally rotate, the lenses of the visible light machine cores are parallel to forward or simultaneously rotate outwards by a certain angle, and the underground field image is acquired through the visible light explosion-proof lens;
s3, under the low-illumination and high-dust environment, the infrared machine core module works, the two infrared machine cores are driven to horizontally rotate by the electric push rod II, lenses of the two infrared machine cores are parallel to forward or simultaneously rotate outwards by a certain angle, and an underground field image is acquired through the infrared explosion-proof lens;
s4, under the environment of good illumination conditions and large smoke and dust, the visible light movement module and the infrared movement module work simultaneously, the two visible light movements and the two infrared movements rotate outwards by a certain angle simultaneously, the imaging range is enlarged, and a field image is acquired; the image is transmitted back to the computer for processing, and the image processing method specifically comprises the following steps:
(1) Respectively performing image stitching on the obtained visible light image and infrared light image through a computer to obtain a visible light large-view-field stitched image and an infrared large-view-field stitched image;
(2) Registering the visible light large view field spliced image and the infrared large view field spliced image, adding a self-adaptive weighting model through a fusion algorithm, setting weight of infrared information and visible light information during fusion according to an environment state, and carrying out semantic segmentation, target detection or target tracking visual tasks on the fused image.
Further, in performing the on-site object localization,
(1) The two visible light machine cores rotate until the two optical axes are parallel to obtain distance information of underground personnel and equipment;
(2) The two visible light cores and the two infrared cores rotate simultaneously until two optical axes are parallel, after the visible light images and the infrared images are spliced and subjected to image registration, the weight occupied by fusion of the visible light images and the infrared images is adaptively set according to the environment state, and underground personnel and equipment are positioned and tracked on the fused images.
The beneficial effects of the invention are as follows:
according to the invention, by arranging the binocular visible light core module and the binocular infrared core module, parallel imaging and simultaneous outward rotation imaging are realized through the lenses, and by independently monitoring the visible light cores or independently monitoring the infrared cores and cooperatively monitoring the infrared cores, the functions of large-field perception, accurate target positioning and effective target identification can be realized, the image detail information can be obtained in the severe environment with low illumination, high dust and high water vapor, and the adaptability and the monitoring flexibility of underground imaging are greatly enhanced.
Drawings
FIG. 1 is a schematic diagram of a double-spectrum fusion visual perception system based on a crow's eyes layout of a crow's tail;
FIG. 2 is a cross-sectional view of a double-spectrum fusion visual perception system based on a crow's eye layout of the present invention;
FIG. 3 is a schematic diagram of a core module of a visible light machine according to the present invention;
FIG. 4 is a schematic diagram of an infrared movement module according to the present invention;
fig. 5 is a schematic structural view of an electric putter I according to the present invention;
fig. 6 is a schematic structural diagram of a fixing base I of the present invention;
FIG. 7 is a schematic view of the structure of the boneless dust removing brush according to the present invention;
FIG. 8 is an overall block diagram of a binocular positioning mode of the present invention;
FIG. 9 is a diagram of the internal structure of the binocular positioning mode of the present invention;
FIG. 10 is an overall block diagram of a large field of view detection mode of the present invention;
FIG. 11 is a diagram of the internal structure of a large field of view detection mode of the present invention;
FIG. 12 is a flow chart of a dual spectrum fusion visual perception method of the present invention;
in the figure, a 1-explosion-proof shell, a 1-1-shell, a 1-2 visible light explosion-proof lens, a 1-3 infrared light explosion-proof lens, a 1-4-upper cover, a 1-5-cable interface, a 1-6-bracket connector, a 2-visible light core module, a 2-1-visible light core, a 2-2-fixing seat I, a 2-2-1-main body plate, a 2-2-2-expansion plate, a 2-2-3-screw, a 2-3-fixing plate I, a 2-4-right angle limiting block, a 2-5-cushioning soft rubber patch, a 2-6-connecting ring, a 3-infrared core module, a 3-1-infrared light, a 3-2-fixing seat II, a 3-3-fixing plate II, a 4-electric control device, a 4-1-electric push rod I, a 4-1-long connecting rod, a 4-1-2-fixing bracket, a 4-2-electric push rod II, a 4-2-1-short connecting rod, a 5-dust removing device, a 5-1-dust removing motor, a 5-2-3-connecting rod, a 5-3-brush-5-brushless bone dust piece, a brushless bone dust removing joint, a 3-4-4-4-brushless bone dust piece, a plastic joint and a brushless bone dust removing device are arranged.
Detailed Description
The invention will be described in further detail with reference to the accompanying drawings and specific examples.
As shown in fig. 1, the double-spectrum fusion visual perception system based on the crow's eyes layout of the crows comprises an explosion-proof shell 1, a visible light machine core module 2, an infrared core module 3, an electric control device 4 and a dust removal device 5. The explosion-proof housing 1 comprises a housing 1-1 and an explosion-proof lens, wherein the housing 1-1 is wide in whole and narrow in whole, the middle of the front end face is outwards convex, and the lower part of the front end face is inwards retracted. Two sides of the middle line of the front end face of the shell 1-1 are symmetrically provided with windows (namely windows), and two windows are arranged on each side and up and down. The anti-explosion lens comprises two visible light anti-explosion lenses 1-2 and two infrared light anti-explosion lenses 1-3, wherein the visible light anti-explosion lenses 1-2 are fixed on the two upper windows, and the infrared light anti-explosion lenses 1-3 are fixed on the two lower windows. When the explosion-proof lens is embedded in the window, namely the surface of the lens and the surface of the front end face are positioned on the same plane. The upper end face of the shell 1-1 is provided with an upper cover 1-4, the rear end face is provided with a cable connector 1-5, the bottom face is provided with a bracket connecting piece 1-6, and a bracket (not shown in the figure) connected with the bracket connecting piece 1-6 can enable the shell 1-1 to rotate up and down by a certain angle relative to the bracket. The shell 1-1 is made of stainless steel, is antistatic, impact-resistant and corrosion-resistant, and meets the underground explosion-proof requirement. The explosion-proof lens is made of polycarbonate material.
As shown in fig. 2, inside the casing 1-1, the visible light machine core module 2, the infrared core module 3 and the electric control device 4 are installed in the casing 1-1, and the dust removing device 5 is disposed on the front end surface of the casing 1-1.
As shown in FIG. 3, two visible light movement modules 2 are symmetrically arranged at the upper part in the shell 1-1, each visible light movement module comprises a visible light movement 2-1, a fixed base I2-2 and a fixed flat plate I2-3, the fixed flat plate I2-3 is horizontally fixed in the shell 1-1, the fixed base I2-2 is rotatably connected with the fixed flat plate I2-3, and the visible light movement 2-1 is arranged in the fixed base I2-2. The lens of the visible light movement 2-1 faces to the visible light explosion-proof lens 1-2. One fixing plate I2-3 carries two fixing seats I2-2. Regarding the rotational connection of the fixing base I2-2 and the fixing plate 2-3, the following is concrete: the bottom of the fixed seat I2-2 is provided with a rotating shaft and a limiting rod (not shown in the figure), the fixed flat plate I2-3 is provided with a rotating shaft hole and an arc notch, the rotating shaft is arranged in the rotating shaft hole, and the limiting rod is arranged in the arc notch. The fixed seat I2-2 is provided with a connecting ring 2-6, and the front end of the long connecting rod 4-1-1 is hinged with the connecting ring 2-6. The electric push rod I4-1 drives the fixed seat I2-2 to rotate on the fixed flat plate I2-3 by a certain angle through expansion and contraction, and the limiting rod and the arc notch are used for limiting the rotation angle.
As shown in FIG. 4, the infrared movement modules 3 are two, symmetrically arranged at the lower part in the shell 1-1, each of which comprises an infrared movement 3-1, a fixed base II3-2 and a fixed flat plate II3-3, wherein the fixed flat plate II3-3 is horizontally fixed in the shell 1-1, the fixed base II3-2 is rotatably connected with the fixed flat plate II3-3, and the infrared movement 3-1 is arranged in the fixed base II 3-2. The lens of the infrared movement 3-1 faces the infrared explosion-proof lens 1-3. The rotation arrangement between the fixing base II3-2 and the fixing plate II3-3 is the same as the device in the visible light machine core module 2, and will not be described here.
The visible light explosion-proof lens 1-2 is used for protecting the lens of the visible light movement 2-1, and the left-right width of the visible light explosion-proof lens can ensure that personnel and equipment conditions on the underground site can be seen after the lens of the visible light movement 2-1 rotates outwards. Similarly, the infrared explosion-proof lens 1-3 is used for protecting the lens of the infrared movement 3-1.
As can be seen from fig. 8, the horizontal moment between the two visible cores 2-1 is greater than the horizontal moment between the two infrared cores 3-1. In addition, regarding the arrangement of the visible light engine core module 2 and the infrared core module 3 in the housing 1-1, the infrared core module 3 may be placed above, and the visible light engine core module 2 may be placed below.
The visible light machine core 2 and the infrared machine core 3 are connected with a computer through a cable interface 1-5, and the acquired images are transmitted to the computer for image processing or on-site monitoring.
As shown in fig. 3, 4 and 5, the electric control device 4 comprises an electric push rod I4-1 and an electric push rod II4-2 which are installed in the shell 1-1, wherein the electric push rod I4-1 is positioned between the two visible light machine core modules 2, the front end of the electric push rod I4-1 is connected with two long connecting rods 4-1-1, and the head of each long connecting rod 4-1-1 is hinged with one fixed seat I2-2 respectively. The electric push rod II4-2 is positioned between the two infrared core modules 3, the front end of the electric push rod II4-2 is symmetrically connected with two short connecting rods 4-2-1, and each short connecting rod 4-2-1 is hinged with one fixed seat II3-2 respectively. The electric push rod I4-1 further comprises a fixing bracket 4-1-2, and the fixing bracket 4-1-2 is arranged on the fixing flat plate I2-3 and used for fixing the electric push rod I4-1.
As shown in 6, right-angle limiting blocks 2-4 are arranged at four corners of the fixed seat I2-2 and used for fixing the visible light movement 2-1. The fixing seat I2-2 comprises a main body plate 2-2-1 and a telescopic plate 2-2-2, wherein the main body plate 2-2-1 is connected with shaft holes at two ends of the telescopic plate 2-2, and the middle is connected through a screw rod 2-2-3 to form a telescopic structure. When the visible light movement 2-1 is installed, the visible light movement needs to be installed stably and is convenient to take down. Thus, a telescopic structure of the body plate 2-2-1 and the telescopic plate 2-2-2 is provided. The structure is that two ends of one side surface of a main body plate 2-2-1 are provided with optical axes, two matched pore channels are arranged on one side of the telescopic plate 2-2-2 opposite to the main body plate 2-2-1, and the optical axes and the pore produce sliding relation. The screw rod 2-2-3 passes through the expansion plate 2-2-2, the end is in threaded connection with the main body plate 2-2-1, when the visible light movement 2-1 is installed on the fixed seat I2-2, the screw rod 2-2-3 is screwed outwards, the bottom area of the fixed seat I2-2 is enlarged, when the visible light movement 2-1 is installed on the fixed seat I2-2 and needs to be fixed, the screw rod 2-2-3 is screwed inwards, and the bottom of the visible light movement 2-1 is firmly fixed, so that the purpose of fixing the visible light movement 2-1 is achieved. The inner side of the right-angle limiting block 2-4 is provided with a cushioning soft rubber patch 2-5, so that the visible light movement 2-1 can be prevented from being damaged when firm.
As shown in fig. 2 and 7, the dust removing device 5 is used for removing dust from an explosion-proof lens, the dust removing device 5 comprises a dust removing motor 5-1, a coupling 5-2, a dust removing brush mounting block 5-3 and a boneless dust removing brush 5-4, the dust removing motor 5-1 is mounted in the shell 1-1, the coupling 5-2 is in shaft connection with the dust removing motor 5-1, the dust removing brush mounting block 5-3 is mounted at the end part of the coupling 5-2, and the boneless dust removing brush 5-4 is fixedly connected with the dust removing brush mounting block 5-3. The boneless dust removing brush 5-4 comprises a plastic joint 5-4-1, a boneless dust removing brush steel sheet 5-4-2, a boneless dust removing brush adhesive tape 5-4-3 and a wiper arm 5-4-4. The boneless dust removing brush steel sheet 5-4-2 is connected with the wiper arm 5-4-4 through the plastic joint 5-4-1, so that the whole wiper can move along with the movement of the wiper arm 5-4-4. The boneless dust removing brush glue 5-4-3 is connected with the boneless dust removing brush steel sheet 5-4-2 in a fastening mode, so that the boneless dust removing brush glue strip 5-4-3 can be ensured to be firmly fixed on the boneless dust removing brush steel sheet 5-4-2. The dust removing device 5 drives the coupler 5-2 to rotate through the dust removing motor 5-1, and the dust removing brush mounting block 5-3 connected with the coupler 5-2 drives the boneless dust removing brush 5-4 to move back and forth. The dust removing devices 5 are two sets which are the same, and the dust removing operation is carried out on the four explosion-proof lenses on the front end face of the shell 1-1.
In addition, the bracket connecting piece 1-6 can be connected with the imaging bracket through a pin shaft, so that the imaging bracket can rotate up and down, and then the imaging bracket is matched with the parallel and outward rotation of the two visible light machine cores 2-1 and the two infrared machine cores 3-1, so that the monitoring range and the imaging range of the site can be enlarged.
The structure of the explosion-proof shell 1 and the two visible light cores 2-1 arranged above the inside and the two infrared cores 3-1 arranged below the inside of the explosion-proof shell refer to the compound eye structure of the rattlesnake, the rattlesnake can realize sensitive identification and positioning under different natural environments by virtue of the compound eye, and the system can realize on-site identification, on-site positioning and image detail information acquisition under complex environments in a mine, so that the adaptability and monitoring flexibility of underground image acquisition and image processing are greatly improved.
Two visible light cores 2-1 and two infrared cores 3-1 are arranged to respectively form a binocular camera, so that depth information of a monitored object can be obtained, and positioning is accurate. The larger the angle of view is, the more serious the image distortion is, if the large-view-range monitoring capability is required to be obtained, the large-scale monitoring can be realized by means of the rotation of the single-aperture movement, but the real-time large-scale monitoring cannot be realized easily, and the two cameras with smaller angles of view are used for monitoring and image splicing, so that the low-distortion large-view-field spliced image can be obtained, the large-view-range can be realized in real time, and the low-distortion image can be obtained. In addition, the images shot by the two apertures can realize high-resolution imaging of the overlapping area.
The invention also discloses a double-spectrum fusion visual perception method, as shown in fig. 12, which comprises the following steps:
s1, detecting dust concentration and illumination conditions of the underground site environment through various sensors.
S2, under the environment of good illumination and no dust, the visible light machine core module 2 works, the two visible light machine cores 2-1 are driven to rotate through the electric push rod I4-1, and the two lenses are parallel to forward or simultaneously rotate outwards by a certain angle to acquire field personnel and equipment images, and the binocular monitoring of fig. 8 and 9 can be referred to.
S3, under the low-illumination and high-dust environment, the infrared movement module 3 works, the two infrared movements 3-1 are driven to rotate through the electric push rod II4-2, and the two lenses are parallel to each other or simultaneously rotate outwards by a certain angle to obtain images of field personnel and equipment, and the binocular monitoring mode can be referred to in fig. 8 and 9.
S4, under the environment of good light but high dust, the visible light movement 2-1 and the infrared movement 3-1 work simultaneously and rotate outwards by a certain angle at the same time, and the large view field mode of fig. 10 and 11 can be referred to. The images obtained by the two modules are respectively subjected to image stitching through a computer to obtain a visible light large-view-field stitched image and an infrared large-view-field stitched image, the visible light stitched image and the infrared stitched image are registered, an adaptive weighting model is added in front of a fusion algorithm, weights of infrared information and visible light information in fusion can be set according to environmental states, and semantic segmentation, target detection or target tracking visual tasks are carried out on the fused images.
S5, when scene target positioning is required to be achieved, under the three environments, the two electric push rods respectively drive the two visible light cores 2-1 to rotate inwards to be parallel to two optical axes independently or with the infrared core 3-1, the visible light binocular camera obtains underground personnel and equipment distance information, the infrared module uses high-spectrum resolution thermal signals to obtain rich texture information, and the infrared binocular camera obtains underground personnel and equipment distance information. After the visible light image and the infrared image are spliced and registered, the weight occupied by fusion of the visible light image and the infrared image is adaptively set according to the environment state, and underground personnel and equipment are positioned and tracked on the fused image.
The embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present invention, and the present invention is defined in the claims.
Claims (7)
1. The double-spectrum fusion visual perception method is based on a double-spectrum fusion visual perception system based on a crow's-tail compound eye layout and is characterized by comprising an explosion-proof shell (1), a visible light machine core module (2), an infrared machine core module (3), an electric control device (4) and a dust removal device (5), wherein the explosion-proof shell (1) comprises a shell (1-1) and an explosion-proof lens, the middle of the front end face of the shell (1-1) is outwards protruded, and the lower part of the front end face is inwards retracted; two sides of the middle line of the front end face of the shell (1-1) are symmetrically provided with windows, two of each side are arranged up and down; the anti-explosion lens comprises two visible light anti-explosion lenses (1-2) and two infrared light anti-explosion lenses (1-3), wherein the visible light anti-explosion lenses (1-2) are fixed on the two upper windows, and the infrared light anti-explosion lenses (1-3) are fixed on the two lower windows;
the visible light machine core module (2), the infrared machine core module (3) and the electric control device (4) are positioned in the shell (1-1), and the dust removing device (5) is arranged on the front end face of the shell (1-1); the two visible light machine core modules (2) are symmetrically arranged at the upper part in the shell (1-1), each visible light machine core module (2-1), a fixed seat I (2-2) and a fixed plate I (2-3), the fixed plates I (2-3) are horizontally fixed in the shell (1-1), the fixed seats I (2-2) are rotatably connected with the fixed plates I (2-3), and the visible light machine cores (2-1) are arranged in the fixed seats I (2-2); the lens of the visible light machine core (2-1) faces the visible light explosion-proof lens (1-2);
the infrared machine core modules (3) are symmetrically arranged at the lower part in the shell (1-1), each infrared machine core module comprises an infrared machine core (3-1), a fixed base II (3-2) and a fixed flat plate II (3-3), the fixed flat plate II (3-3) is horizontally fixed in the shell (1-1), the fixed base II (3-2) is rotatably connected with the fixed flat plate II (3-3), and the infrared machine cores (3-1) are arranged in the fixed base II (3-2); the lens of the infrared movement (3-1) faces the infrared explosion-proof lens (1-3);
the electric control device (4) comprises an electric push rod I (4-1) and an electric push rod II (4-2) which are arranged in the shell (1-1), the electric push rod I (4-1) is positioned between the two visible light machine core modules (2), the front end of the electric push rod I (4-1) is connected with two long connecting rods (4-1-1), and the head of each long connecting rod (4-1-1) is hinged with one fixing seat I (2-2) respectively; the electric push rod II (4-2) is positioned between the two infrared movement modules (3), the front end of the electric push rod II (4-2) is symmetrically connected with two short connecting rods (4-2-1), and each short connecting rod (4-2-1) is hinged with one fixed seat II (3-2) respectively;
the dust removing device (5) is used for removing dust from the explosion-proof lens, the dust removing device (5) comprises a dust removing motor (5-1), a coupler (5-2), a dust removing brush mounting block (5-3) and a boneless dust removing brush (5-4), the dust removing motor (5-1) is mounted in the shell (1-1), the coupler (5-2) is in shaft connection with the dust removing motor (5-1), the dust removing brush mounting block (5-3) is mounted at the end part of the coupler (5-2), and the boneless dust removing brush (5-4) is fixedly connected with the dust removing brush mounting block (5-3);
the double-spectrum fusion visual perception method comprises the following steps:
s1, detecting the dust concentration and illumination condition of the site environment through a sensor;
s2, under the environment with good illumination condition and no dust, the visible light machine core module (2) works, the two visible light machine cores (2-1) are driven to horizontally rotate by the electric push rod I (4-1), lenses of the visible light machine cores (2-1) are parallel to forward or simultaneously rotate outwards by a certain angle, and an underground field image is acquired through the visible light explosion-proof lens (1-2);
s3, under the low-illumination and high-dust environment, the infrared machine core module (3) works, the two infrared machine cores (3-1) are driven to horizontally rotate by the electric push rod II (4-2), lenses of the two infrared machine cores (3-1) are parallel to forward or simultaneously rotate outwards by a certain angle, and an underground field image is acquired through the infrared explosion-proof lens (1-3);
s4, under the environment of good illumination conditions and large smoke and dust, the visible light machine core module (2) and the infrared machine core module (3) work simultaneously, and the two visible light machine cores (2-1) and the two infrared machine cores (3-1) rotate outwards for a certain angle simultaneously, so that the imaging range is enlarged, and a field image is acquired; the image is transmitted back to the computer for processing, and the image processing method specifically comprises the following steps:
(1) Respectively performing image stitching on the obtained visible light image and infrared light image through a computer to obtain a visible light large-view-field stitched image and an infrared large-view-field stitched image;
(2) Registering the visible light large view field spliced image and the infrared large view field spliced image, adding a self-adaptive weighting model through a fusion algorithm, setting weight of infrared information and visible light information during fusion according to an environment state, and carrying out semantic segmentation, target detection or target tracking visual tasks on the fused image.
2. The dual spectrum fusion visual perception method according to claim 1, wherein the lower parts of the left and right end surfaces of the shell (1-1) are additively retracted; the upper end face of the shell (1-1) is provided with an upper cover (1-4), the rear end face is provided with a cable interface (1-5), and the bottom face is provided with a bracket connecting piece (1-6).
3. The double-spectrum fusion visual perception method according to claim 1, wherein right-angle limiting blocks (2-4) are arranged at four corners of the fixed seat I (2-2); the fixing seat I (2-2) comprises a main body plate (2-2-1) and a telescopic plate (2-2-2), wherein the main body plate (2-2-1) is connected with shaft holes at two ends of the telescopic plate (2-2) and the middle of the main body plate is connected with the two shaft holes through a screw (2-2-3) to form a telescopic structure.
4. The dual-spectrum fusion visual perception method according to claim 3, wherein a cushioning soft adhesive patch (2-5) is arranged on the inner side of the right-angle limiting block (2-4).
5. The dual spectrum fusion visual perception method according to claim 1, wherein the electric putter I (4-1) further comprises a fixing bracket (4-1-2), and the fixing bracket (4-1-2) is mounted on the fixing plate I (2-3) for fixing the electric putter I (4-1).
6. The double-spectrum fusion visual perception method according to claim 1, wherein a rotating shaft and a limiting rod are arranged at the bottom of the fixed seat I (2-2), a rotating shaft hole and an arc-shaped notch are arranged on the fixed plate I (2-3), the rotating shaft is arranged in the rotating shaft hole, and the limiting rod is arranged in the arc-shaped notch; a connecting ring (2-6) is arranged on the fixed seat I (2-2), and the front end of the long connecting rod (4-1-1) is hinged with the connecting ring (2-6); the electric push rod I (4-1) drives the fixed seat I (2-2) to rotate on the fixed flat plate I (2-3) by a certain angle through expansion and contraction, and the limiting rod and the arc notch are used for limiting the rotation angle.
7. The method of claim 1, wherein, in performing in-situ targeting,
(1) The two visible light machine cores (2-1) rotate until the two optical axes are parallel to obtain distance information of underground personnel and equipment;
(2) The two visible light machine cores (2-1) and the two infrared machine cores (3-1) rotate simultaneously until two optical axes are parallel, after the visible light images and the infrared images are spliced and registered, the weight occupied by the fusion of the visible light images and the infrared images is adaptively set according to the environment state, and underground personnel and equipment are positioned and tracked on the fused images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311444601.7A CN117177027B (en) | 2023-11-02 | 2023-11-02 | Double-spectrum fusion visual perception system and method based on crow's eyes layout |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311444601.7A CN117177027B (en) | 2023-11-02 | 2023-11-02 | Double-spectrum fusion visual perception system and method based on crow's eyes layout |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117177027A CN117177027A (en) | 2023-12-05 |
| CN117177027B true CN117177027B (en) | 2024-01-30 |
Family
ID=88937891
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311444601.7A Active CN117177027B (en) | 2023-11-02 | 2023-11-02 | Double-spectrum fusion visual perception system and method based on crow's eyes layout |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117177027B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112991250A (en) * | 2021-04-19 | 2021-06-18 | 吉林大学 | Infrared and visible light image fusion method based on sonodon acutus visual imaging |
| CN213479542U (en) * | 2020-09-10 | 2021-06-18 | 深圳市博铭维智能科技有限公司 | Double-lens holder detection equipment |
| CN216527253U (en) * | 2021-12-28 | 2022-05-13 | 郑州信大先进技术研究院 | Visible light and far infrared image fusion device |
| CN115526897A (en) * | 2022-09-20 | 2022-12-27 | 国网安徽省电力有限公司电力科学研究院 | Flame positioning method and system for fire-fighting robot of extra-high voltage converter station |
-
2023
- 2023-11-02 CN CN202311444601.7A patent/CN117177027B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN213479542U (en) * | 2020-09-10 | 2021-06-18 | 深圳市博铭维智能科技有限公司 | Double-lens holder detection equipment |
| CN112991250A (en) * | 2021-04-19 | 2021-06-18 | 吉林大学 | Infrared and visible light image fusion method based on sonodon acutus visual imaging |
| CN216527253U (en) * | 2021-12-28 | 2022-05-13 | 郑州信大先进技术研究院 | Visible light and far infrared image fusion device |
| CN115526897A (en) * | 2022-09-20 | 2022-12-27 | 国网安徽省电力有限公司电力科学研究院 | Flame positioning method and system for fire-fighting robot of extra-high voltage converter station |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117177027A (en) | 2023-12-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110244314A (en) | One kind " low slow small " target acquisition identifying system and method | |
| CN110514126A (en) | A method of the tunnel tunnel face displacement monitoring based on machine vision | |
| CN108897342B (en) | Positioning and tracking method and system for fast-moving civil multi-rotor unmanned aerial vehicle | |
| CN206409819U (en) | A kind of pipe video detects robot | |
| CN107064172A (en) | A kind of Tunnel Lining Cracks rapid detection system | |
| CN209441584U (en) | A kind of Multi-purpose underwater robot | |
| CN111898563A (en) | Comprehensive safety monitoring equipment and method for protected area | |
| CN209319821U (en) | A kind of converter station protection room crusing robot | |
| CN117177027B (en) | Double-spectrum fusion visual perception system and method based on crow's eyes layout | |
| CN207593836U (en) | For the robot of Underwater Welding | |
| CN210268541U (en) | Tunnel face displacement monitoring device based on machine vision | |
| CN110954003A (en) | Shield tail clearance measurement system based on high resolution camera image acquisition and processing | |
| CN212251704U (en) | Intelligent robot for drainage pipeline detection | |
| CN111930131B (en) | Device and application method for obtaining graph suitable for complex environment | |
| CN116898172B (en) | Helmet device for detecting complex operation scene and application method | |
| CN211491547U (en) | Electric power inspection robot | |
| CN113418107A (en) | Panoramic real-time monitoring device | |
| CN109696085A (en) | A kind of intelligence removal of mines robot | |
| CN110007672A (en) | Unmanned survey ship suitable for shield constructs and passes through river construction monitoring | |
| CN221340845U (en) | Image acquisition device for engineering supervision | |
| CN215968708U (en) | Intelligent robot for power plant inspection | |
| CN220456986U (en) | Substation inspection device | |
| CN215222334U (en) | Device capable of self-cleaning camera lens | |
| CN215000808U (en) | Damaged retractable shooting device of rearmounted road of road detection car | |
| CN221010225U (en) | Camera convenient to adjust |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |