KR102302907B1 - Stereo awareness apparatus, and method for generating disparity map in the stereo awareness apparatus - Google Patents

Abstract

According to one embodiment of the present invention, provided are a stereo context recognition device and a method for generating disparity map data in the stereo context recognition device. A front part of a combat vehicle includes the stereo context recognition device. The stereo context recognition device includes a plurality of first camera modules respectively disposed at the lower left and lower right sides of the stereo context recognition device; a plurality of second camera modules disposed on the top of the first camera modules; and a third camera module disposed in an upper center between the second camera modules. The combat vehicle is one of a tank, an armored vehicle, a self-propelled gun, and a small tactical vehicle.

Description

Stereo awareness apparatus, and method for generating disparity map in the stereo awareness apparatus

The present invention relates to a stereo context recognition apparatus, and more particularly, to a structure of the stereo context recognition apparatus and a method of generating a disparity map in the stereo context recognition apparatus.

In modern land warfare, it is important that combat vehicles such as tanks, armored vehicles, and self-propelled artillery are aware of the surrounding conditions in the battle area to determine whether the battle is won or lost. Such a surrounding situation is recognized through a stereo situation recognition device.

Such a stereo situational recognition device includes a stereo camera including two cameras based on a single visible light sensor. Using the stereo camera, a disparity map is generated, and distance information about surrounding objects is extracted based on the disparity map. do.

However, although such a stereo situational awareness device can be operated limitedly during the day, there is a problem that it cannot be used in haze or foe, smoke, sand dust, fine dust, and at night. That is, the stereo context recognition apparatus has environmental limitations in the case of complex operation with or without remote presence. In addition, stereo cameras including two cameras based on a single infrared sensor of a stereo situational awareness device do not have a clear output image from the infrared sensor. There was a problem.

Therefore, the need for a method for solving these problems has emerged.

An embodiment of the present invention proposes a stereo context recognition apparatus that is not subject to environmental restrictions.

In addition, an embodiment of the present invention proposes a stereo situation recognition apparatus in which an error does not occur when generating a disparity map.

According to an embodiment of the present invention, in a stereo situation recognition device, a front part of a combat vehicle includes a stereo situation recognition device, wherein the stereo situation recognition device is disposed at the lower left and lower right sides of the stereo situation recognition device, respectively a plurality of first camera modules; a plurality of second camera modules disposed on top of the first camera modules; and a third camera module disposed at an upper center between the second camera modules, wherein the combat vehicle is one of a tank, an armored vehicle, a self-propelled artillery, and a small tactical vehicle.

According to an embodiment of the present invention, in a method for generating disparity map data in a stereo context recognition apparatus, a signal processing unit of the stereo context recognition apparatus applies a corner point or a contour filter to first image data of a first camera module to extracting a plurality of feature points; extracting, by the signal processing unit, a plurality of feature points by applying a corner point or a contour filter to second image data of a plurality of second camera modules; and comparing, by the signal processing unit, a plurality of feature points of the first camera module with a plurality of feature points of the second camera modules to determine the most similar feature point, and generating parallax map data by matching the most similar feature point including, wherein the stereo situation recognition device is provided in a combat vehicle, and a plurality of third camera modules respectively disposed on the lower left and right bottom sides of the stereo situation recognition device, and the first camera modules disposed on top of the a second camera module, and the first camera module disposed at the upper center between the second camera modules, wherein the combat vehicle is one of a tank, an armored vehicle, a self-propelled artillery, and a small tactical vehicle .

An embodiment of the present invention may provide a stereo context recognition apparatus that is not subject to environmental restrictions.

In addition, an embodiment of the present invention may provide a stereo context recognition apparatus in which an error does not occur when generating a disparity map.

1 is a block diagram of a combat vehicle according to an embodiment of the present invention.
2 is an external configuration diagram of an apparatus for recognizing a stereo situation according to an embodiment of the present invention.
3 is an internal configuration diagram of an apparatus for recognizing a stereo situation according to a first embodiment of the present invention.
4 is an internal configuration diagram of an apparatus for recognizing a stereo situation according to a second embodiment of the present invention.
5 is a diagram illustrating a horizontal angle of a stereo situation recognition apparatus according to an embodiment of the present invention.
6 is a block diagram of a stepeo situation recognition apparatus according to an embodiment of the present invention.
7 is a diagram illustrating a procedure for generating a disparity map in the apparatus for recognizing a stereo situation according to the first embodiment of the present invention.
8 is a diagram illustrating a procedure for generating a disparity map in a stereo situation recognition apparatus according to a second embodiment of the present invention.
9 is a diagram illustrating a procedure for generating a disparity map in a stereo situation recognition apparatus according to a third embodiment of the present invention.
10 is a diagram illustrating a procedure for comparing feature points in a stereo context recognition apparatus according to an embodiment of the present invention.
11 is a flowchart of generating a disparity map in the stereo context recognition apparatus according to an embodiment of the present invention.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

Terms used in the embodiments of the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. . In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

Embodiments of the present invention can apply various transformations and can have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiments, and it should be understood to include all transformations, equivalents and substitutes included in the spirit and scope of the invention. In describing the embodiments, if it is determined that a detailed description of a related known technology may obscure the subject matter, the detailed description thereof will be omitted.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It is to be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In an embodiment of the present invention, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' are integrated into at least one module except for 'modules' or 'units' that need to be implemented with specific hardware and are implemented with at least one processor (not shown). can be

In an embodiment of the present invention, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" with another element interposed therebetween. also includes In addition, when a part "includes" a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

When the remote unmanned combat vehicle performs a mission, the operator inputs a driving route to the autonomous driving processing unit of the remote unmanned combat vehicle in advance. In addition, the remote or unmanned combat vehicle moves on a certain route by autonomous driving or is driven by the operator's remote control. For example, combat vehicles may include tanks, armored vehicles, self-propelled artillery, small tactical vehicles, and the like.

When the remote unmanned combat vehicle starts driving, the stereo situation recognition device of the remote unattended or unattended combat vehicle transmits a stereo image to the control device through the communication unit. The control unit determines the target of the remote unmanned combat vehicle according to intelligent situation monitoring such as automatic detection/tracking and recognition of moving obstacles within the operating area in which it is currently traveling, and transmits the determined target to the signal and control processing unit. And the signal and control processing unit controls the stereo context recognition device. For example, the control device may include a control unit for remote control of an unmanned combat vehicle, an image display unit for displaying images, a control processing unit, and a communication unit.

The combat vehicle according to the present invention is equipped with a stereo situational recognition device. For example, the vehicle 101 may mount the stereo situation recognition device 103 on the front end of the vehicle 101 as shown in FIG. 1 .

2 is an external configuration diagram of the stereo context recognition apparatus 103 according to an embodiment of the present invention.

Referring to FIG. 2 , the stereo context recognition apparatus 201 includes a plurality of camera modules. For example, as shown in FIG. 3 , the stereo context recognition apparatus 201 may include a near-infrared (SWIR) camera module, two IR camera modules, and two TV camera modules. For example, the stereo context recognition device 201 arranges a first TV camera module 305 and a first IR camera module 303 on the left side, and a second TV camera module 309 and a second IR camera module 307 on the right side. , and the near-infrared camera module 301 may be disposed on top of the first and second IR camera modules 303 and 307 . For example, the first IR camera module 303 may be disposed on the top of the first IR camera module 305 , and the second IR camera module 307 may be disposed on the top of the second IR camera module 309 .

As another example, the stereo context recognition device 201 may include two IR camera modules, an IR-LED, and three boards, as shown in FIG. 4 . For example, the stereo context recognition device 201 arranges the first IR-LED and the board 405 and the first IR camera module 403 on the left side, and the second IR-LED and the board 409 and the first IR camera on the right side. The module 407 may be disposed, and the third IR-LED and the board 401 may be disposed on top of the first and second IR-LEDs and the boards 403 and 407 . For example, the first IR-LED and the board 403 may be disposed on the top of the first IR camera module 405 , and the second IR-LED and the board 407 may be disposed on the top of the second IR camera module 409 . have.

5 is a view showing a horizontal angle of the stereo context recognition apparatus 103 according to an embodiment of the present invention.

Referring to FIG. 5 , the stereo context recognition device 501 may include two IR cameras, two TV camera modules, and one near-infrared camera module. Two IR camera modules with a viewing angle of 60 degrees recognize a 120-degree horizontal angle 503, two TV camera modules with a viewing angle of 60 degrees recognize a 120-degree horizontal angle 503, and a near-infrared camera module with a 60-degree viewing angle recognizes a 60-degree horizontal angle.

Since the near-infrared camera module has a viewing angle of 60 degrees with respect to the center, it is necessary to capture the same area as the horizontal angle 501 of the near-infrared camera module based on the center in the composite image of two IR camera modules taken through a 120-degree horizontal angle 503. do. In addition, the near-infrared camera module needs to capture the same area as the horizontal angle 501 of the near-infrared camera module based on the center in the composite image of two TV camera modules photographed through a 120-degree horizontal angle 503 .

6 is a block diagram of a stereo context recognition apparatus 103 according to an embodiment of the present invention.

Referring to FIG. 6 , the stereo context recognition apparatus 103 includes a signal processing unit 601 , a memory 603 , a communication unit 605 , and a plurality of camera modules 607 .

Looking at each component, the memory 603 stores various programs and data necessary for the operation of the stereo context recognition device 103 . For example, the memory unit 603 may be implemented as a non-volatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD). For example, the memory 603 may store a plurality of image data received from the plurality of camera modules 607 or store parallax map data.

The communication unit 605 performs communication with various types of external devices according to various types of communication methods. For example, the communication unit 605 may transmit the disparity map data to the control device.

Each of the plurality of camera modules 607 generates image data by photographing the surrounding situation of the combat vehicle 101 , and outputs the generated image data to the signal processing unit 601 . For example, the plurality of camera modules 607 may include two IR cameras, two TV cameras, and one near-infrared camera. As another example, the plurality of camera modules 607 may be configured with three IR-LEDs and a board and two TV cameras.

The signal processing unit 601 controls the overall operation of the stereo context recognition apparatus 103 using various programs stored in the memory 603 . For example, the signal processing unit 601 may be an IR or TV camera control board.

For example, the signal processing unit 601 may generate disparity map data using a plurality of image data input from the plurality of camera modules 607 . For example, the disparity map may represent different differences in the position of an object in two images (eg, an upper image and a lower image) for stereo registration.

In more detail, the signal processing unit 601 is configured to generate disparity map data of the near-infrared camera module based on the TV camera module according to the disparity map generation procedure 701 shown in FIG. 7 , from the TV camera module. A plurality of feature points may be extracted by applying a corner point or a contour filter to the input image data. In addition, the signal processing unit 601 may extract a plurality of feature points by applying a corner point or a contour filter to the image data input from the near-infrared camera module.

In addition, the signal processing unit 601 may generate parallax map data using a plurality of feature points of the TV camera module and a plurality of feature points of the near-infrared camera module.

For example, the signal processing unit 601 may determine the most similar feature point by comparing a plurality of feature points of the TV camera module and a plurality of feature points of the near-infrared camera module from the lower left to the upper right of the image data of the TV camera module. This is because the TV camera module and the near-infrared camera module are not arranged horizontally, but vertically. In addition, the signal processing unit 601 may generate disparity map data by matching the most similar feature points.

In this case, the signal processing unit 601 finds and matches the most similar feature points using various methods. For example, various methods include a sum absolute difference value (Sum of Absolute Difference, hereinafter referred to as 'SAD'), a mean absolute difference value (hereinafter referred to as 'MAD'), and a normalized correlation value (Normalized Correlation). , hereinafter referred to as 'NC'), a normalized cross correlation value (hereinafter referred to as 'NCC'), and the like.

For example, the signal processing unit 601 may compare the plurality of feature points of the TV camera module with the plurality of feature points of the near-infrared camera module by the MAD method using Equation 1 as follows.

Here, MAD(A _TV , B _SWIR ) represents a value extracted between the image data of the TV camera module A _TV and the image data of the near-infrared camera module B _{SWIR using the MAD method.} And i and j represent pixel index values of image data.

At this time, the signal processor 601, MAD way between 10 the 3x3 block of image data 1001 of the TV camera module shown in (a ₁₎ and the 3x3 block (b ₁₎ of the image data 1003 of the near-infrared camera module Equation 1 is applied to extract a value between the two image data 1001 and 1003 . Next, the signal processing unit 601 moves over one pixel of the image data 1001 and 1003) to a 3x3 block (a ₂ ) of the image data of the TV camera module and a 3x3 block (b) of the image data of the near-infrared camera module ₂ ) The value between the two images is extracted using the MAD method. Next, the signal processing unit 601 extracts values by repeating these operations up to the upper block of the image data 1001 and 1003 . Thereafter, the signal processing unit 601 moves to the right of one pixel of the image data 1001 and 1003 and repeats the previous process. The signal processing unit 601 repeats this process until a value between _{the 3x3 block (a n} ) of the image data 1001 of the TV camera module _{and the 3x3 block (b n} ) of the image data 1003 of the near-infrared camera module is extracted. carry out

And the signal processing unit 601 selects the smallest value (the most similar value) among the extracted values, and the pixel of the image data 1001 of the TV camera module corresponding to the selected smallest value and the image data 1003 of the near-infrared camera module ) to create a disparity map.

In addition, the signal processing unit 601 checks whether matching is well performed based on the feature points of the two image data. In this case, the signal processing unit 601 uses normalized mutual information as a cost function for checking whether matching is well performed. For example, the signal processing unit 601 can measure mutual information normalized for the image data of _{the TV camera module (A TV} ) and the image data of the near-infrared camera module (B _{SWIR) to be matched using Equation 2} have.

Here, I _N represents a value output based on normalized mutual information, and N represents the number of frames. A _TV indicates image data of the TV camera module, and B _SWIR indicates image data of the near-infrared camera module. H(A _TV ) is determined using Equation 3, H(B _SWIR ) is determined using Equation 4, and H(A _TV , B _SWIR ) is determined using Equation 5.

Here, P denotes a probability distribution of pixel brightness values of image data.

On the other hand, the signal processing unit 601, according to the disparity map generation procedure 801 shown in FIG. 8, to generate disparity map data of the IR camera module based on the TV camera module, image data input from the TV camera module A plurality of feature points may be extracted by applying a corner point or a contour filter to . In addition, the signal processing unit 601 may extract a plurality of feature points by applying a corner point or a contour filter to the image data input from the IR camera module.

In addition, the signal processing unit 601 may generate disparity map data using a plurality of feature points of the TV camera module and a plurality of feature points of the IR camera module.

For example, the signal processing unit 601 may determine the most similar feature point by comparing a plurality of feature points of the TV camera module and a plurality of feature points of the IR camera module from the lower left to the upper right of the image data of the TV camera module. This is because the TV camera module and the IR camera module are not arranged horizontally, but vertically. In addition, the signal processing unit 601 may generate disparity map data by matching the most similar feature points.

In addition, the signal processing unit 601, according to the disparity map generation procedure 901 shown in FIG. 9, to generate the disparity map data of the IR camera module based on the near-infrared camera module, the image data input from the near-infrared camera module A plurality of feature points may be extracted by applying a corner point or a contour filter to . In addition, the signal processing unit 601 may extract a plurality of feature points by applying a corner point or a contour filter to the image data input from the IR camera module.

In addition, the signal processing unit 601 may generate disparity map data using a plurality of feature points of the near-infrared camera module and a plurality of feature points of the IR camera module.

For example, the signal processing unit 601 may determine the most similar feature point by comparing a plurality of feature points of the near-infrared camera module and a plurality of feature points of the IR camera module from the lower left to the upper right of the image data of the near-infrared camera module. This is because the near-infrared camera module and the IR camera module are not arranged horizontally, but vertically. In addition, the signal processing unit 601 may generate disparity map data by matching the most similar feature points.

11 is a flowchart of generating a disparity map in the stereo context recognition apparatus 103 according to an embodiment of the present invention.

Referring to FIG. 11 , the signal processing unit 601 of the stereo context recognition apparatus 103 extracts a plurality of feature points by applying a corner point or a contour filter to image data of the TV camera module in step 1101 .

In step 1103 , the signal processing unit 601 extracts a plurality of feature points by applying a corner point or a contour filter to the image data input from the near-infrared camera module.

In operation 1105 , the signal processing unit 601 generates parallax map data using a plurality of feature points of the TV camera module and a plurality of feature points of the near-infrared camera module.

For example, the signal processing unit 601 may determine the most similar feature point by comparing a plurality of feature points of the TV camera module and a plurality of feature points of the near-infrared camera module from the lower left to the upper right of the image data of the TV camera module. This is because the TV camera module and the near-infrared camera module are not arranged horizontally, but vertically. In addition, the signal processing unit 601 may generate disparity map data by matching the most similar feature points.

An embodiment of the present invention proposes a stereo situational recognition device including a camera module based on heterogeneous visible/infrared sensors and a near-infrared camera module in a remote wired/wireless combat vehicle. In addition, an embodiment of the present invention proposes a method of generating a disparity map for extracting distance information using a stereo context recognition apparatus. For example, the stereo context recognition device has a structure in which a plurality of camera modules based on visible light or infrared sensors are stacked and fastened in a block manner.

According to an embodiment of the present invention, a disparity map is generated using a stereo situation recognition device including three types of camera modules, and distance information is extracted based on the disparity map. Therefore, the apparatus for recognizing a stereo situation according to an embodiment of the present invention can be operated regardless of day/night. In particular, the stereo contextual recognition apparatus may generate a more accurate disparity map than the stereo contextual recognition apparatus including two TV cameras in fog, smoke, sand dust, fine dust, and at night. That is, the apparatus for recognizing a stereo situation according to an embodiment of the present invention is less affected by the environment.

In addition, in an embodiment of the present invention, since the stereo context recognition device includes three infrared sensor-based camera modules (for example, one near-infrared camera module and two IR camera modules), the output image of the infrared sensor is clear. Therefore, there is no error when generating the parallax map.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and it is common in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Various modifications may be made by those having the knowledge of, of course, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

For example, in the present invention, it has been described that a stereo situation recognition device including a near-infrared camera, two IR camera modules, and two TV camera modules generates parallax map data, but the present invention is not limited thereto. For example, the method of generating disparity map data in the present invention can be sufficiently applied to a stereo situation recognition device including two IR camera modules, an IR-LED, and three boards.

101: tank 103: stereo situational awareness device
201: stereo situational recognition device 301: near-infrared camera module
303, 307: IR camera module 305, 309: TV camera module
401, 405, 409: IR-LED and board 403, 407: IR camera module
501: stereo situational awareness device
503: Horizontal angle of IR camera module and TV camera module
505: horizontal angle of the near-infrared camera module
601: signal processing unit 603: memory
605: communication unit 607: a plurality of camera modules

Claims (6)

The front part of the combat vehicle is equipped with a stereo situational awareness device,
The stereo situation recognition device,
a plurality of first camera modules disposed at the lower left and lower right sides of the stereo context recognition device, respectively, to photograph a first area;
a plurality of second camera modules disposed on top of the first camera modules to photograph the same first area as the first camera modules;
a third camera module disposed at an upper center between the first camera modules and the second camera modules to photograph a second area that is a partial area within the first area; and
A corner point or a contour filter is applied to the first image data of the third camera module to extract a plurality of feature points, and a corner point or a contour filter is applied to the second image data of the second camera modules to extract a plurality of feature points and sequentially comparing a plurality of feature points of the third camera module and a plurality of feature points of the second camera modules from the lower left to the upper right of the first image data and the second image data to determine the most similar feature point and a signal processing unit for generating disparity map data by matching the first image data and the second image data based on the most similar feature points,
The combat vehicle is a stereo situation recognition device, characterized in that one of a tank, an armored vehicle, a self-propelled artillery, and a small tactical vehicle.
According to claim 1,
The first camera modules are TV camera modules,
The second camera modules are IR camera modules,
The third camera module is a stereo situation recognition device, characterized in that the near-infrared camera module.
According to claim 1,
The first camera modules and the third camera module are IR-LEDs and boards,
The second camera module is a stereo context recognition device, characterized in that the IR camera module.
A process of extracting, by a signal processing unit of the stereo situation recognition apparatus, a plurality of feature points by applying a corner point or a contour filter to the first image data of the first camera module;
extracting, by the signal processing unit, a plurality of feature points by applying a corner point or a contour filter to second image data of a plurality of second camera modules;
The signal processing unit sequentially compares the plurality of feature points of the first camera module and the plurality of feature points of the second camera modules from the lower left to the upper right of the first image data and the second image data to be the most similar the process of determining the feature point; and
and generating disparity map data by matching the first image data and the second image data based on the most similar feature points,
The stereo situation recognition device is provided on the front part of the combat vehicle, and is disposed at the lower left and lower right sides of the stereo situation recognition device, respectively, a plurality of third camera modules for photographing the first area, the first camera module The second camera modules for photographing the same first area as the third camera modules disposed on the top of the It includes the first camera module for photographing a second area that is a partial area within the first area,
The method for generating parallax map data in a stereo situational recognition device, characterized in that the combat vehicle is one of a tank, an armored vehicle, a self-propelled artillery, and a small tactical vehicle.
5. The method of claim 4,
The third camera modules are TV camera modules,
The second camera modules are IR camera modules,
The first camera module is a method for generating parallax map data in a stereo situation recognition device, characterized in that the near-infrared camera module.
6. The method of claim 5,
The third camera modules and the first camera module are IR-LEDs and boards,
The method for generating parallax map data in a stereo situation recognition device, characterized in that the second camera modules are IR camera modules.

Images