KR102339825B1 - Device for situation awareness and method for stitching image thereof - Google Patents

Abstract

The problem to be solved by the present invention is to provide a stitching-based device for situation recognition and a method for stitching an image thereof capable of generating a panoramic image reflecting information from an observer's viewpoint in stitching images taken by a plurality of cameras having different viewing angles. The method comprises: a plurality of modular cameras based on an infrared or visible light sensor; an integrated control unit that controls an image period for stitching on an overlapping area of each image captured by the plurality of modular cameras, enlarges or reduces an image to correspond to an observer viewpoint selected by a user from among images taken by the plurality of modular cameras, and generates a panoramic image by stitching the enlarged or reduced image; and a display unit for outputting the generated panoramic image.

Description

Context recognition device and its image stitching method

The present invention relates to a situation recognition apparatus and an image stitching method thereof.

In the prior art, in a visible light sensor (CCD, CMOS) or infrared sensor (thermal)-based camera image, a 360-degree monitoring device, a remote monitoring device, and a wired/wireless communication device are set in the same resource to detect an object on the ground. By setting the resolution of the field of view (FOV) from the beginning, a digital pan operation and a tilt operation and a zoom operation of a specific viewing angle were performed. This is performed without change until a separate mission is completed, and a method of adjusting a part of the resolution compression ratio according to the communication environment is applied.

In this prior art, a plurality of cameras use a fixed viewing angle, and accordingly, in the case of a 360-degree panoramic image centered on a fixed viewing angle, digital pan & tilt and zoom of a specific viewing angle are possible, but There was a disadvantage that the image was not generated.

In addition, even when a 360-degree panoramic image with different viewing angles is generated, the observer's viewpoint is not considered in the same way, so the image resolution is set to the same (maximum, minimum, intermediate, etc.) for different viewing angles, and then the panoramic image is generated. have to create That is, in this case, there is a problem in that the process of outputting the image by adjusting it to the same image resolution is necessarily required.

However, the problem to be solved by the present invention and the problem to be solved by the present invention is that in stitching images taken by a plurality of cameras including different viewing angles, a panoramic image reflecting the observer's viewpoint information can be generated. A recognition apparatus and an image stitching method thereof are provided.

However, the problems to be solved by the present invention are not limited to the problems described above, and other problems may exist.

An image stitching-based context recognition apparatus according to a first aspect of the present invention for solving the above-mentioned problems is a plurality of modular cameras based on an infrared or visible light sensor, and an overlapping area of each image captured by the plurality of modular cameras. Controls the image period for stitching, enlarges or reduces the image to correspond to the observer viewpoint selected by the user among each image captured by the plurality of modular cameras, and stitches the enlarged or reduced image to produce a panoramic image It includes an integrated control unit for generating and a display unit for outputting the generated panoramic image.

In some embodiments of the present invention, the integrated control unit may extract key points from each of a plurality of images, and match the extracted key points of the enlarged or reduced image based on the image corresponding to the selected observer's viewpoint.

In some embodiments of the present invention, the plurality of modular cameras are composed of three two-stage modular cameras, and the image corresponding to the first modular camera having the largest viewing angle among the plurality of modular cameras is an observer's point of view. is selected, the integrated control unit may reduce the image of the remaining second modular cameras except for the first modular camera, and match the extracted feature points of the reduced image.

In some embodiments of the present invention, the plurality of modular cameras is composed of three two-stage modular cameras, and the remaining second modular cameras except for the first modular camera having the largest viewing angle among the plurality of modular cameras When the image corresponding to the camera is selected as the observer's viewpoint, the integrated control unit may magnify the image of the first modular camera, and match the extracted feature points of the magnified image.

In some embodiments of the present invention, the first modular camera is a pair of modular cameras with a viewing angle of 95 degrees and an overlap area of 6.5 degrees, and the second modular cameras each have a viewing angle of 54 degrees and an overlap of 0.5 degrees It may be composed of a pair of modular cameras having an area, and another pair of modular cameras each having a field of view of 50 degrees and an overlapping area of 2.5 degrees.

In some embodiments of the present invention, the integrated control unit may generate a panoramic image having a viewing angle of up to 386 degrees by adjusting the overlapping area of the images captured by the plurality of modular cameras.

In some embodiments of the present invention, the plurality of modular cameras may be configured to have different viewing angles, respectively.

Some embodiments of the present invention may further include a 3-axis gyro sensor attached to the lower end of the plurality of modular cameras to perform digital pan and tilt operations of the modular cameras.

In addition, the image stitching method in the context recognition device according to the second aspect of the present invention comprises the steps of: receiving an image photographed through a plurality of modular cameras based on infrared or visible light sensors; extracting feature points from each of the captured images; receiving an observer's viewpoint selection from among the captured images; enlarging or reducing the image to correspond to the selected observer's viewpoint; and generating a panoramic image by stitching the feature points of the enlarged or reduced image to match.

In some embodiments of the present invention, when an image corresponding to the first modular camera having the largest viewing angle among the plurality of modular cameras is selected as the observer's viewpoint, the image is enlarged or reduced to correspond to the selected observer's viewpoint In the step of reducing the image of the remaining second modular camera excluding the first modular camera may be processed.

In some embodiments of the present invention, when an image corresponding to the second modular camera other than the first modular camera having the largest viewing angle among the plurality of modular cameras is selected as the observer viewpoint, it corresponds to the selected observer viewpoint In the step of enlarging or reducing the image so that the image is enlarged, the image of the first modular camera may be enlarged.

In some embodiments of the present invention, the first modular camera is a pair of modular cameras with a viewing angle of 95 degrees and an overlap area of 6.5 degrees, and the second modular cameras each have a viewing angle of 54 degrees and an overlap of 0.5 degrees It may be composed of a pair of modular cameras having an area, and another pair of modular cameras each having a field of view of 50 degrees and an overlapping area of 2.5 degrees.

In some embodiments of the present invention, the step of generating a panoramic image by stitching to match the feature points of the enlarged or reduced image includes adjusting the overlapping area of the images captured by the plurality of modular cameras at a maximum of 386 degrees. A panoramic image having a viewing angle may be generated.

In some embodiments of the present invention, the plurality of modular cameras may be configured to have different viewing angles, respectively.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer, which is hardware, to execute the context recognition device and its image stitching method, and is stored in a computer-readable recording medium.

Other specific details of the invention are included in the detailed description and drawings.

According to the above-described embodiment of the present invention, it is possible to generate a panoramic image centered on the viewpoint of an observer in generating a panoramic image, and through this, the process of adjusting the resolution of each image to be the same can be omitted. have.

In addition, the present invention has the effect of monitoring and observing a wider area as a panoramic image having a viewing angle of 360 degrees or more can be generated by adjusting the overlapping area of each image.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

The accompanying drawings below are provided to help understanding of the present embodiment, and provide embodiments together with detailed description. However, the technical features of the present embodiment are not limited to specific drawings, and features disclosed in each drawing may be combined with each other to constitute a new embodiment.
1 is a block diagram of a context recognition apparatus according to an embodiment of the present invention.
2 is a view showing an example of a monitoring area by a plurality of modular cameras.
3 is a diagram illustrating a block upper portion and a lower block portion of a plurality of modular cameras separately.
4 is a view for explaining a type of viewing angle for a monitoring area in a plurality of modular cameras.
5 is a diagram illustrating an example of the A/B/C-zone monitoring area described in FIG. 4 .
6 and 7 are diagrams illustrating an example of generating a panoramic image from the viewpoint of an observer based on the A/B-zone monitoring area.
8 and 9 are diagrams illustrating an example of generating a panoramic image from the viewpoint of an observer based on the C-zone monitoring area.
10 is a flowchart of an image stitching method according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, an image stitching-based context recognition apparatus 100 (hereinafter referred to as a context recognition apparatus) according to an embodiment of the present invention will be described with reference to FIGS. 1 to 9 .

1 is a block diagram of a context recognition apparatus 100 according to an embodiment of the present invention. 2 is a view showing an example of a monitoring area by a plurality of modular cameras (110). 3 is a view showing the block upper part and the block lower part of the plurality of modular cameras 110 separately. 4 is a view for explaining the types of viewing angles for the monitoring area in the plurality of modular cameras (110).

The context recognition apparatus 100 according to an embodiment of the present invention includes a plurality of modular cameras 110 , an integrated control unit 120 , and a display unit.

The plurality of modular cameras 110 are cameras based on an infrared sensor or a visible light sensor. In one embodiment, the modular camera 110 in the present invention may be mounted on the upper end of the mixed-use vehicle. Through such a modular camera 110, the present invention can detect an object located at a distance of 30 m or more or an intermediate distance.

The integrated control unit 120 controls the image period for stitching the overlapping area of each image captured by the plurality of modular cameras 110 . In addition, the integrated control unit 120 enlarges or reduces an image to correspond to an observer viewpoint selected by a user among each image captured by the plurality of modular cameras 110 , and stitches the enlarged or reduced image to generate a panoramic image do.

The display unit 130 outputs the generated panoramic image.

Meanwhile, in an embodiment of the present invention, the plurality of modular cameras 110 may be three two-stage modular cameras.

2, the plurality of modular cameras 110 having a pair of modular cameras having a monitoring area A-zone, a pair of modular cameras having a monitoring area B-zone and a monitoring area C-zone A total of six can be provided as a pair of modular cameras. There is an overlapping area overlapping each other between the monitoring areas by each pair of the modular cameras 110, and in an embodiment of the present invention, a monitoring area of up to 386 degrees can be secured by controlling the degree of overlap of the overlapping areas.

Referring to FIG. 3 , the plurality of modular cameras 110 may be configured in the form of upper and lower blocks, and the three-axis gyro sensor 140 that is the x, y, and z center axis is mounted at the lower end of the block. can Through the three-axis gyro sensor 140, the operator can recognize the directions of east, west, south, north (azimuth, elevation), and when performing digital pan and tilt operations, zoom in and out of the panoramic image from the observer's point of view.

In one embodiment, three pairs of a plurality of modular cameras 110 may be configured to have different viewing angles, respectively.

4 shows a monitoring zone for each viewing angle of a plurality of modular cameras 110 . As an example, the camera viewing angle of the innermost monitoring area A/F-zone is 54 degrees, respectively, except for the overlapping area of +/- 0.5 degrees, each has a viewing angle of 53 degrees. . Next, the camera viewing angle of the monitoring area B/E-zone located in the middle is 50 degrees, respectively, except for the overlapping area of +/-2.5 degrees, each has a viewing angle of 45 degrees. Finally, the camera viewing angle of the C/D-zone located in the outer part is 95 degrees, respectively, except for the overlapping area of +/- 6.5 degrees, each has a viewing angle of 82 degrees.

FIG. 5 is a diagram illustrating an example of the A/B/C-zone monitoring area described in FIG. 4 , and is a view prepared for setting an overlapping area in a 180 degree image. In FIG. 5, the viewing angles of A-zone and B-zone are 4 degrees different, the size of the infrared sensor is 1024Χ768 for C-zone, 640Χ480 for A/B-zone, and the viewing angle for C-zone is 95 degrees. Therefore, you can see a wider area, but the size of the object is smaller than that of A/B-zone.

As such, when the image for each monitoring area is prepared, the integrated control unit 120 extracts a feature point from each of the plurality of images, and the extracted feature point of the enlarged or reduced image based on the image corresponding to the observer's viewpoint selected by the user. can be matched. Through this, the integrated control unit 120 may generate a panoramic image by stitching the enlarged or reduced image.

6 and 7 are diagrams illustrating an example of generating a panoramic image from the viewpoint of an observer based on the A/B-zone monitoring area.

In one embodiment, when an image corresponding to the second modular camera other than the first modular camera having the largest viewing angle among the plurality of modular cameras 110 is selected as the observer's point of view, the integrated control unit 120 is 1 A panoramic image can be created by magnifying the image of the modular camera and matching the extracted feature points of the magnified image.

6 shows images captured by the first modular camera of the C-zone monitoring area having the largest viewing angle and the second modular camera of the remaining A/B-zone monitoring area, and C-zone monitoring The object included in the image captured by the first modular camera having an area appears smaller in size than the object in the image captured by the second modular camera having the A/B-zone monitoring area.

In this case, when the integrated control unit 120 receives a selection input using the image of the second modular camera corresponding to the A/B-zone as the observer's point of view, it is photographed by the first modular camera having the C-zone monitoring area. The enlarged image is processed so that the feature points can be matched.

Thereafter, when the key point matching process is performed, a panoramic image with the A/B-zone monitoring area as the observer's viewpoint may be generated as shown in FIG. 7 .

8 and 9 are diagrams illustrating an example of generating a panoramic image from the viewpoint of an observer based on the C-zone monitoring area.

In one embodiment, when the image corresponding to the first modular camera having the largest viewing angle among the plurality of modular cameras 110 is selected as the observer's point of view, the integrated control unit 120 controls the second module except for the first modular camera. 2 It is possible to reduce the image of the modular camera and create a panoramic image by matching the extracted feature points of the reduced image.

8 shows images captured by the first modular camera of the C-zone monitoring area having the largest viewing angle and the second modular camera of the remaining A/B-zone monitoring area, and C-zone monitoring The object included in the image captured by the first modular camera having an area appears smaller in size than the object in the image captured by the second modular camera having the A/B-zone monitoring area.

In this case, when the integrated control unit 120 receives a selection input using the image of the first modular camera corresponding to the C-zone as an observer's point of view, it is photographed by the second modular camera having the A/B-zone monitoring area. The reduced image is processed so that feature points can be matched.

Thereafter, when the keypoint matching process is performed, a panoramic image with the C-zone monitoring area as the observer's viewpoint may be generated as shown in FIG. 9 .

Hereinafter, a method performed by the image stitching-based context recognition apparatus 100 according to an embodiment of the present invention will be described with reference to FIG. 10 .

10 is a flowchart of an image stitching method according to an embodiment of the present invention.

In the image stitching method according to an embodiment of the present invention, first, an image photographed through a plurality of modular cameras 110 based on an infrared or visible light sensor is received ( S110 ).

Next, a feature point is extracted from each of the captured images (S120), and a selection of an observer's viewpoint from among the captured images is received (S130).

Next, the image is enlarged or reduced to correspond to the selected observer's viewpoint (S140), and a panoramic image is generated by stitching to match feature points of the enlarged or reduced image (S150).

In addition, an embodiment of the present invention includes a homography processing process for a photographed image, an image warping process to a central region and an immersion region based on the gyro sensor 140 , and a panoramic image synthesis process. An image blending process may be additionally performed.

Meanwhile, in the above description, steps S110 to S150 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between steps may be changed. In addition, the contents of the image stitching-based context recognition apparatus 100 of FIGS. 1 to 9 may also be applied to the image stitching method of FIG. 11 even if other contents are omitted.

The image stitching method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

The above-mentioned program, in order for the computer to read the program and execute the methods implemented as a program, C, C++, JAVA, Ruby, which the processor (CPU) of the computer can read through the device interface of the computer; It may include code coded in a computer language such as machine language. Such code may include functional code related to a function defining functions necessary for executing the methods, etc., and includes an execution procedure related control code necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, the code may further include additional information necessary for the processor of the computer to execute the functions or code related to memory reference for which location (address address) in the internal or external memory of the computer to be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the above functions, the code uses the communication module of the computer to determine how to communicate with any other computer or server remotely. It may further include a communication-related code for whether to communicate and what information or media to transmit and receive during communication.

The storage medium is not a medium that stores data for a short moment, such as a register, a cache, a memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and an optical data storage device. That is, the program may be stored in various recording media on various servers accessible by the computer or in various recording media on the computer of the user. In addition, the medium may be distributed in a computer system connected to a network, and a computer-readable code may be stored in a distributed manner.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: image stitching-based context recognition device
110: modular camera
120: integrated control
130: exhibition department
140: 3-axis gyro sensor

Claims (14)

Multiple modular cameras based on infrared or visible light sensors;
The image period is controlled for stitching on the overlapping area of each image captured by the plurality of modular cameras, and the image is displayed to correspond to the observer viewpoint selected by the user among the images captured by the plurality of modular cameras. An integrated control unit that enlarges or reduces and stitches the enlarged or reduced image to generate a panoramic image; and
And a display unit for outputting the generated panoramic image,
The integrated control unit extracts feature points from each of a plurality of images and matches the extracted feature points of the enlarged or reduced image based on the image corresponding to the selected observer's viewpoint,
The plurality of modular cameras is composed of three two-stage modular cameras,
When the image corresponding to the first modular camera having the largest viewing angle among the plurality of modular cameras is selected as the observer's viewpoint, the integrated control unit reduces the image of the remaining second modular cameras except for the first modular camera processing and matching the extracted feature points of the reduced image,
The plurality of modular cameras is composed of three two-stage modular cameras,
When the image corresponding to the second modular camera other than the first modular camera having the largest viewing angle among the plurality of modular cameras is selected as the observer's point of view, the integrated control unit enlarges the image of the first modular camera processing and matching the extracted feature points of the enlarged image,
Video stitching-based situational awareness device.
delete delete delete According to claim 1,
the first modular camera is a pair of modular cameras having a viewing angle of 95 degrees and an overlap area of 6.5 degrees;
The second modular camera is composed of a pair of modular cameras each having a field of view of 54 degrees and an overlap area of 0.5 degrees, and another pair of modular cameras each having a field of view of 50 degrees and an overlap area of 2.5 degrees sign,
Video stitching-based situational awareness device.
6. The method of claim 5,
The integrated control unit generates a panoramic image having a viewing angle of up to 386 degrees by adjusting the overlapping area of the images captured by the plurality of modular cameras,
Video stitching-based situational awareness device.
According to claim 1,
The plurality of modular cameras will each be configured to have different viewing angles,
Video stitching-based situational awareness device.
According to claim 1,
Further comprising a 3-axis gyro sensor attached to the bottom of the plurality of modular cameras to perform a digital pan operation and tilt operation of the modular camera,
Video stitching-based situational awareness device.
In the image stitching method in the context recognition device,
Receiving an image taken through a plurality of modular cameras based on infrared or visible light sensors;
extracting feature points from each of the captured images;
receiving an observer's viewpoint selection from among the captured images;
enlarging or reducing the image to correspond to the selected observer's viewpoint; and
Creating a panoramic image by stitching to match the feature points of the enlarged or reduced image,
When the image corresponding to the first modular camera having the largest viewing angle among the plurality of modular cameras is selected as the observer's viewpoint,
Enlarging or reducing the image to correspond to the selected observer's viewpoint comprises:
Reduce the image of the second modular camera other than the first modular camera, or
When an image corresponding to the second modular camera other than the first modular camera having the largest viewing angle among the plurality of modular cameras is selected as the observer's viewpoint,
Enlarging or reducing the image to correspond to the selected observer's viewpoint comprises:
Enlarging the image of the first modular camera,
How to stitch video.
delete delete 10. The method of claim 9,
the first modular camera is a pair of modular cameras having a viewing angle of 95 degrees and an overlap area of 6.5 degrees;
The second modular camera is composed of a pair of modular cameras each having a field of view of 54 degrees and an overlap area of 0.5 degrees, and another pair of modular cameras each having a field of view of 50 degrees and an overlap area of 2.5 degrees sign,
How to stitch video.
13. The method of claim 12,
The step of creating a panoramic image by stitching to match the feature points of the enlarged or reduced image includes:
By adjusting the overlap area of the image taken by the plurality of modular cameras to generate a panoramic image having a viewing angle of up to 386 degrees,
How to stitch video.
10. The method of claim 9,
The plurality of modular cameras will each be configured to have different viewing angles,
How to stitch video.

Images