KR102264111B1 - Apparatus and method for three-dimensional calibration of video image - Google Patents

Abstract

An apparatus and method for three-dimensional correction of an image are disclosed. An apparatus for correcting a 3D image according to an embodiment of the present invention includes: a template overlapping module for obtaining an image, and superimposing a 3D template on the obtained image to generate an overlay image; and a template adjustment module for arranging a preset virtual object on the overlay image, and adjusting a characteristic parameter of the 3D template through feature comparison between the placed virtual object and a real object in the image.

Description

Apparatus and method for three-dimensional correction of images {APPARATUS AND METHOD FOR THREE-DIMENSIONAL CALIBRATION OF VIDEO IMAGE}

Embodiments of the present invention relate to techniques for recognizing objects in an image using a camera.

As surveillance cameras such as Closed-Circuit Television (CCTV) become common, video data is increasingly used to track the movement path of a suspect or a vehicle of the suspect. Control using CCTV is mainly event-driven control that detects the size and movement of objects (such as people or vehicles) in the video through video analysis of the cameras installed in specific places. is mainly made

Since an image photographed through a camera is two-dimensional whereas an actual photographed subject is three-dimensional, information such as a viewing angle, a height, and an angle of view of the currently installed camera is required to detect a three-dimensional object on a two-dimensional screen. To this end, conventionally, a system operator or the like manually inputs the information while viewing the screen. However, since this is a task that requires a long experience and trial and error of the operator, it takes a very long time, and there is a limit in accuracy as the whole process is done manually.

Republic of Korea Patent Publication No. 10-0863507 (2008. 10. 08)

SUMMARY Embodiments of the present invention are to provide a means for effectively overlaying a 3D template on an image of a surveillance camera or the like.

According to an exemplary embodiment of the present invention, there is provided a template superimposition module for acquiring an image, and superimposing a 3D template on the acquired image to generate an overlay image; and a template adjustment module for arranging a preset virtual object on the overlay image, and adjusting a characteristic parameter of the three-dimensional template through feature comparison between the placed virtual object and a real object in the image. A calibration device is provided.

The template overlapping module displays the one or more pre-stored candidate templates together with the acquired image on a screen, and when any one candidate template among the displayed one or more candidate templates is selected, the selected candidate template is displayed on the image An overlay image can be created by superimposing it on the .

The template overlay module may generate the overlay image by extracting feature information of the image from the image, and selecting a candidate template having the highest similarity to the extracted feature information from among pre-stored candidate templates.

The template overlapping module may select a candidate template having the highest degree of similarity to the extracted feature information from among pre-stored candidate templates based on at least one of a photographing area name of a camera that has captured the image or a keyword stored in association with the camera. have.

The template adjustment module may arrange a candidate virtual object having the highest similarity to the real object among one or more previously stored candidate virtual objects on the overlay image.

The template adjustment module may select a candidate virtual object having the highest similarity to the real object by comparing the size of the real object input from the user with the stored size of the one or more candidate virtual objects.

The template adjustment module may select a candidate virtual object to be placed on the overlay image from among previously stored candidate virtual objects based on at least one of a photographing area name of a camera that captured the image or a keyword stored in association with the camera.

The template adjustment module is configured to: a size change in the image of the real object measured as the real object moves in the image, and a three-dimensional template of the real object calculated at the same location as the real object. A change in the size of may be compared, and the characteristic parameter of the 3D template may be adjusted according to the comparison result.

The characteristic parameter may include one or more of a projection height, a projection direction, an inclination, an angle of view, and a focal position of the 3D template.

The template adjustment module may adjust the characteristic parameter of the 3D template by reflecting the input value when a correction value of the characteristic parameter of the 3D template is input by the user.

According to another exemplary embodiment of the present invention, generating an overlay image by acquiring an image, and superimposing a 3D template on the acquired image; and arranging a preset virtual object on the overlay image, and adjusting a characteristic parameter of the three-dimensional template through feature comparison between the placed virtual object and a real object in the image. this is provided

The generating of the overlay image may include: displaying one or more pre-stored candidate templates together with the obtained image on a screen; and overlaying the selected candidate template on the image as any one candidate template among the displayed one or more candidate templates is selected.

The adjusting of the characteristic parameter may include disposing a candidate virtual object having the highest similarity to the real object among one or more previously stored candidate virtual objects on the overlay image.

The generating of the overlay image may include extracting feature information of the image from the image, and selecting a candidate template having the highest similarity to the extracted feature information from among pre-stored candidate templates to generate the overlay image.

The template overlapping module may select a candidate template having the highest degree of similarity to the extracted feature information from among pre-stored candidate templates based on at least one of a photographing area name of a camera that has captured the image or a keyword stored in association with the camera. have.

The adjusting of the characteristic parameter may include selecting a candidate virtual object having the highest similarity to the real object by comparing the size of the real object input from the user with the stored size of the one or more candidate virtual objects.

The adjusting of the characteristic parameter may include selecting a candidate virtual object to be placed in the overlay image from among pre-stored candidate virtual objects based on at least one of a photographing area name of a camera that captured the image or a keyword stored in association with the camera. can

The adjusting of the characteristic parameter may include: a size change in the image of the real object measured as the real object moves in the image, and the three values of the real object calculated at the same location as the real object. The size change on the dimension template may be compared, and the characteristic parameter of the three-dimensional template may be adjusted according to the comparison result.

The characteristic parameter may include one or more of a projection height, a projection direction, a tilt angle of view, and a focal position of the 3D template.

The adjusting of the characteristic parameter may include adjusting the characteristic parameter of the 3D template by reflecting the input value when a correction value of the characteristic parameter of the 3D template is input from the user.

According to the embodiments of the present invention, three-dimensional templates of various types and shapes are stored in advance, and then, using a template determined to be appropriate among the stored templates according to the characteristics of the image to be photographed, is superimposed on the actual image to quickly and easily shoot. 3D image correction of the image may be performed.

In addition, according to embodiments of the present invention, a three-dimensional (3D) object is arranged by placing a pre-stored virtual object in the three-dimensional template and correcting parameters of the overlaid three-dimensional template through feature comparison between the virtual object and the real object in the image It is possible to improve the accuracy of image correction.

In addition, according to the embodiments of the present invention, it is possible to correct not only the distance change of the object but also the change in the size of the object according to the change in the viewing angle of the camera.

1 is a block diagram illustrating an apparatus 100 for correcting a 3D image according to an embodiment of the present invention.
2 is an exemplary diagram for explaining a three-dimensional template according to an embodiment of the present invention;
3 is an exemplary view showing an example in which the 3D template shown in FIG. 2 is superimposed on an image
4 is an exemplary view illustrating an example of arranging a virtual object on the overlay image shown in FIG. 3
5 is an exemplary diagram for explaining the process of adjusting the characteristic parameters of the three-dimensional template in the template adjustment module 106 according to an embodiment of the present invention.
6 and 7 are exemplary views for explaining a process for calculating a characteristic value of a camera according to an embodiment of the present invention;
8 is a flowchart illustrating a 3D correction method 800 of an image according to an embodiment of the present invention.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to provide a comprehensive understanding of the methods, apparatus, and/or systems described herein. However, this is merely an example and the present invention is not limited thereto.

In describing the embodiments of the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification. The terminology used in the detailed description is for the purpose of describing embodiments of the present invention only, and should in no way be limiting. Unless explicitly used otherwise, expressions in the singular include the meaning of the plural. In this description, expressions such as “comprising” or “comprising” are intended to indicate certain features, numbers, steps, acts, elements, some or a combination thereof, one or more other than those described. It should not be construed to exclude the presence or possibility of other features, numbers, steps, acts, elements, or any part or combination thereof.

1 is a block diagram illustrating an apparatus 100 for correcting a 3D image according to an embodiment of the present invention. The apparatus 100 for correcting a 3D image according to an embodiment of the present invention is an apparatus for calculating a position in 3D space of an object (subject) in the 2D image by superimposing a 3D template on the 2D image to be. The three-dimensional correction apparatus 100 of an image according to an embodiment of the present invention is included as an element inside a camera device, such as a surveillance camera, or is configured as a separate device from the surveillance camera and connected to the surveillance camera. can be As shown, the apparatus 100 for correcting a 3D image according to an embodiment of the present invention includes a data storage module 102 , a template overlapping module 104 , and a template adjustment module 106 .

The data storage module 102 is a space for storing one or more three-dimensional templates and one or more virtual objects to be overlaid on an image captured by a surveillance camera or the like.

In one embodiment of the present invention, the 3D template is a guideline for calculating the position in 3D space of an object located in an image, and as shown in FIG. 2 , a plurality of It may be configured to include a grid. Each grid constituting the three-dimensional template corresponds to a space of the same size in the real space even though the sizes appear to be different on the two-dimensional screen. When the three-dimensional template superimposed on the image is used, the object on the screen It becomes possible to calculate the size and position of in real space. 3 shows an example in which the 3D template shown in FIG. 2 is superimposed on an image.

The data storage module 102 may store one or more different three-dimensional templates. For example, the data storage module 102 may store three-dimensional templates having different shapes according to the characteristics of spaces, such as a flat ground, a road, an intersection, an entrance, an office, and a hallway.

In addition, a virtual object is a kind of virtual object to be displayed overlaid with a 3D template in an image, and has the shape of a person (man, woman, child, etc.), vehicle (van, bus, passenger car, etc.), street tree, street lamp, etc. can The data storage module 102 may store size information (height, width, etc.) of each object together with the virtual object. The virtual object is used to correct the 3D template through comparison with the real object in the image. FIG. 4 illustrates an example of arranging virtual objects (trees, people, cars, etc.) on the overlay image shown in FIG. 3 .

The template overlay module 104 acquires an image from a camera or the like, and superimposes a 3D template on the acquired image to generate an overlay image.

In one embodiment, the template overlay module 104 stores and manages information of one or more three-dimensional templates (candidate templates) to be superimposed on an image in the data storage module 102, and stores and manages the one or more three-dimensional templates (candidate templates) together with the obtained image. A dimension template can be displayed on the screen. Then, the user selects any one of the displayed 3D templates, and the template overlay module 104 may generate an overlay image by superimposing the selected candidate template on the image.

In another embodiment, the template overlapping module 104 derives the characteristics of the obtained image (the presence or absence of a horizon in the image, the position and connection relationship of main feature lines, etc.), and automatically selects an optimal 3D template suitable for the derived characteristics. It may be configured to select. To this end, the template overlapping module 104 may use various algorithms capable of deriving characteristic information in the image. Also, for this purpose, the template overlapping module 104 may include an algorithm for recognizing an object and its type (car, person, etc.) in the screen in order to accurately recognize the position and connection relationship of the main feature lines in the image.

Also, the template overlapping module 104 may select an optimal 3D template using information of a camera that has captured an image. For example, the name of the shooting area or shooting area (POSCO Intersection, Namsan No. 1 Tunnel, etc.) is added to the camera that captured the video, or keywords that can recognize the shape of the shooting area separately (entrance, intersection, etc.) , conference room, hallway, etc.) is added, the template overlapping module 104 may be used to select an optimal template from among pre-stored templates based on this.

The template adjustment module 106 arranges a virtual object on the overlay image generated by the template overlay module 104, and compares the placed virtual object with a real object in the image to obtain a characteristic parameter of the three-dimensional template. to adjust

In an embodiment of the present invention, the characteristic parameter is information including at least one of a projection height, a projection direction, a tilt, an angle of view, and a focal position of the 3D template, and is mainly based on the ground portion of the 3D template on the ground of the actual image. information necessary to match Since the characteristic parameter varies depending on the installation height, lens type (focal length, etc.), tilt angle, etc. of a camera that has taken an image, when the characteristic parameter of the three-dimensional template is calculated, the installation height of the camera and angle from the subject It becomes possible to calculate information such as distance and angle of view.

Specifically, the template adjustment module 106 may select a candidate virtual object having the highest similarity to the size of the real object in the image from among one or more virtual objects stored in the data storage module 102 and place it on the overlay image. To this end, the template adjustment module 106 may receive size information (such as height or width) of the real object in the image from the user. For example, when a woman having a height of 160 cm exists on the screen, the template adjustment module 106 may arrange a virtual object having the same height as the woman on the overlay image. Also, the template adjustment module 106 may automatically place an optimal virtual object on the overlay image according to the characteristics of the image. For example, when the image is an intersection, a vehicle may be disposed as a virtual object, and in the case of a hallway or an entrance, a person may be disposed. Such characteristics of the image can be found by using the name or keyword of the camera that captured the image as described above.

The size of the virtual object in the image varies according to the position of the virtual object in the image and information obtained from the superimposed 3D template. For example, even if a virtual object of the same size is a virtual object at a location determined to be far from the camera on the 3D template, the size on the screen is smaller than a virtual object at a location determined to be closer than that.

When the virtual object is arranged in the overlay image as described above, the template adjustment module 106 adjusts the characteristic parameter of the 3D template by comparing the real object and the virtual object in the image. Specifically, the template adjustment module 106 is configured to configure a size change in the image of the real object measured as the real object moves in the image, and a change in size of the real object calculated at the same location as the real object. The size change on the 3D template is compared, and the characteristic parameter of the 3D template is adjusted according to the comparison result. That is, in the present invention, the characteristic of a real object or a virtual object means a degree of change in size when the corresponding object moves in an image. A more detailed explanation of this is as follows.

5 is an exemplary diagram for explaining a process of adjusting the characteristic parameter of a 3D template in the template adjustment module 106 according to an embodiment of the present invention. In the illustrated example, 502 denotes a real object in the image, and 504 denotes a virtual object, respectively. As shown, as a real object (eg, a person) in the image moves from position A to position B, its height (height) is changed from h1 to h1'. In the case of a virtual object 504 at the same position, the height Assume that (key) is changed from h2 to h2'. As described above, the size of the virtual object in the image is determined according to the characteristic parameter of the 3D template superimposed on the image. Therefore, if h1 is the same as h2, but h1' is not the same as h2', it means that the characteristic parameters of the overlapped three-dimensional template do not match the real space, so the template adjustment module 106 is the three-dimensional template The characteristic parameter is adjusted by changing one or more of the projection height, projection direction, tilt, angle of view, and focal position of . Such adjustment is repeated until the change according to the movement of the real object in the image becomes the same as that of the virtual object.

Also, the template adjustment module 106 may receive a correction value for the characteristic parameter from the user and reflect it in the adjustment of the characteristic parameter. To this end, the template adjustment module 106 may provide an interface for the user to intuitively adjust the template using a keyboard, mouse, or screen touch. As such, when the template adjustment module 106 uses the correction value input by the user along with the change in the size of the object in the image, the accuracy of the characteristic parameter can be further increased.

When the 3D template is adjusted as described above, the installation characteristic value of the camera can be calculated from the object in the image. 6 and 7 are for explaining this. As shown, there are two objects, object A and object B, on the screen, object A is located on the center line (indicated by a dotted line) connected from the camera, and object B is located at an angle x distance from object A Suppose we do

Assuming that the distance between the object A and the object B is y and the distance between the object B and the center line is z, the installation position cl of the camera can be calculated as follows.

cl = z / tan(x) + y

Assuming that the size of the object A on the screen is ah and the size of the object B on the screen is bh, the camera installation height ch can be calculated as follows.

ch = (bh - ah) * cl / y

From this, the angle of view (cv) and the installation angle (ct) of the camera can be calculated as follows.

cv = arctan(ch/cl)

ct = 90 - arctan(ch/cl)

8 is a flowchart illustrating a 3D correction method 800 of an image according to an embodiment of the present invention.

In step 802 , the template overlay module 104 receives a selection of a 3D template on which an image is to be superposed from a user or the like. To this end, the template overlapping module 102 may provide an appropriate interface for the user to visually browse pre-stored templates and select a template suitable for an image among them. For example, the template overlapping module 104 may display one or more pre-stored candidate templates together with the obtained image on a screen, and select any one candidate template from among the displayed one or more candidate templates.

In step 804 , the template nesting module 104 determines whether an input for a set value (a characteristic parameter, etc.) for the selected template is necessary, and if it is determined that it is necessary, receives it from the user in step 806 .

In step 808, the template adjustment module 106 receives a virtual object to be placed on the image from a user or the like. In this case, the virtual object may be a candidate virtual object having the highest degree of similarity to the real object among one or more previously stored candidate virtual objects.

In step 810, the template adjustment module 106 determines whether it is necessary to input a set value (such as the height or length of the object) for the selected virtual object, and if it is determined that it is necessary, input it from the user in step 812 receive

In step 814 , the template adjustment module 106 determines whether the characteristics of the virtual object and the real object coincide with each other through feature comparison between the virtual object disposed in the image and the real object in the image. As described above, the template adjustment module 106 is configured to include a change in size of the real object in the image measured as the real object moves in the image, and the real object calculated at the same location as the real object. may compare the size change on the 3D template, and adjust the characteristic parameter of the 3D template according to the comparison result.

If it is determined in step 814 that the characteristics of the real object and the virtual object do not match, the template adjustment module 106 adjusts the characteristic parameters of the template until the characteristics match in step 816 .

In step 818, the template adjustment module 106 determines whether a request for adding a virtual object is received from the user, and if there is a request for adding a virtual object, repeats steps 808 and below for the newly selected virtual object.

Meanwhile, an embodiment of the present invention may include a computer-readable recording medium including a program for performing the methods described in this specification on a computer. The computer-readable recording medium may include program instructions, local data files, local data structures, and the like alone or in combination. The medium may be specially designed and configured for the present invention, or may be commonly used in the field of computer software. Examples of the computer-readable recording medium include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and ROMs, RAMs, flash memories, and the like. Hardware devices specifically configured to store and execute program instructions are included. Examples of program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although representative embodiments of the present invention have been described in detail above, those of ordinary skill in the art to which the present invention pertains will understand that various modifications are possible without departing from the scope of the present invention with respect to the above-described embodiments. . Therefore, the scope of the present invention should not be limited to the described embodiments and should be defined by the claims described below as well as the claims and equivalents.

100: three-dimensional correction device of the image
102: data storage module
104: Template nested module
106: template adjustment module

Claims (20)

a template overlapping module for acquiring an image and superimposing a 3D template on the acquired image to generate an overlay image; and
a template adjustment module for arranging a preset virtual object on the overlay image, and adjusting a characteristic parameter of the three-dimensional template through feature comparison between the placed virtual object and a real object in the image;
The template adjustment module is configured to: a size change in the image of the real object measured as the real object moves in the image, and a three-dimensional template of the real object calculated at the same location as the real object. A three-dimensional correction apparatus for an image, comparing the size change of , and adjusting the characteristic parameter of the three-dimensional template according to the comparison result.
The method according to claim 1,
The template nesting module is
Displaying one or more pre-stored candidate templates together with the obtained image on the screen,
When any one candidate template among the displayed one or more candidate templates is selected, the selected candidate template is superimposed on the image to generate an overlay image.
The method according to claim 1,
The template overlay module extracts feature information of the image from the image, selects a candidate template having the highest similarity to the extracted feature information from among pre-stored candidate templates to generate the overlay image, 3D correction of the image Device.
4. The method according to claim 3,
The template overlapping module is configured to select a candidate template having the highest degree of similarity to the extracted feature information from among pre-stored candidate templates based on at least one of a photographing area name of a camera that has captured the image or a keyword stored in association with the camera. , a three-dimensional correction device for images.
The method according to claim 1,
The template adjustment module is configured to arrange, on the overlay image, a candidate virtual object having the highest similarity to the real object among one or more previously stored candidate virtual objects.
6. The method of claim 5,
The template adjustment module is configured to select a candidate virtual object to be placed in the overlay image from among previously stored candidate virtual objects based on at least one of a photographing area name of a camera that has captured the image or a keyword stored in association with the camera, of the three-dimensional correction device.
6. The method of claim 5,
The template adjustment module is configured to select a candidate virtual object having the highest similarity to the real object by comparing the size of the real object input from the user with the stored size of the one or more candidate virtual objects.
delete The method according to claim 1,
The characteristic parameter includes at least one of a projection height, a projection direction, a tilt, an angle of view, and a focal position of the 3D template.
The method according to claim 1,
wherein the template adjustment module adjusts the characteristic parameter of the 3D template by reflecting the input value when a correction value of the characteristic parameter of the 3D template is input from the user.
generating an overlay image by acquiring an image and superimposing a 3D template on the acquired image; and
disposing a preset virtual object on the overlay image, and adjusting a characteristic parameter of the three-dimensional template through feature comparison between the placed virtual object and a real object in the image,
Adjusting the characteristic parameter comprises:
comparing the size change in the image of the real object measured as the real object moves in the image, and the size change in the 3D template of the real object calculated at the same location as the real object, , a three-dimensional correction method of an image for adjusting the characteristic parameter of the three-dimensional template according to the comparison result.
12. The method of claim 11,
The step of generating the overlay image comprises:
displaying one or more pre-stored candidate templates together with the obtained image on a screen; and
The method of claim 1, further comprising: superimposing the selected candidate template on the image as any one candidate template among the displayed one or more candidate templates is selected.
12. The method of claim 11,
The step of generating the overlay image comprises:
A 3D correction method of an image, comprising extracting feature information of the image from the image, and generating the overlay image by selecting a candidate template having the highest similarity to the extracted feature information from among pre-stored candidate templates.
14. The method of claim 13,
The generating of the overlay image may include: a candidate template having the highest similarity to the feature information extracted from pre-stored candidate templates based on at least one of a photographing area name of a camera capturing the image or a keyword stored in association with the camera A three-dimensional correction method of the image to select.
12. The method of claim 11,
The adjusting of the characteristic parameter may include disposing a candidate virtual object having the highest similarity to the real object among one or more previously stored candidate virtual objects in the overlay image.
16. The method of claim 15,
The adjusting of the characteristic parameter may include selecting a candidate virtual object to be placed in the overlay image from among pre-stored candidate virtual objects based on at least one of a photographing area name of a camera that captured the image or a keyword stored in association with the camera. A three-dimensional correction method of an image.
16. The method of claim 15,
The adjusting of the characteristic parameter may include selecting a candidate virtual object having the highest similarity to the real object by comparing the size of the real object input from a user with the stored size of the one or more candidate virtual objects. Calibration method.
delete 12. The method of claim 11,
The characteristic parameter includes at least one of a projection height, a projection direction, an inclination angle of view, and a focal position of the 3D template.
12. The method of claim 11,
In the adjusting of the characteristic parameter, when a correction value of the characteristic parameter of the 3D template is input from a user, the characteristic parameter of the 3D template is adjusted by reflecting the input value.

Images