KR20200025019A - Server, method and computer program for generating normalized video - Google Patents

Abstract

A server for generating a normalized image comprises: an image receiving part receiving an image of a target place in which at least three markers are placed that is photographed by a camera installed in the target place; a detection part detecting a first feature area for each marker based on each marker in the received image and detecting a second feature area placed in the detected first feature area; an analysis part analyzing the second feature area of each marker; and a generation part generating a planar image for the target place based on the analyzed second feature area and generating a normalized image from the planar image.

Description

Servers, methods and computer programs for generating normalized images {SERVER, METHOD AND COMPUTER PROGRAM FOR GENERATING NORMALIZED VIDEO}

The present invention relates to a server, a method and a computer program for generating a normalized image.

The time slice technique is to install a plurality of cameras facing the subject at various angles and shoot them at the same time, and then connect the pictures using a computer, so that the still motions of the subject appear as if they were taken with a movie camera. Says imaging technique. Time slices can not only describe the subject in three dimensions, but also provide a sense of time and space.

In connection with such a time slice technique, Korean Patent Laid-Open Publication No. 2007-0000994 discloses a system and method for recording and reproducing an image.

In order to provide a time slice image, it is necessary to estimate a pose of a camera. Conventionally, the pose of the camera is estimated using a pattern such as a square pattern or a circular pattern. However, when estimating the pose of the camera in a large area, it is difficult to install the pattern to cover a large area, and it is difficult to obtain an image of the camera and the ground vertically due to the large area. There is a problem that occurs.

The present invention provides a server, a method, and a computer program for generating a normalized image for easily and accurately estimating a pose according to a camera movement through a method of recognizing a marker.

Server, method and computer program for generating normalized images that can be applied to various situations by changing color values according to the shooting environment by rotating and installing the marker design in each place using one marker design. To provide.

A server, method, and computer program for generating normalized images for applying camera pose estimation in various imaging applications such as a time slice imaging system, a global augmented reality, and a surveillance system.

To provide a server, a method and a computer program for generating a normalized image for estimating poses easily and accurately for a camera installed in a large area.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems as described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention, an image receiving unit for receiving an image photographing the target place where at least three markers are arranged, the camera installed in the target place, within the received image A detector configured to detect a first feature region for each marker based on the respective markers, and detect a second feature region located within the detected first feature region, and analyze the second feature region of each marker The normalization server may include a generation unit configured to generate a planar image of the target place based on the analyzed second feature region and generate a normalized image from the planar image.

According to another embodiment of the present invention, a camera installed at a target place receives an image of photographing the target place where at least three markers are arranged, and a method for each marker is based on the respective markers in the received image. Detecting a first feature region, detecting a second feature region located within the detected first feature region, analyzing a second feature region of each marker, based on the analyzed second feature region The method may include generating a planar image of the target place and generating a normalized image from the planar image.

According to another embodiment of the present invention, when the computer program is executed by a computing device, a camera installed at a target place receives an image of the target place where at least three markers are arranged, and the respective images within the received image. Detect a first feature region for each marker based on the marker, detect a second feature region located within the detected first feature region, analyze a second feature region of each marker, and analyze the A computer program stored in a medium including a sequence of instructions for generating a planar image of the target location based on a second feature region and generating a normalized image from the planar image.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the aforementioned problem solving means of the present invention, to provide a server, method and computer program for generating a normalized image to be able to easily and accurately estimate the pose according to the movement of the camera through a method of recognizing the marker. Can be.

Server, method and computer program for generating normalized images that can be applied to various situations by changing color values according to the shooting environment by rotating and installing the marker design in each place using one marker design. Can provide.

A server, method, and computer program for generating normalized images for applying camera pose estimation in various imaging applications such as a time slice imaging system, global augmented reality, and a surveillance system may be provided.

Servers, methods, and computer programs for generating normalized images that enable easy and accurate estimation of poses for cameras installed in large area locations can be provided.

1 is a block diagram of a normalization system according to an embodiment of the present invention.
2 is a block diagram of a normalization server according to an embodiment of the present invention.
3A to 3D are exemplary views for explaining a process of generating a normalized image according to an embodiment of the present invention.
4 is a flowchart of a method of generating a normalized image in a normalization server according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, which means that it may further include other components, without excluding other components, unless specifically stated otherwise, one or more other features It is to be understood that the present disclosure does not exclude the possibility of adding or presenting numbers, steps, operations, components, parts, or combinations thereof.

In the present specification, the term 'unit' includes a unit realized by hardware, a unit realized by software, and a unit realized by both. In addition, one unit may be realized using two or more pieces of hardware, and two or more units may be realized by one piece of hardware.

Some of the operations or functions described as being performed by the terminal or the device in the present specification may instead be performed in a server connected to the terminal or the device. Similarly, some of the operations or functions described as being performed by the server may be performed by a terminal or a device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a normalization system according to an embodiment of the present invention. Referring to FIG. 1, the normalization system 1 may include a camera 110 and a normalization server 120. Camera 110 and normalization server 120 illustratively illustrate components that may be controlled by normalization system 1.

Each component of the normalization system 1 of FIG. 1 is generally connected via a network. For example, as shown in FIG. 1, the camera 110 may be connected to the normalization server 120 at the same time or at a time interval.

A network refers to a connection structure capable of exchanging information between nodes such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired / wireless data communication network, telephone network, wired / wireless television communication network, and the like. Examples of wireless data networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, Infrared communication, Ultrasound Communication, visible light communication (VLC), liFi (LiFi) and the like, but are not limited thereto.

The camera 110 may be installed at the target place 100 to photograph the target place 100 in which at least three markers are disposed. Here, the target place 100 may be a sports stadium, a performance hall, etc. of a large area.

The normalization server 120 may receive an image of the camera 110 installed at the target place 100 photographing the target place 100 where at least three markers are disposed.

The normalization server 120 may detect a first feature region for each marker based on each marker in the received image, and detect a second feature region located in the detected first feature region.

Each marker may include an edge area composed of at least one color. For example, the normalization server 120 detects an edge region of each marker from the received image based on a preset color model, and detects a first feature region for each marker using the detected edge region of each marker. can do.

Each marker may further include a pattern region. Here, the pattern region of each marker may be composed of at least two divided inner regions (for example, four divided inner regions), and each inner region may be binary coded. At this time, the inner region of each marker may be binary coded differently in a clockwise or counterclockwise order with respect to each marker disposed in the target place 100. As another example, the pattern region of each marker may be configured as an inner region of a circle shape divided into two based on a boundary line. At this time, each marker may be arranged such that the boundary line is at an angle with the tangent line to the boundary of the target place.

The normalization server 120 may detect the second feature area corresponding to the pattern area by excluding the edge area from the detected first feature area.

The normalization server 120 may analyze the second feature region of each marker. For example, the normalization server 120 may analyze the pattern region of each marker detected as the second feature region.

The normalization server 120 may analyze the pattern region of each marker and index it for each marker.

The normalization server 120 may generate a planar image of the target place based on the analyzed second feature region, and generate a normalized image from the planar image.

The normalization server 120 may generate a planar image by mapping a center point of each indexed marker to a vertex coordinate of the ground of the target place 100, and generate a normalized image by warping the planar image.

The normalization server 120 may estimate pose information of another camera based on the normalized image and the image of the target place 100 photographed from another camera installed in the target place 100. For example, the normalization server 120 may estimate pose information of another camera through a transformation matrix for the matched feature point calculated by performing feature point matching between the image of the target place 100 and the normalized image.

2 is a block diagram of a normalization server according to an embodiment of the present invention. Referring to FIG. 2, the normalization server 120 may include an image receiver 210, a detector 220, an analyzer 230, a generator 240, and an estimator 250.

The image receiver 210 may receive an image of the camera 110 installed in the target place 100 photographing the target place 100 where at least three markers are disposed. Here, the at least three markers may be arranged in the order of the rotation of the pattern on the corner of the target place 100 as a polygonal or circular structure in which the same pattern is printed.

The detector 220 may detect a first feature region for each marker based on each marker in the received image, and detect a second feature region located in the detected first feature region.

The detector 220 may detect a first feature region for each marker based on each marker in the image. Here, each marker includes a border area composed of at least one color, and the color of the border area of each marker may be selectively configured according to a shooting environment, and may be configured in a color contrasted with the color of the bottom surface on which each marker is to be installed. Can be.

The detector 220 may detect an edge region of each marker from the received image based on the preset color model. For example, the detector 220 may detect the color of the border area by limiting the 'Hue' area to the range of the border area by using a Hue Saturation Value (HSV) color space.

Here, the reason for detecting the edge region of the marker by using the color information is that when the camera 110 installed in the target place 100 is inclined, distortion of characteristic information, image degradation, etc., in the image received from the camera 110 are included. Is generated, making it difficult to detect the edge area. Therefore, it is for easily and accurately estimating the edge area using clear color information so as to be robust against changes in the external environment.

The detector 220 may detect the first feature region for each marker by using the detected edge region of each marker. In this case, each of the connection components may be labeled such that the detected edge region is included in the edge by closing the empty region through the morphology process. At this time, an index may be given to each border region through a labeling operation.

The detector 220 may detect the second feature area corresponding to the pattern area by excluding the edge area from the detected first feature area. Here, each marker may further include a pattern region. For example, the pattern region of each marker may be composed of an inner region of a polygon divided into at least two, and each inner region may be binary coded. As another example, the pattern region of each marker may be configured as an inner region of a circle shape divided into two based on a boundary line. Each marker may be arranged such that the boundary line is at an angle with the tangent line to the boundary of the target place.

The analyzer 230 may analyze the second feature region of each marker. For example, the analyzer 230 may analyze the pattern region of each marker detected by the detector 220 as the second feature region. The analyzer 230 may analyze the pattern region of each marker and index the marker for each marker.

The pattern region of each marker may be composed of four divided internal regions. The inner region of each marker may be binary coded differently in a clockwise or counterclockwise order for each marker disposed in the target location 100. For example, the inner region of each marker may consist of a black region and a white region, with the black region having a value of '0' and the white region having a value of '1' to be indexed with four digit codes per marker. Can be. In this way, up to 16 codes are possible, and up to 16 hexagonal planes can be estimated. For example, if each marker is coded clockwise in an evenly divided internal region of a binary coded square, the upper left is '1001', the upper right is '1100', the left is '0011', the right is '0011' Can be defined as' The reason for performing binary coding on each marker is to prevent occurrence of a warping error in the warping process for the planar image in the generation unit 240.

The generation unit 240 may generate a planar image of the target place based on the analyzed second feature region and generate a normalized image from the planar image. For example, the generation unit 240 may generate a planar image by mapping a center point of each indexed marker to a vertex coordinate of the ground of the target place 100, and generate a normalized image by warping the planar image. have. In this way, the generation unit 240 may generate a normalized image in which the value of the main axis that is perpendicular to the center of the camera 110 and the coordinate axis of the space passing through the center of the camera 110 is located at '0'.

The normalized image is a real world coordinate system image obtained by converting a 2D image into a 3D space, and obtains coordinate values of a matched normalized image by converting feature coordinates of a new frame input based on the image into a homogeneous coordinate system. It is possible to obtain a matrix including magnitude, rotation, torsion, movement, and the like.

The estimator 250 may estimate pose information of another camera 115 based on an image and a normalized image of the target place 100 photographed from another camera 115 installed in the target place 100. For example, the estimator 250 estimates pose information of another camera 115 through a transformation matrix of the matched feature point calculated by performing feature point matching between the image of the target place 100 and the normalized image. Can be. In this case, when it is difficult to match the feature point between the image of the target place 100 and the normalized image, the markers are estimated by capturing each marker disposed in the target place 100 through multiple cameras, and the image of the target place 100 is estimated. And feature points between the normalized image and the normalized image.

The normalization server 120 may be executed by a computer program stored in a medium including a sequence of instructions for generating a normalized image. When the computer program is executed by the computing device, the camera 110 installed in the target place 100 receives an image of the target place 100 in which at least three markers are arranged, and based on each marker in the received image. Detect a first feature region for each marker, detect a second feature region located within the detected first feature region, analyze a second feature region of each marker, and based on the analyzed second feature region It may include a sequence of instructions for generating a planar image for the target place 100 and generating a normalized image from the planar image.

3A to 3D are exemplary views for explaining a process of generating a normalized image according to an embodiment of the present invention.

3A is an exemplary diagram for describing a process of photographing a target place on which a marker is disposed through a camera installed at the target place according to an embodiment of the present invention. Referring to FIG. 3A, the camera 110 installed in the target place 100 may photograph the target place 100 where four markers 300 are disposed.

The four markers 300 are square instruments in which one marker 301 is printed in the same pattern, and the four markers 300 may be rotated at corner portions of the target place 100 and arranged in sequence.

3B is an exemplary diagram for describing a process of analyzing and indexing a pattern region from a rectangular marker according to an embodiment of the present invention. Referring to FIG. 3B, the rectangular marker 310 may include a border area 311 formed in red and a pattern area 312 formed in black and white. At this time, the pattern region 312 may be composed of four inner regions, and each inner region may be binary coded. For example, the black region of the internal region may have a value of '0' and the white region may have a value of '1'.

The normalization server 120 detects an edge region 311 of the rectangular marker 310 based on a preset color model, and uses the detected rectangular edge region 311 to form a rectangular marker 310. A first feature region for may be detected.

The normalization server 120 may detect and analyze a second feature region located in the detected first feature region. For example, the normalization server 120 may detect the second feature area corresponding to the pattern area 312 by excluding the edge area 311 from the first feature area. In this case, the normalization server 120 may analyze the pattern region 312 of the rectangular marker 310 and index each of the rectangular markers 310.

For example, when four markers are disposed 320 at the target place 100, the normalization server 120 may have upper left (1001, 9), upper right (1100, 12), lower right (0110, 5), and right ( 0011, 3) can be indexed for each rectangular marker.

For example, when six markers are disposed 330 in the target place 100, the normalization server 120 may include a first marker (0000, 0), a second marker (0001, 1), and a third marker ( 0010, 2), a fourth marker (0011, 3), a fifth marker (0100, 4), the sixth marker (0101, 5) can be indexed for each quadrant marker.

Through this, up to 16 binary codes can be estimated up to a plane of up to 16 hexagons.

FIG. 3C is an exemplary diagram for describing a process of analyzing and indexing a pattern region from a circular marker according to an embodiment of the present invention. FIG. Referring to FIG. 3C, the marker 340 having a circular shape may include an edge region 341 formed in red and a pattern region 342 composed of black and white. At this time, the pattern region 342 may be composed of two divided internal regions, and each inner region may be binary coded. For example, the black region of the internal region may have a value of '0' and the white region may have a value of '1'. The circular marker 340 may be arranged such that the boundary line is at an angle with the tangent line with respect to the boundary of the target place 100.

The normalization server 120 detects an edge region 341 of the marker 340 having a circular shape based on a preset color model, and uses the detected edge region 341 for the marker 340 having a circular shape. One feature area can be detected.

The normalization server 120 may detect and analyze a second feature region located in the detected first feature region. For example, the normalization server 120 may detect the second feature area corresponding to the pattern area 342 by excluding the edge area 341 from the first feature area. In this case, the normalization server 120 may analyze the pattern region 342 of the marker 340 having a circular shape and index each of the markers 340 having a circular shape.

The normalization server 120 may, for example, have four circular markers 340 arranged in the target place 100 or 350 if six circular markers 340 are disposed 360. After warping so that the second feature region becomes a garden, a line forming a boundary between black and white may be detected through Hough transform, and the marker may be independently indexed by the angle information formed by the line.

3D is an exemplary diagram for explaining a process of estimating pose information for another camera according to an embodiment of the present invention. Referring to FIG. 3D, the normalization server 120 may transmit images to other cameras based on the image 370 and the normalized image 380 of the target place 100 photographed from another camera 115 installed in the target place 100. Pose information may be estimated. At this time, the normalization server 120 performs a matching of the feature points 390 between the image 370 and the normalized image 380 with respect to the target place 100 through another transformation matrix for the matched feature points 390 calculated. The pose information for the camera 115 may be estimated.

4 is a flowchart of a method of generating a normalized image in a normalization server according to an embodiment of the present invention. The method for generating a normalized image in the normalization server 120 illustrated in FIG. 4 includes steps processed in time series by the normalization system 1 according to the embodiment illustrated in FIGS. 1 to 3D. Therefore, even if omitted below, the method is also applied to a method for generating a normalized image in the normalization server 120 according to the embodiment shown in FIGS. 1 to 3D.

In operation S410, the normalization server 120 may receive an image of the camera 110 installed at the target place 100 photographing the target place 100 where at least three markers are disposed.

In operation S420, the normalization server 120 may detect a first feature region for each marker based on each marker in the received image.

In operation S430, the normalization server 120 may detect a second feature region located in the detected first feature region.

In operation S440, the normalization server 120 may analyze the second feature region of each marker.

In operation S450, the normalization server 120 may generate a planar image of the target location based on the analyzed second feature region.

In operation S460, the normalization server 120 may generate a normalized image from the planar image.

In the above description, steps S410 to S460 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the invention. In addition, some steps may be omitted as necessary, and the order between the steps may be switched.

The method for generating a normalized image in the normalization server described with reference to FIGS. 1 to 4 may be implemented in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by the computer. In addition, the method for generating a normalized image in the normalization server described with reference to FIGS. 1 to 4 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer readable medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

110: camera
120: normalization server
210: video receiver
220: detector
230: analysis unit
240: generation unit
250: estimator

Claims (18)

In the server for generating a normalized image,
An image receiver configured to receive an image of the target location where at least three markers are arranged by a camera installed at the target location;
A detector configured to detect a first feature region for each marker based on the markers in the received image, and detect a second feature region located within the detected first feature region;
An analyzer for analyzing a second feature region of each marker; And
A generator configured to generate a planar image of the target place based on the analyzed second feature region, and generate a normalized image from the planar image;
That, including the normalization server.
The method of claim 1,
Each marker includes a border region composed of at least one color;
The detection unit detects an edge region of each marker from the received image based on a preset color model, and detects a first feature region for each marker using the detected edge region of each marker. , Normalization server.
The method of claim 2,
Each marker further comprises a pattern region,
The detection unit detects a second feature region corresponding to the pattern region by excluding the edge region from the detected first feature region,
The analyzing unit analyzes the pattern region of each marker detected as the second feature region.
The method of claim 3, wherein
The pattern region of each marker is composed of at least two divided inner regions, wherein each inner region is binary coded.
The method of claim 4, wherein
The analyzing unit analyzes the pattern region of each marker and indexes the markers.
The method of claim 5, wherein
The generation unit generates the planar image by mapping a center point of each indexed marker with a vertex coordinate of the ground of the target place, and generates the normalized image by warping the planar image.
The method of claim 4, wherein
Wherein the pattern region of each marker is composed of four divided internal regions.
The method of claim 7, wherein
Wherein the inner region of each marker is binary coded differently in a clockwise or counterclockwise order for each marker placed at the target location.
The method of claim 4, wherein
The pattern region of each marker is composed of a circle-shaped inner region divided into two based on the boundary line, normalization server.
The method of claim 9,
Wherein each of the markers is arranged such that the boundary line is at an angle with the tangent line to the boundary of the target place.
The method of claim 1,
An estimator for estimating pose information of the other camera based on the normalized image and the image of the target place photographed from another camera installed at the target place;
It further comprises a normalization server.
The method of claim 11,
And the estimating unit estimates pose information of the other camera through a transformation matrix of the matched feature point calculated by performing feature point matching between the image of the target place and the normalized image.
In the method for generating a normalized image,
Receiving, by a camera installed at a target location, an image of the target location having at least three markers arranged thereon;
Detecting a first feature region for each marker based on the markers in the received image;
Detecting a second feature region located within the detected first feature region;
Analyzing a second feature region of each marker;
Generating a planar image of the target location based on the analyzed second feature region; And
Generating a normalized image from the planar image
To include, normalized image generation method.
The method of claim 13,
Each marker includes a border region composed of at least one color;
The detecting of the first feature region may include detecting an edge region of each marker from the received image based on a preset color model, and using the detected edge region of each marker, the first feature for each marker. Detecting a feature region.
The method of claim 14,
Each marker further comprises a pattern region,
The detecting of the second feature region may include detecting the second feature region corresponding to the pattern region by excluding the edge region from the detected first feature region,
And analyzing the second feature region of each marker comprises analyzing a pattern region of each marker detected as the second feature region.
The method of claim 15,
Analyzing a pattern region of each marker and indexing the marker region
To further include, normalized image generation method.
The method of claim 16,
The generating of the normalized image may include generating the planar image by mapping a center point of each indexed marker to a vertex coordinate of the ground of the target location, and generating the normalized image by warping the planar image. , Normalized image generation method.
A computer program stored in a medium containing a sequence of instructions for generating a normalized image,
When the computer program is executed by a computing device,
A camera installed at a target place receives an image of the target place where at least three markers are arranged,
Detecting a first feature region for each marker based on the markers in the captured image,
Detecting a second feature region located within the detected first feature region,
Analyze a second feature region of each marker,
Generating a planar image of the target location based on the analyzed second feature region;
And a sequence of instructions for generating a normalized image from the planar image.

Images