KR102593103B1 - Apparatus, method and computer program for aligning of camera - Google Patents

Abstract

A device for aligning a plurality of cameras installed in a shooting location including a first zone, a second zone, and a third zone located between the first zone and the second zone is configured to align the plurality of cameras through multi-camera calibration. It includes a first projection matrix deriving unit that derives a first projection matrix, and at least one camera that photographs the first area among the plurality of cameras based on the derived first projection matrix and a plane constituting the photographing location. a reference normal vector deriving unit for deriving each reference normal vector for the first time slice group and the second time slice group including at least one camera for photographing the second zone among the plurality of cameras, the derived A correction unit that sets each reference normal vector as a reference axis to perform correction for the plurality of cameras, and derives a second projection matrix by which each camera will be aligned for the corrected plurality of cameras. and an alignment unit that sorts images captured from the plurality of cameras based on the derived first and second projective matrices.

Description

Apparatus, method and computer program for aligning a camera {APPARATUS, METHOD AND COMPUTER PROGRAM FOR ALIGNING OF CAMERA}

The present invention relates to devices, methods, and computer programs for aligning cameras.

The time slice technique is to set up multiple cameras facing the subject at various angles, take photos at the same time, and then connect the photos using a computer to make the still movement of the subject look like it was taken with a movie camera. This refers to a video technique that uses Time slice not only allows you to depict a subject three-dimensionally, but also provides a feeling that transcends time and space.

In relation to this time slice technique, Korean Patent Publication No. 2007-0000994, a prior art, discloses a system and method for recording and reproducing images.

In order to provide time slice images, preliminary preparation work such as camera alignment is required. This allows the cameras to be aligned by adjusting the positions of the cameras for each of the plurality of cameras provided in the sports stadium and performance hall.

Conventionally, in order to provide time slice images, a plurality of cameras are aligned so that the centers of the plurality of cameras are reference points located within the shooting location. However, this method has the disadvantage of being suitable when a major event occurs in one area and not suitable when an event occurs in two opposing areas.

A first time slice group containing at least one camera for capturing a first area, a second time slice group containing at least one camera for capturing a second area, and at least one camera for capturing a third area. The present invention seeks to provide an apparatus, method, and computer program for aligning cameras that perform alignment on images captured in an intermediate group.

For a plurality of cameras, derive a first projection matrix before each camera is aligned and a second projection matrix in which each camera will be aligned, and images captured from the plurality of cameras based on the first projection matrix and the second projection matrix It is intended to provide an apparatus, method, and computer program for aligning a camera that performs alignment for.

However, the technical challenges that this embodiment aims to achieve are not limited to the technical challenges described above, and other technical challenges may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention includes a first projection matrix deriving unit for deriving a first projection matrix for the plurality of cameras through multi-camera calibration, and the derived first projection matrix. A first time slice group including at least one camera for photographing the first zone among the plurality of cameras based on a matrix and a plane constituting the shooting location, and at least one camera for photographing the second zone among the plurality of cameras A reference normal vector derivation unit that derives each reference normal vector for a second time slice group including one camera, and sets each derived reference normal vector as a reference axis to perform correction for the plurality of cameras. a correction unit, a second projection matrix derivation unit that derives a second projection matrix by which each camera will be aligned for the plurality of corrected cameras, and a plurality of projection matrices based on the derived first projection matrix and the second projection matrix. A camera alignment device including an alignment unit that performs alignment of images captured from a camera may be provided.

Another embodiment of the present invention includes deriving a first projection matrix for the plurality of cameras through multi-camera calibration, deriving a first projection matrix for the plurality of cameras based on the derived first projection matrix and a plane constituting the shooting location. Each reference for a first time slice group including at least one camera for photographing the first zone and a second time slice group including at least one camera for photographing the second zone among the plurality of cameras. Deriving a normal vector, performing correction for the plurality of cameras by setting each of the derived reference normal vectors as a reference axis, a second projection in which each camera is aligned with respect to the calibrated plurality of cameras A camera alignment method may be provided, including deriving a matrix and aligning images captured from the plurality of cameras based on the derived first and second projective matrices.

In another embodiment of the present invention, when executed by a computing device, the computer program derives a first projection matrix for the plurality of cameras through multi-camera calibration, and uses the derived first projection matrix and the shooting location. A first time slice group including at least one camera for photographing the first zone among the plurality of cameras based on the constituting plane and at least one camera for photographing the second zone among the plurality of cameras Derive each reference normal vector for the second time slice group, set each derived reference normal vector as a reference axis, perform correction for the plurality of cameras, and perform correction for each of the plurality of corrected cameras. Contains a sequence of instructions for deriving a second projection matrix by which the cameras of will be aligned, and performing alignment on images captured from the plurality of cameras based on the derived first projection matrix and the second projection matrix. A computer program stored on a medium may be provided.

The above-described means for solving the problem are merely illustrative and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description of the invention.

According to one of the above-described means of solving the problem of the present invention, a plurality of cameras were conventionally placed in the stadium around a single reference point, but the arrangement and alignment of the cameras were divided into a time slice group and an intermediate group to align them so that they were centered on a single reference point. Apparatus, methods, and computer programs for aligning cameras can be provided that enable problems arising through camera placement to be resolved.

Cameras of the first time slice group and the second time slice group are placed for events that occur in opposite areas, such as a ball game sports game, and an area located between the first time slice group and the second time slice group. It is possible to provide an apparatus, method, and computer program for arranging cameras so that all event occurrence areas occurring in a stadium can be captured by placing an intermediate group of cameras in the stadium.

An apparatus, method, and computer for arranging cameras that enable the creation of multi-view or time-slice images with improved immersion by capturing game images through a single connected camera arrangement using cameras included in a time slice group and an intermediate group. Programs can be provided.

1A and 1B are exemplary diagrams showing camera arrangements for providing conventional time slice images.
Figure 2 is an exemplary diagram showing a shooting location divided into a first zone, a second zone, and a third zone to provide a time slice image according to an embodiment of the present invention.
Figure 3 is a configuration diagram of a camera alignment device according to an embodiment of the present invention.
FIG. 4 is an exemplary diagram illustrating a process for deriving reference normal vectors for a first time slice group and a second time slice group according to an embodiment of the present invention.
5A to 5C are exemplary diagrams for explaining a process of correcting the center point of each of a plurality of cameras according to an embodiment of the present invention.
Figure 6 is an exemplary diagram for explaining the process of calculating each reference point based on the reference axis according to an embodiment of the present invention.
7A and 7B are exemplary diagrams for explaining a process for correcting the positions of each of a plurality of cameras according to an embodiment of the present invention.
FIGS. 8A and 8B are exemplary diagrams showing the arrangement of each camera included in a time slice group and an intermediate group according to an embodiment of the present invention.
Figure 9 is a flowchart of a method for aligning cameras in a camera alignment device according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this does not mean excluding other components unless specifically stated to the contrary, but may further include other components, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware.

In this specification, some of the operations or functions described as being performed by a terminal or device may instead be performed on a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed on a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

1A and 1B are exemplary diagrams showing camera arrangements for providing conventional time slice images.

Referring to FIG. 1A, conventionally, a plurality of cameras 110 are arranged in a hemispherical shape at a shooting location (e.g., a sports stadium, a performance hall, etc.) to provide a time slice image, and the plurality of cameras 110 are arranged in a single Images were captured aligned around the reference point (100). At this time, the main shooting area of each camera is formed around a single reference point 100.

This camera alignment method using a single reference point 100 is suitable for sports in which events occur in one main shooting area, such as kendo, taekwondo, wrestling, judo, etc.

Referring to FIG. 1B, a plurality of cameras 110 are arranged in a hemispherical shape at a shooting location to provide time slice images in a ball game, and the plurality of cameras 110 are aligned around a single reference point 100. In this case, the plurality of cameras 110 can film a ball game centered on the main filming area 120.

However, most ball games, such as soccer, volleyball, basketball, badminton, tennis, handball, and American football, occur in areas where the main event occurrence areas 121 and 122 are opposite to each other. For example, when the plurality of cameras 110 capture the main event occurrence area 122 where team A is located, the main event occurrence area 121 where team B is located cannot be captured.

As such, when a plurality of cameras 110 are aligned around a single reference point 100 in a ball game, the stadium is relatively large compared to the camera's angle of view, or the shooting distance to secure the camera's angle of view is not sufficient. Therefore, it had the disadvantage that it was difficult for the plurality of cameras 110 to capture images including all of the major event occurrence areas 121 and 122.

Figure 2 is an exemplary diagram showing a shooting location divided into a first zone, a second zone, and a third zone to provide a time slice image according to an embodiment of the present invention. Referring to FIG. 2, the plurality of cameras 200 include a first time slice group 201 including at least one camera for photographing the first zone 211 of the shooting location, and a second zone 212 for photographing the second zone 212. It may be divided into a second time slice group 202 including at least one camera and an intermediate group 203 including at least one camera for photographing the third area 213. Here, the filming location 210 (e.g., a sports stadium for a ball game) is a first zone 211, a second zone 212, and a first zone located between the first zone 211 and the second zone 212. It can be divided into 3 zones (213).

For example, at least one camera included in the first time slice group 201 is arranged to look at one reference point 220 so as to photograph the first zone 211, which is one of the main event occurrence areas, and At least one camera included in the 2 time slice group 202 may be arranged to look at one reference point 221 so as to photograph the second zone 212, which is another one of the main event occurrence areas.

Here, at least one camera included in the middle group 203 operates in the first zone 211 and the second zone 212 so that the images of the first time slice group 201 and the second time slice group 202 are smoothly connected. ), and is arranged to photograph the third area 213 between ), at least one camera may be arranged so that a predetermined divided position is each reference point.

Figure 3 is a configuration diagram of a camera alignment device according to an embodiment of the present invention. Referring to FIG. 3, the camera alignment device 300 includes a first projection matrix derivation unit 310, a reference normal vector derivation unit 320, a correction unit 330, a second projection matrix derivation unit 340, and an alignment unit. It may include (350).

The first projection matrix deriving unit 310 may derive a first projection matrix for a plurality of cameras through multi-camera calibration. The first projective matrix can be derived through Equation 1 below.

Referring to Equation 1, the first projection matrix (PC _i ) can be derived based on the rotation matrix (RC _i ), the center point transformation matrix (tC _i ), and the first internal parameter (KC _i ).

The reference normal vector derivation unit 320 includes a first time slice group including at least one camera for photographing the first zone among the plurality of cameras based on the derived first projection matrix and the plane constituting the photographing location, and a plurality of Each reference normal vector for the second time slice group including at least one camera that photographs the second area among the cameras may be derived. The process of deriving the reference normal vector will be explained in detail with reference to FIG. 4.

FIG. 4 is an exemplary diagram illustrating a process for deriving reference normal vectors for a first time slice group and a second time slice group according to an embodiment of the present invention. Referring to FIG. 4, the reference normal vector derivation unit 320 may derive at least three feature points on the plane of the first zone and the second zone. To this end, the reference normal vector derivation unit 320 may restore three points on a plane within the shooting location in the first time slice group and the second time slice group into three-dimensional coordinates. Points corresponding to the two dimensions required for restoration of three-dimensional coordinates can be used as feature points that originally exist, such as lines marked within the shooting location, and can also be extracted through separately arranged patterns, if necessary. The reference normal vector derivation unit 320 can extract feature points from images of a certain camera without the need to extract images from all cameras.

The reference normal vector derivation unit 320 may derive a first normal vector perpendicular to the plane of the first zone and a second normal vector perpendicular to the plane of the second zone based on each derived feature point. For example, the reference normal vector derivation unit 320 derives three feature points (401, 402, 403) for the first zone, and based on the three derived feature points, a first feature perpendicular to the plane of the first zone The normal vector 410 can be derived. In addition, the reference normal vector derivation unit 320 derives three feature points (421, 422, 423) for the second zone, which is the opposite zone to the first zone, and based on the three derived feature points, the A second normal vector 430 perpendicular to the plane can be derived.

The reference normal vector derivation unit 320 calculates a unit vector 440 for each of the first normal vector 410 and the second normal vector 430, and calculates the unit vector 440 based on the average between the calculated unit vectors 440. A reference normal vector 450 can be derived. The reference normal vector 450 can be derived through Equation 2 below.

For example, the reference normal vector derivation unit 320 derives the first unit vector for the first normal vector 410 of the first time slice group and the second normal vector 430 of the second time slice group. A second unit vector may be derived, and a reference normal vector 450 may be derived based on the average of the derived first unit vector and the second unit vector.

The reason for using the average of the first unit vector and the second unit vector to derive the reference normal vector 450 is that the first normal vector 410 of the first time slice group and the second normal vector of the second time slice group This is because (430) was restored through measurement of each local corresponding point, there may be differences between the normal vectors.

Returning to FIG. 3 , the correction unit 330 may perform correction for a plurality of cameras by setting each derived reference normal vector as a reference axis.

The second projection matrix deriving unit 340 may derive a second projection matrix by which each camera will be aligned for the plurality of corrected cameras.

The correction unit 330 may correct the center point of each of the plurality of cameras. The process of correcting the center point for each of the plurality of cameras will be described in detail with reference to FIGS. 5A to 5C.

5A to 5C are exemplary diagrams for explaining a process of correcting the center point of each of a plurality of cameras according to an embodiment of the present invention.

Referring to FIG. 5A , the correction unit 330 may project the center point (Co _i ) of each of the plurality of cameras onto the derived reference normal vector (510, Nv). At this time, the correction unit 330 can determine the height (h _i , 530) of each center point (Co _i , 503) projected on the reference normal vector 510.

For example, the correction unit 330 projects the center point (501, Co ₀ ) of at least one camera included in the first time slice group onto the derived reference normal vector (510, Nv) and the second time slice group At least one center point (502, Co ₁ ) included in can be projected onto the derived reference normal vector (510, Nv). At this time, the center point (501, Co ₀ ) of at least one camera included in the first time slice group is located at the height of 'h ₀ ' (520) on the reference normal vector 510 and is included in the second time slice group. The center point (502, Co ₁ ) of at least one camera may be located at a height of 'h ₁ ' (521) on the reference normal vector 510.

Referring to FIG. 5B, the correction unit 330 may define a height standard based on the height distribution of the center point of each projected camera. For example, the height reference (h _new , 542) can be defined to be located between h _min (540) and h _max (541) on the reference normal vector 510 based on the height distribution for the center point of each camera. .

Referring to FIG. 5C, the correction unit 330 may correct the center point (Co _i ) of each camera according to a defined height standard. The correction unit 330 can correct the center point of each camera using Equation 3 below.

For example, when the height standard is defined as 'h _new ' (542), the correction unit 330 changes the center point (Co _i , 503) of each unaligned camera to the center point (Cn _i , 550) of each aligned camera. ) can be corrected to be.

Through this process, the centers of all cameras included in the first time slice group, the second time slice group, and the middle group are located at the same height based on the reference normal vector (Nv, 510), so that the centers of all cameras are It can be corrected.

Returning to FIG. 3 , the correction unit 330 may calculate a reference point for at least one camera included in the first time slice group, the second time slice group, and the middle group. The process of calculating a reference point for at least one camera included in each group will be described in detail with reference to FIG. 6.

Figure 6 is an exemplary diagram for explaining the process of calculating each reference point based on the reference axis according to an embodiment of the present invention. Referring to FIG. 6, the correction unit 330 calculates a first reference point 601 perpendicularly from the center point 600 of at least one camera included in the first time slice group based on the set reference axis, and A second reference point 611 obtained by a perpendicular line from the center point 610 of at least one camera included in the 2 time slice group can be calculated.

The correction unit 330 is the center point of each of at least one camera included in the middle group including at least one camera for photographing the third area based on the line segment 620 connecting the calculated first and second reference points. At least one third reference point 631 can be calculated from 630 to a perpendicular line. The correction unit can calculate the first to third reference points 601, 611, and 631 using Equation 4 below.

The second projection matrix deriving unit 340 may derive a rotation matrix in which each of the plurality of cameras will be aligned using the set reference axis and the calculated first to third reference points 601, 611, and 631.

For example, the reference normal vector (Nv) becomes yn _i , the new y-axis calibrated for all cameras, the reference point (Dn _i ) of the middle group corresponding to the camera center point (Cn _i ), and the first and second time slices. The z-axis corrected as a unit vector in the direction connecting the reference point (Dts _i ) of the group can be zn _i . At this time, a new corrected x-axis component can be derived by cross producting the new z-axis and the new y-axis (zXy=xn _i ). If these x-axis, y-axis, and z-axis components are arranged in rows in order to create a 3x3 matrix, a new rotation matrix can be derived as shown in Equation 5.

Returning to FIG. 3 , the correction unit 330 may correct the position of each of the plurality of cameras based on the calculated at least one third reference point and the center point of each camera included in the middle group. The process of correcting the positions of each of the plurality of cameras will be described in detail with reference to FIGS. 7A and 7B.

7A and 7B are exemplary diagrams for explaining a process for correcting the positions of each of a plurality of cameras according to an embodiment of the present invention.

Referring to FIG. 7A, the first reference point (Dts _{i -1} ) and the second reference point (Dts _i ) are connected to a perpendicular line from the center point (Cn) of each of at least one camera 700 included in the middle group based on the line segment. When at least one third reference point (Dn) is calculated, the distance from the center point (Cn) of each of the at least one camera 700 included in the middle group to each third reference point (Dn) lowered to the perpendicular line is It may vary.

For example, the distance from 'Dn _{i -1} '(712), one of the third reference points, to 'Cn _{i -1} '(701), one of the center points of the camera, is 'M _{i -1} ', and the third reference point The distance from 'Dn _i ' (713), another of the camera's center points, to 'Cn _i ' (702), another of the camera's center points, is 'M _i ', and 'Dn _{i +1} ', another of the third reference points. As in the case where the distance from 714 to 'Cn _{i +1} ' (703), which is another one of the center points of the camera, is 'M _{i +1} ', a plurality of cameras included in the middle group and each third reference point The distance to varies as 'M _{i -1} ', 'Mi', and 'M _{i + 1} ', respectively.

In this way, as the center point of each camera included in the middle group and at least one third reference point perpendicular from the center point of each camera change, the focal length and angle of view of at least one camera included in the middle group may vary. there is.

Referring to FIG. 7B, the correction unit 330 may move the position of each camera so that the center point of all cameras has a constant distance from each reference point. For example, the correction unit 330 may set the shortest distance among the distances between each camera and the reference point as the reference distance (M) for the camera to move, and move the position of the camera based on the set reference distance. For another example, the correction unit 330 may move the position of the camera using Equation 6 below.

Referring to Equation 6, the correction unit 330 uses the camera center point (Cn _i ) and the third reference point (Dn _i ) to derive the new camera center point (Cdn _i ) as the location where the camera will move, and the derived The position of each camera can be moved based on the center point (Cdn _i ) of the new camera.

Returning to FIG. 3, the second projection matrix deriving unit 340 is configured to correct the focal distance for each of the plurality of cameras based on the first internal parameter (KP _i ) constituting the first projection matrix (PC _i ). The second internal parameter (KN _i ) can be derived. The second projective matrix deriving unit 340 can derive the second internal parameter (KN _i ) using Equation 7 below.

Referring to Equation 7, in order to correct the focal length (fn) for each of the plurality of cameras, the second projection matrix deriving unit 340 calculates the first internal parameter (KCi) before performing correction as the largest value. The value proportional to can be determined as the focal length (fn).

Through this, the distance from all cameras to the target object becomes the same, and by applying the same focal length to all cameras, it is possible to ensure that all cameras have the same angle of view.

The second projective matrix deriving unit 340 may derive the second projective matrix ₍ PN i ) based on the second internal parameter (KN _i ) and the rotation matrix (RN _i ). The second projective matrix deriving unit 340 can derive the second projective matrix (PN _i ) using Equation 8 below.

Referring to Equation 8, the second projection matrix (PNi) can be derived based on the second internal parameter (KN _i ), the rotation matrix (RN _i ), and the center point transformation matrix (tN _i ). Here, the center point transformation matrix (tN _i ) may be the inverse transformation of the rotation matrix to the center point.

The alignment unit 350 may perform alignment of images captured from a plurality of cameras based on the derived first projection matrix (PC _i ) and second projection matrix (PN _i ).

The alignment unit 350 may include a transformation matrix calculation unit 355.

The transformation matrix calculation unit 355 may calculate a movement transformation matrix based on the relationship between the derived first projective matrix (PC _i ) and the second projective matrix (PN _i ). The movement transformation matrix can be calculated using Equation 9 below.

The transformation matrix calculation unit 355 projects the first and second reference points (Dts _i ) and the third reference point (Dn _i ) onto the first projection matrix (PC _i ) and the second projection matrix (PN _i ), and the projected first A shift transformation matrix (H _T ) that corrects the difference between the first projective matrix (PC _i ) and the second projective matrix (PN _i ) can be calculated. At this time, the alignment unit 350 may perform movement transformation for each camera based on the calculated movement transformation matrix (H _T ).

The transformation matrix calculation unit 355 may calculate the rotation transformation matrix (H _R ) based on the relationship between the derived first projective matrix (PC _i ) and the second projective matrix (PN _i ). The transformation matrix calculation unit 355 can calculate the rotation transformation matrix (H _R ) using the following equations 10 and 11.

Referring _{to Equation} 10 _{, a} certain point Assume that the image is projected to coordinates (u', v', 1) after correction by . When the movement of PC _i and PN _i is corrected by the movement transformation matrix, it can be expressed as Equation 10.

Referring to Equation 10 and Equation 11, if each equation derived from Equation 10 is organized with respect to X _w , the rotation transformation matrix H _R can be expressed as Equation 11.

The alignment unit 350 may perform rotation transformation for each camera based on the calculated rotation transformation matrix (H _R ).

The conversion matrix calculation unit 355 may calculate a size conversion matrix ( _HS ) so that the image size of each of the plurality of cameras is converted to a specific resolution. The transformation matrix calculation unit 355 can calculate the size transformation matrix ( _HS ) using Equation 11 below.

The alignment unit 350 may perform size conversion on the images from each camera based on the calculated size conversion matrix.

The alignment unit 350 uses an integrated transformation matrix (H _Total =H _S H _R H _T ) generated by integrating the movement transformation matrix (H _T ), rotation transformation matrix (H _R ), and size transformation matrix (H _S ). You can manage information about movement transformation, rotation transformation, and size transformation. For example, the integrated transformation matrix may be structured in real number form 2x2 or 3x3. By managing information about translation, rotation, and size conversion through this integrated transformation matrix, the efficiency of translation, rotation, and size conversion processing is improved, and images can be corrected in real time when acquiring time slice images. It can provide advantages.

This camera alignment device can be executed by a computer program stored on a medium containing a sequence of instructions for aligning the camera. When executed by a computing device, the computer program derives a first projection matrix for a plurality of cameras through multi-camera calibration, and uses a first projection matrix among the plurality of cameras based on the derived first projection matrix and a plane constituting the shooting location. Derive each reference normal vector for a first time slice group including at least one camera for photographing a zone and a second time slice group including at least one camera for photographing a second zone among a plurality of cameras, Calibrating a plurality of cameras is performed by setting each derived reference normal vector as a reference axis, deriving a second projection matrix in which each camera will be aligned for the calibrated plurality of cameras, and deriving a first projection matrix and a sequence of instructions for performing alignment on images captured from a plurality of cameras based on a second projection matrix.

FIGS. 8A and 8B are exemplary diagrams showing the arrangement of each camera included in a time slice group and an intermediate group according to an embodiment of the present invention.

Referring to Figure 8a, in order to generate a time slice image suitable for a ball game, a time slice group is set on the left and right sides of the stadium, and a middle group is set on the upper and lower sides of the stadium, and are included in the time slice group and middle group. At least one camera can be placed and aligned.

For example, after setting the first time slice group 800 in the A team area of the stadium and the second time slice group 801 in the B team area, the first time slice group 800 and the second At least one camera included in the time slice group 801 can be placed. In addition, at least one included in the first middle group 810 and the second middle group 811 based on the line segment 812 connecting the first time slice group 800 and the second time slice group 801. Cameras can be placed.

Thereafter, the images for the cameras included in the first time slice group 800, the second time slice group 801, the first intermediate group 811, and the second intermediate group 811 are aligned to be suitable for the ball game. You can.

Referring to Figure 8b, in order to generate a time slice image suitable for track competition, a time slice group is set at each corner of the stadium, and a middle group is set at the remaining upper, lower, left, and right sides, and the time slice group and At least one camera included in the middle group can be placed and aligned.

For example, the first time slice group 820 is set in the upper left area of the track stadium, the second time slice group 821 is set in the upper right area, and the third time slice group 822 is set in the lower right area. ), and after setting the fourth time slice group 823 in the lower left area, at least one camera included in the first time slice group 820 to the fourth time slice group 823 can be placed. there is.

In addition, a first middle group 830 is set in the upper area based on the line segment 840 connecting the first time slice group 820 to the fourth time slice group 823, and a second middle group is set in the right area. After setting 831, setting the third middle group 832 in the lower area, and setting the fourth middle group 833 in the left area, the first middle group 830 to the fourth middle group 833 ) can be placed at least one camera included in.

Afterwards, the images for the cameras included in the first time slice group 820 to the fourth time slice group 823 and the first intermediate group 831 to the fourth intermediate group 833 are aligned to be suitable for the track competition. You can.

As such, there is no limit to the number of time slice groups and intermediate groups, and various application arrangements are possible depending on the characteristics of the game, such as ball games, track games, etc.

Figure 9 is a flowchart of a method for aligning cameras in a camera alignment device according to an embodiment of the present invention. The method of aligning cameras in the camera alignment device 300 shown in FIG. 9 includes steps processed in time series according to the embodiments shown in FIGS. 3 to 8B. Therefore, even if the content is omitted below, it also applies to the method of aligning the camera in the camera alignment device 300 according to the embodiment shown in FIGS. 3 to 8B.

In step S910, the camera alignment device 300 may derive a first projection matrix for a plurality of cameras through multi-camera calibration.

In step S920, the camera alignment device 300 includes a first time slice group including at least one camera for photographing a first area among a plurality of cameras based on the derived first projection matrix and a plane constituting the photographing location, and a plurality of first time slice groups. Each reference normal vector for the second time slice group including at least one camera that photographs the second area among the cameras of can be derived.

In step S930, the camera alignment device 300 may perform correction for a plurality of cameras by setting each derived reference normal vector as a reference axis.

In step S940, the camera alignment device 300 may derive a second projection matrix by which each camera will be aligned for the plurality of calibrated cameras.

In step S950, the camera alignment device 300 may perform alignment of images captured from a plurality of cameras based on the derived first and second projection matrices.

In the above description, steps S910 to S950 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present invention. Additionally, some steps may be omitted or the order between steps may be switched as needed.

The method of aligning the camera in the camera alignment device described with reference to FIGS. 2 to 9 may also be implemented in the form of a computer program stored on a medium executed by a computer or a recording medium containing instructions executable by a computer. Additionally, the method of aligning cameras in the camera alignment device described with reference to FIGS. 2 to 9 may also be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

300: Camera alignment device
310: first projective matrix derivation unit
320: Reference normal vector derivation unit
330: Correction unit
340: Second projective matrix derivation part
350: Alignment unit
355: Transformation matrix calculation unit

Claims (17)

In a device for aligning a plurality of cameras installed in a shooting location including a first zone, a second zone, and a third zone located between the first zone and the second zone,
a first projection matrix deriving unit that derives a first projection matrix for the plurality of cameras through multi-camera calibration;
A first time slice group including at least one camera for photographing the first zone among the plurality of cameras based on the derived first projection matrix and a plane constituting the shooting location, and the first time slice group among the plurality of cameras a reference normal vector deriving unit for deriving each reference normal vector for a second time slice group including at least one camera for photographing two zones;
a correction unit that performs correction for the plurality of cameras by setting each of the derived reference normal vectors as a reference axis;
a second projection matrix deriving unit that derives a second projection matrix by which each camera will be aligned for the plurality of corrected cameras; and
An alignment unit that sorts images captured from the plurality of cameras based on the derived first and second projection matrices.
A camera alignment device comprising:
According to claim 1,
The reference normal vector derivation unit derives at least three feature points on a plane of the first zone and the second zone, and based on each of the derived feature points, a first normal vector perpendicular to the plane of the first zone and the A camera alignment device for deriving a second normal vector perpendicular to the plane of the second zone.
According to claim 2,
The reference normal vector deriving unit calculates a unit vector for each of the first normal vector and the second normal vector, and derives the reference normal vector based on the average between the calculated unit vectors. Device.
According to claim 3,
The correction unit projects the center point of each of the plurality of cameras onto the derived reference normal vector, and corrects the center point of each camera according to a height standard defined based on a height distribution for the center point of each of the projected cameras. In,camera alignment device.
According to claim 1,
The correction unit calculates a first reference point perpendicularly from the center point of at least one camera included in the first time slice group based on the set reference axis, and calculates a first reference point of at least one camera included in the second time slice group. A camera alignment device that calculates a second reference point perpendicular to the center point.
According to claim 5,
The correction unit moves a perpendicular line from the center point of each of at least one camera included in the middle group including at least one camera for photographing the third area based on a line segment connecting the calculated first reference point and the second reference point. A camera alignment device, wherein the device calculates at least one third reference point.
According to claim 6,
The second projection matrix deriving unit derives a rotation matrix in which each of the plurality of cameras will be aligned using the set reference axis and the calculated first to third reference points.
According to claim 6,
The correction unit corrects the position of each of the plurality of cameras based on the calculated at least one third reference point and the center point of each camera included in the middle group.
According to claim 7,
The second projection matrix deriving unit derives a second internal parameter for correcting the focal distance for each of the plurality of cameras based on the first internal parameter constituting the first projective matrix.
According to clause 9,
The second projection matrix deriving unit derives the second projection matrix based on the second internal parameter and the rotation matrix.
According to claim 10,
The alignment unit further includes a transformation matrix calculation unit that calculates a movement transformation matrix based on the relationship between the derived first and second projective matrices.
According to claim 11,
The transformation matrix calculation unit projects the first to third reference points onto the first projection matrix and the second projection matrix, and corrects the difference between the projected first projection matrix and the second projection matrix. Calculate ,
The alignment unit performs movement transformation for each camera based on the calculated movement transformation matrix.
According to claim 12,
The transformation matrix calculation unit calculates a rotation transformation matrix based on the relationship between the derived first and second projective matrices,
The alignment unit performs rotation transformation for each camera based on the calculated rotation transformation matrix.
According to claim 13,
The conversion matrix calculation unit calculates a size conversion matrix so that the image size of each of the plurality of cameras is converted to a specific resolution,
The alignment unit performs size conversion on the images of each camera based on the calculated size conversion matrix.
According to claim 14,
The alignment unit manages information about the movement transformation, the rotation transformation, and the size transformation through an integrated transformation matrix generated by integrating the movement transformation matrix, the rotation transformation matrix, and the size transformation matrix. .
In the camera alignment device, a method of aligning a plurality of cameras installed in a shooting location including a first zone, a second zone, and a third zone located between the first zone and the second zone,
Deriving a first projection matrix for the plurality of cameras through multi-camera calibration;
A first time slice group including at least one camera for photographing the first zone among the plurality of cameras based on the derived first projection matrix and a plane constituting the shooting location, and the first time slice group among the plurality of cameras Deriving each reference normal vector for a second time slice group including at least one camera capturing two regions;
performing correction for the plurality of cameras by setting each of the derived reference normal vectors as a reference axis;
Deriving a second projection matrix by which each camera will be aligned for the plurality of calibrated cameras; and
Sorting images captured from the plurality of cameras based on the derived first and second projective matrices.
A camera alignment method comprising:
A camera alignment device stored on a medium including a sequence of instructions for aligning a plurality of cameras installed in a shooting location including a first zone, a second zone, and a third zone located between the first zone and the second zone. In a computer program,
When the computer program is executed by a computing device,
Deriving a first projection matrix for the plurality of cameras through multicamera calibration,
A first time slice group including at least one camera for photographing the first zone among the plurality of cameras based on the derived first projection matrix and a plane constituting the shooting location, and the first time slice group among the plurality of cameras Derive each reference normal vector for a second time slice group containing at least one camera capturing two regions,
Perform correction for the plurality of cameras by setting each of the derived reference normal vectors as a reference axis,
Deriving a second projection matrix by which each camera will be aligned for the plurality of calibrated cameras,
A computer program stored in a medium, comprising a sequence of instructions for performing alignment on images captured from the plurality of cameras based on the derived first projection matrix and the second projection matrix.

Images