JP6933110B2 - Camera external parameter estimation method, estimation device and estimation program - Google Patents

Description

The present invention relates to an estimation method, an estimation device and an estimation program for external parameters of a camera.

There is known a technique for generating an image of an arbitrary viewpoint (hereinafter, also referred to as a free viewpoint image) desired by a viewer by using images taken by a plurality of cameras having different viewpoints. When a free-viewpoint image is generated from images taken by a plurality of cameras, calibration for estimating the positional relationship between the cameras is executed in order to know the positional relationship between the cameras (see, for example, Patent Document 1). ).

For example, a multi-viewpoint video expression device that estimates external parameters including a rotation matrix and a translation vector indicating the positional relationship between cameras using a method called SFM (Structure from Motion) has been proposed (for example, Patent Document 2). reference). In the method using SFM, the external parameters of the cameras are estimated based on the correspondence of the feature points extracted from the plurality of images taken by the plurality of cameras.

Further, a method of generating a free-viewpoint image in a sports scene has been proposed (see, for example, Non-Patent Document 1).

Japanese Unexamined Patent Publication No. 2004-235934 Japanese Unexamined Patent Publication No. 2014-27528

Tadashi Kitahara, Yuichi Ota, "Free-viewpoint video generation of sports scenes by fusing multi-viewpoint video", Image Recognition and Understanding Symposium (MIRU2000), (2000)

When generating a free-viewpoint image in a sports scene, for example, using information obtained by simultaneously photographing a calibration board installed in the court with a plurality of cameras in order to estimate the positional relationship between cameras, etc. The external parameters of the camera are measured in advance. The calibration board will be removed before the start of the match after the external parameters of the camera have been measured. Therefore, it is difficult to remeasure the external parameters of the camera during a sports match by the method of photographing the calibration board in advance and measuring the external parameters of the camera. Therefore, when the floor vibrates and the posture of the camera changes due to the movement of the players during the game, the external parameters of the camera are calculated from the values measured in advance, that is, the values of the external parameters used to generate the free viewpoint image. Change. In this case, the quality of the free-viewpoint image is lower than that in the case where the value of the external parameter of the camera during the game does not change from the value measured in advance.

In sports such as basketball and volleyball, players of the same team wear the same uniform. In this case, in the method of estimating the external parameters of the camera based on the correspondence between the images of the feature points extracted from a plurality of images used for generating the free viewpoint image, the feature points indicating a uniform or the like are erroneously corresponded between the images. It may be attached. If the correspondence between the images of the feature points is incorrect, the estimation accuracy of the external parameters of the camera will decrease.

In one aspect, the present invention aims to accurately estimate the external parameters of the camera even if the posture or position of the camera changes during the imaging period.

In one embodiment, an estimation device that estimates external parameters of a plurality of cameras that capture a subject from different positions extracts feature points of the captured space from images acquired from each of the plurality of cameras, and a plurality of images. The skeleton of a person is extracted from the images acquired from each of the cameras of the above, and the feature points and the two images extracted from one of the two images for each pair of two images out of the plurality of images acquired from the plurality of cameras. The first corresponding point between the two images is determined by associating with the feature points extracted from the other of the two images, and each pair of the two images is extracted from the skeleton extracted from one of the two images and the other of the two images. The second correspondence point between the two images is determined in association with the skeleton, and the external parameters of the camera are estimated based on the first correspondence point and the second correspondence point between the two images.

In one aspect, the present invention can accurately estimate the external parameters of the camera even if the posture or position of the camera changes during the imaging period.

It is a figure which shows one Embodiment of the estimation method, the estimation apparatus and the estimation program of the external parameter of a camera. It is a figure which shows an example of the operation of the estimation apparatus shown in FIG. It is a figure which shows another embodiment of the estimation method, the estimation device and the estimation program of the external parameter of a camera. It is a figure which shows an example of the skeleton extracted by the skeleton extraction part shown in FIG. It is a figure which shows an example of the correspondence between images by the estimation apparatus shown in FIG. It is a figure which shows an example of the operation of the estimation apparatus shown in FIG. It is a figure which shows an example of the 1st calculation process shown in FIG. It is a figure which shows another embodiment of the estimation method, the estimation device and the estimation program of the external parameter of a camera. It is a figure which shows an example of the operation of the estimation apparatus shown in FIG. It is a figure which shows an example of the 1st calculation process shown in FIG.

Hereinafter, embodiments will be described with reference to the drawings.

FIG. 1 shows an embodiment of an estimation method, an estimation device, and an estimation program for external parameters of a camera. The estimation device 10 shown in FIG. 1 is included in, for example, the free viewpoint image generation system SYS that generates an image (free viewpoint image) of an arbitrary viewpoint desired by the viewer. In addition to the estimation device 10, the free-viewpoint video generation system SYS includes a plurality of camera CAMs (CAMa, CAMb, CAMc, ..., CAMn) for capturing a common subject from different viewpoints, and a free-viewpoint video generation unit 20. Have. The estimation device 10 performs calibration for estimating the external parameters of the camera CAM including the rotation matrix and the translation vector indicating the positional relationship between the two camera CAMs. For example, the estimation device 10 estimates the external parameters of a plurality of camera CAMs that shoot the subject from different positions, and transfers the estimated external parameters to the free viewpoint image generation unit 20. The free-viewpoint image generation unit 20 generates a free-viewpoint image by using the images taken by the plurality of camera CAMs and the external parameters of the camera CAMs estimated by the estimation device 10. That is, the external parameters of the camera CAM estimated by the estimation device 10 are used to generate the free viewpoint image.

The estimation device 10 is realized by, for example, an information processing device such as a computer, and has a processor 100 such as a CPU (Central Processing Unit) and a memory 1000. The processor 100 and the memory 1000 are connected to the bus BUS.

For example, the processor 100 executes an estimation program (an estimation program for external parameters of the camera CAM) stored in the memory 1000 and controls the operation of the estimation device 10. The estimation program may be stored in a storage device other than the memory 1000 among the storage devices of the estimation device 10, or may be stored in a storage device outside the estimation device 10. Further, the estimation program may be stored in a computer-readable recording medium REC such as a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc), or a USB (Universal Serial Bus) memory. In this case, the estimation program stored in the recording medium REC is transferred from the recording medium REC to the memory 1000 or the like via an input / output interface (not shown) provided in the estimation device 10. The estimation program may be transferred from the recording medium REC to a hard disk (not shown) and then transferred from the hard disk to the memory 1000.

The processor 100 functions, for example, by executing an estimation program stored in the memory 1000, to perform the functions of the feature point extraction unit 200, the skeleton extraction unit 300, the feature point association unit 400, the skeleton association unit 500, and the estimation unit 600. Realize. That is, the estimation device 10 includes a feature point extraction unit 200, a skeleton extraction unit 300, a feature point association unit 400, a skeleton association unit 500, and an estimation unit 600. The feature point extraction unit 200, the skeleton extraction unit 300, the feature point association unit 400, the skeleton association unit 500, and the estimation unit 600 may be realized only by hardware.

The feature point extraction unit 200 sequentially acquires images from each of a plurality of camera CAMs that shoot a subject at a predetermined frame rate. Then, the feature point extraction unit 200 extracts the feature points of the space taken by the camera CAM from each of the images acquired from the plurality of camera CAMs. The feature points of the space photographed by the camera CAM are natural feature points such as the corners of an object. Hereinafter, the feature points of the space photographed by the camera CAM are also referred to as natural feature points.

The skeleton extraction unit 300 acquires an image to be transferred to the feature point extraction unit 200. That is, the images taken by each camera CAM are transferred to the feature point extraction unit 200 and the skeleton extraction unit 300. Then, the skeleton extraction unit 300 extracts the skeleton of the person in the image from each of the images acquired from the plurality of camera CAMs. For example, the skeleton extraction unit 300 extracts the position of a person's joint (hereinafter, also referred to as a skeleton point) in the image as the skeleton of the person.

The feature point associating unit 400 receives feature information indicating natural feature points extracted from each image from the feature point extraction unit 200. Then, the feature point mapping unit 400 extracts from the natural feature points extracted from one of the two images and the other of the two images for each pair of two images out of the plurality of images acquired from the plurality of camera CAMs. The first corresponding point between the two images is determined by associating with the natural feature points. For example, the feature point mapping unit 400 can display the features of the same object between the natural feature points extracted from the image taken by the camera CAMa and the natural feature points extracted from the image taken by the camera CAMb. Pair the indicated natural feature points. As a result, the first corresponding point (pair of natural feature points) between the two images taken by the cameras CAMa and CAMb is specified. Further, for example, the feature point associating unit 400 uses the same object between the natural feature points extracted from the image taken by the camera CAMb and the natural feature points extracted from the image taken by the camera CAMc. Pair the natural feature points that show the feature. As a result, the first corresponding point (pair of natural feature points) between the two images taken by the cameras CAMb and CAMc is specified. Then, the feature point associating unit 400 transfers the information indicating the first corresponding point (pair of natural feature points) between the images to the estimation unit 600.

The skeleton association unit 500 receives skeleton information indicating the skeleton (for example, skeleton points) of a person extracted from each image from the skeleton extraction unit 300. Then, the skeleton association unit 500 includes a skeleton extracted from one of the two images and a skeleton extracted from the other of the two images for each pair of two images among the plurality of images acquired from the plurality of camera CAMs. To determine the second corresponding point between the two images. For example, the skeleton association unit 500 positions the same joint of the same person between the skeleton points extracted from the image taken by the camera CAMa and the skeleton points extracted from the image taken by the camera CAMb. Pair the indicated skeletal points. Thereby, the second corresponding point (pair of skeleton points) between the two images taken by the cameras CAMa and CAMb is specified. Further, for example, the skeleton association unit 500 has the same joint of the same person between the skeleton points extracted from the image taken by the camera CAMb and the skeleton points extracted from the image taken by the camera CAMc. Pair the skeletal points that indicate the position. Thereby, the second corresponding point (pair of skeleton points) between the two images taken by the cameras CAMb and CAMc is specified. In this way, the estimation device 10 specifies a second corresponding point (a pair of skeleton points) in addition to the first corresponding point (a pair of natural feature points) as a corresponding point between the two images. The skeleton association unit 500 transfers information indicating a second corresponding point (pair of skeleton points) between images to the estimation unit 600.

Whether or not they are the same person is determined based on, for example, the length and angle between skeletal points adjacent to each other in each person. The lengths and angles between adjacent skeletal points correspond to the lengths and angles between the joints of a person. Therefore, for example, even in sports scenes such as basketball and volleyball in which a plurality of people (players) wear the same uniform, it is possible to identify a person having similar lengths and angles between the joints of the two images. , The same person can be identified between images. In this case, the accuracy of identifying the same person can be improved as compared with the case where the skeleton point is not used to identify the same person. That is, the estimation device 10 can suppress erroneous association of feature points and the like as compared with the case where the skeleton points are not used to identify the same person.

The estimation unit 600 estimates the external parameters of the camera CAM based on the first correspondence point and the second correspondence point between the two images. For example, the estimation unit 600 selects eight correspondence points from the first correspondence point and the second correspondence point between the two images specified by the feature point association unit 400 and the skeleton association unit 500. Then, the estimation unit 600 calculates the basic matrix including the information of the internal parameter and the external parameter of the camera CAM by executing the 8-point algorithm using the selected eight corresponding points. The internal parameters of the camera CAM are the focal length of the camera CAM, the center coordinates of the image, and the like, and are calculated in advance. In this case, the estimation unit 600 calculates a basic matrix including information on only the external parameters among the internal parameters and the external parameters of the camera CAM from the internal parameters and the basic matrix of the camera CAM. Then, the estimation unit 600 decomposes the basic matrix, calculates an external parameter, and transfers the calculated external parameter to the free viewpoint video generation unit 20.

In this way, the estimation unit 600 is based on, for example, the correspondence of the natural feature points extracted from the plurality of images taken by the plurality of camera CAMs and the correspondence of the skeleton of the person for each frame. Estimate external parameters. Since the estimation device 10 determines the corresponding points between the images using the skeleton in addition to the natural feature points, it is possible to suppress the determination of the corresponding points by erroneously associating the feature points and the like, and the outside of the camera CAM. It is possible to suppress a decrease in parameter estimation accuracy. That is, even if the posture or position of the camera CAM changes during the shooting period, the estimation device 10 can accurately estimate the external parameters of the camera CAM from the image used for generating the free viewpoint image.

The configuration of the estimation device 10 is not limited to the example shown in FIG. For example, the estimation device 10 may include an interface that receives an image taken by the camera CAM and transfers the image received from the camera CAM to the feature point extraction unit 200 and the skeleton extraction unit 300 via the bus BUS. Further, for example, the processor 100 may realize the function of the free viewpoint image generation unit 20 by executing a program for generating a free viewpoint image.

FIG. 2 shows an example of the operation of the estimation device 10 shown in FIG. The operation shown in FIG. 2 is an example of a method of estimating the external parameters of the camera CAM. Further, the program for causing the estimation device 10 such as a computer to execute the operation shown in FIG. 2 is an example of an estimation program for external parameters of the camera CAM. The operation shown in FIG. 2 is executed for each frame of the image captured by the camera CAM. The operation shown in FIG. 2 may be executed every few frames.

In step S10, the feature point extraction unit 200 extracts natural feature points from each of the images acquired from the plurality of camera CAMs.

Next, in step S20, the skeleton extraction unit 300 extracts the skeleton points of the person from each of the images acquired from the plurality of camera CAMs. As a result, for example, when the free-viewpoint image generation system SYS generates a free-viewpoint image of a sports scene, the skeleton points of the players during the match are extracted by the skeleton extraction unit 300.

Next, in step S30, the feature point associating unit 400 associates the natural feature points extracted from the images taken by the cameras CAM adjacent to each other among the natural feature points extracted in step S10 between the images. The first correspondence point of is determined. As a result, the first corresponding point (pair of natural feature points) is specified as the corresponding point between the images taken by the cameras CAM adjacent to each other.

Next, in step S40, the skeleton association unit 500 associates the skeleton points extracted from the images taken by the cameras CAM adjacent to each other among the skeleton points extracted in step S20, and the second skeleton between the images. Determine the corresponding points. As a result, the second corresponding point is specified separately from the first corresponding point (pair of natural feature points) as the corresponding points between the images taken by the cameras CAM adjacent to each other. In this way, since the estimation device 10 uses the skeleton points in addition to the natural feature points to identify the corresponding points between the images, even in a sports scene in which a plurality of athletes wear the same uniform, the skeleton points are not extracted. Compared with, it is possible to suppress erroneous association of people (players).

Next, in step S50, the estimation unit 600 estimates the external parameters of the camera CAM based on the first correspondence point between the images determined in step S30 and the second correspondence point between the images determined in step S40. Compared with the case where the skeleton points are not used for the mapping of the feature points, the estimation device 10 can accurately map the feature points between the images, so that the external parameters of the camera CAM can be estimated accurately for each frame. ..

The operation of the estimation device 10 is not limited to the example shown in FIG. For example, the process of step S20 may be executed before the process of step S10. Alternatively, the process of step S20 may be implemented after the process of step S30.

As described above, in the embodiment shown in FIGS. 1 and 2, the estimation device 10 extracts the skeleton of the person from each of the images acquired from the plurality of camera CAMs in addition to the natural feature points. Then, the estimation device 10 associates the natural feature points extracted from the two images to determine the first corresponding point between the two images. Further, the estimation device 10 associates the skeletons extracted from the two images to determine the second corresponding point between the two images. As a result, the first correspondence point and the second correspondence point are specified as the correspondence points between the two images. Since the estimation device 10 determines the corresponding points between the images using the skeleton in addition to the natural feature points, it is possible to suppress erroneous association of the feature points and the like, and the external parameters of the camera CAM are used by using the wrong corresponding points. Can be suppressed from estimating. As a result, it is possible to suppress a decrease in the estimation accuracy of the external parameters of the camera CAM.

For example, even if the posture or position of the camera CAM changes during the shooting period, the estimation device 10 can estimate the external parameters of the camera CAM from the image acquired after the posture or position of the camera CAM changes. Therefore, the estimation device 10 can accurately estimate the external parameters of the camera CAM even when the posture or position of the camera CAM changes during the shooting period.

FIG. 3 shows another embodiment of a method of estimating external parameters of a camera, an estimation device and an estimation program. Elements that are the same as or similar to the elements described with reference to FIGS. 1 to 2 are designated by the same or similar reference numerals, and detailed description thereof will be omitted. The estimation device 12 shown in FIG. 3 is included in the free viewpoint image generation system SYS together with, for example, a plurality of camera CAMs and a free viewpoint image generation unit 20. The estimation device 12 performs calibration for estimating the external parameters of the camera CAM in the same manner as the estimation device 10 described with reference to FIG. For example, the estimation device 12 estimates the external parameters of a plurality of camera CAMs that shoot the subject from different positions, and transfers the estimated external parameters to the free viewpoint image generation unit 20. The free-viewpoint image generation unit 20 generates a free-viewpoint image by using the images taken by the plurality of camera CAMs and the external parameters of the camera CAMs estimated by the estimation device 12. That is, the external parameters of the camera CAM estimated by the estimation device 12 are used to generate the free viewpoint image. In the example shown in FIG. 3, a plurality of camera CAMs are arranged so as to surround the subject, the camera CAMa is arranged between the camera CAMb and the camera CAMn, and the camera CAMb is arranged between the camera CAMa and the camera CAMc. Be placed.

The estimation device 12 is the same as or similar to the estimation device 10 shown in FIG. 1, except that it has a processor 102 instead of the processor 100 shown in FIG. For example, the estimation device 12 is realized by an information processing device such as a computer, and has a processor 102 and a memory 1000. The processor 102 and the memory 1000 are connected to the bus BUS. Further, the memory 1000 stores the learning data LDP used in the skeleton extraction unit 302 (302a, 302b, 302c, ..., 302n) described later. In FIG. 3, since the training data LDP is transferred from the memory 1000 to each skeleton extraction unit 302 via the bus BUS, a part of the data path between each skeleton extraction unit 302 and the bus BUS is shown by a broken line. The learning data LDP may be stored in a storage device other than the memory 1000 among the storage devices of the estimation device 12, or may be stored in a storage device outside the estimation device 12.

The processor 102 executes, for example, an estimation program (an estimation program for external parameters of the camera CAM) stored in the memory 1000, and controls the operation of the estimation device 12. The estimation program may be stored in a storage device other than the memory 1000 among the storage devices of the estimation device 12, or may be stored in a storage device outside the estimation device 12. Further, the estimation program may be stored in a recording medium REC that can be read by a computer such as a CD-ROM, a DVD, or a USB memory. In this case, the estimation program stored in the recording medium REC is transferred from the recording medium REC to the memory 1000 or the like via an input / output interface (not shown) provided in the estimation device 12. The estimation program may be transferred from the recording medium REC to a hard disk (not shown) and then transferred from the hard disk to the memory 1000.

The estimation device 12 has a plurality of feature point extraction units 202 (202a, 202b, 202c, ..., 202n) and a plurality of skeleton extraction units 302 (302a, 302b, 302c, ..., 302n). Further, the estimation device 12 includes a plurality of feature point association units 402 (402a, 402b, 402c, ..., 402n) and a plurality of skeleton association units 502 (502a, 502b, 502c, ..., 502n). And an estimation unit 602.

For example, by executing the estimation program, the processor 102 may have a plurality of feature point extraction units 202, a plurality of skeleton extraction units 302, a plurality of feature point association units 402, a plurality of skeleton association units 502, and an estimation unit 602. Realize the function. The plurality of feature point extraction units 202, the plurality of skeleton extraction units 302, the plurality of feature point association units 402, the plurality of skeleton association units 502, and the estimation unit 602 may be realized only by hardware.

Each of the plurality of feature point extraction units 202 is provided corresponding to each of the plurality of camera CAMs, and images are sequentially acquired from the corresponding camera CAMs. Then, each feature point extraction unit 202 extracts the feature points (natural feature points) of the space taken by the camera CAM from the images acquired from the corresponding camera CAMs. The plurality of feature point extraction units 202 is an example of a feature point extraction unit that extracts feature points in the captured space from images acquired from each of the plurality of camera CAMs.

Each of the plurality of skeleton extraction units 302 is provided corresponding to each of the plurality of camera CAMs, and images are sequentially acquired from the corresponding camera CAMs. That is, each of the plurality of skeleton extraction units 302 is provided corresponding to each of the plurality of feature point extraction units 202, and acquires an image transferred to the corresponding feature point extraction unit 202. In this way, the images taken by each camera CAM are transferred to the corresponding feature point extraction unit 202 and the skeleton extraction unit 302. Then, each skeleton extraction unit 302 extracts the skeleton of the person in the image from the image acquired from the corresponding camera CAM.

For example, each skeleton extraction unit 302 receives learning data LDP, which is image data of a person, from memory 1000 via a bus BUS. Then, each skeleton extraction unit 302 extracts the position (skeleton point) of the joint of the person in the image as the skeleton of the person by deep learning using the learning data LDP. In addition, each skeleton extraction unit 302 may extract the skeleton points of a person in an image by machine learning or the like other than deep learning. The plurality of skeleton extraction units 302 is an example of a skeleton extraction unit that extracts the skeleton of a person from images acquired from each of the plurality of camera CAMs.

Each of the plurality of feature point mapping units 402 is provided corresponding to a pair of camera CAMs adjacent to each other. That is, each of the plurality of feature point associating units 402 is provided corresponding to the pair of the feature point extraction unit 202. For example, each of the plurality of feature point associating units 402 receives feature information indicating natural feature points from each of the corresponding pair of feature point extraction units 202. Then, the plurality of feature point associating units 402 use the natural feature points extracted from one of the two images and the other of the two images for each pair of two images among the plurality of images acquired from the plurality of camera CAMs. The first corresponding point between the two images is determined by associating with the natural feature points extracted from.

For example, the feature point associating unit 402a is provided corresponding to the pair of the feature point extraction units 202a and 202b. In this case, the feature point associating unit 402a receives the feature information indicating the natural feature points extracted from the image taken by the camera CAMa from the feature point extraction unit 202a, and the nature extracted from the image taken by the camera CAMb. The feature information indicating the feature point is received from the feature point extraction unit 202b. Then, the feature point associating unit 402a is a natural feature point that shows the characteristics of the same object between the natural feature point extracted by the feature point extraction unit 202a and the natural feature point extracted by the feature point extraction unit 202b. To pair. As a result, the first corresponding point (pair of natural feature points) between the two images taken by the cameras CAMa and CAMb adjacent to each other is specified.

Similarly, the feature point associating unit 402b is provided corresponding to the pair of the feature point extraction units 202b and 202c. In this case, the feature point associating unit 402b receives the feature information indicating the natural feature points extracted from the image taken by the camera CAMb from the feature point extraction unit 202b, and the nature extracted from the image taken by the camera CAMc. The feature information indicating the feature point is received from the feature point extraction unit 202c. Then, the feature point associating unit 402b shows the characteristics of the same object between the natural feature points extracted by the feature point extraction unit 202b and the natural feature points extracted by the feature point extraction unit 202c. To pair. As a result, the first corresponding point (pair of natural feature points) between the two images taken by the cameras CAMb and CAMc adjacent to each other is specified.

Further, the feature point associating unit 402n is provided corresponding to the pair of the feature point extraction units 202n and 202a. In this case, the feature point associating unit 402n receives the feature information indicating the natural feature points extracted from the image taken by the camera CAMn from the feature point extraction unit 202n, and the nature extracted from the image taken by the camera CAMa. The feature information indicating the feature point is received from the feature point extraction unit 202a. Then, the feature point associating unit 402n is a natural feature point that shows the characteristics of the same object between the natural feature point extracted by the feature point extraction unit 202n and the natural feature point extracted by the feature point extraction unit 202a. To pair. As a result, the first corresponding point (pair of natural feature points) between the two images taken by the cameras CAMn and CAMa adjacent to each other is specified.

The plurality of feature point mapping units 402 is an example of a feature point mapping unit that determines a first correspondence point between two images for each pair of two images among a plurality of images acquired from a plurality of camera CAMs. Is.

Each of the plurality of skeleton association units 502 is provided corresponding to a pair of camera CAMs adjacent to each other. That is, each of the plurality of skeleton association units 502 is provided corresponding to the pair of skeleton extraction units 302. For example, each of the plurality of skeleton association units 502 receives skeleton information indicating the skeleton (for example, skeleton points) of a person in an image from each of the corresponding pairs of skeleton extraction units 302. Then, the plurality of skeleton association units 502 extracted the skeleton extracted from one of the two images and the other of the two images for each pair of two images among the plurality of images acquired from the plurality of camera CAMs. The second correspondence point between the two images is determined by associating with the skeleton.

For example, the skeleton association unit 502a is provided corresponding to the pair of the skeleton extraction units 302a and 302b. In this case, the skeleton association unit 502a receives skeleton information indicating the skeleton points extracted from the image taken by the camera CAMa from the skeleton extraction unit 302a, and indicates the skeleton points extracted from the image taken by the camera CAMb. The skeleton information is received from the skeleton extraction unit 302b. Then, the skeleton association unit 502a pairs skeleton points indicating the positions of the same joints of the same person between the skeleton points extracted by the skeleton extraction unit 302a and the skeleton points extracted by the skeleton extraction unit 302b. do. As a result, the second corresponding point (pair of skeleton points) between the two images taken by the cameras CAMa and CAMb adjacent to each other is specified.

Similarly, the skeleton association unit 502b is provided corresponding to the pair of the skeleton extraction units 302b and 302c. In this case, the skeleton association unit 502b receives skeleton information indicating the skeleton points extracted from the image captured by the camera CAMb from the skeleton extraction unit 302b, and indicates the skeleton points extracted from the image captured by the camera CAMc. The skeleton information is received from the skeleton extraction unit 302c. Then, the skeleton association unit 502b pairs skeleton points indicating the positions of the same joints of the same person between the skeleton points extracted by the skeleton extraction unit 302b and the skeleton points extracted by the skeleton extraction unit 302c. do. As a result, the second corresponding point (pair of skeleton points) between the two images taken by the cameras CAMb and CAMc adjacent to each other is specified.

Further, the skeleton association unit 502n is provided corresponding to the pair of the skeleton extraction units 302n and 302a. In this case, the skeleton association unit 502n receives skeleton information indicating the skeleton points extracted from the image taken by the camera CAMn from the skeleton extraction unit 302n, and indicates the skeleton points extracted from the image taken by the camera CAMa. The skeleton information is received from the skeleton extraction unit 302a. Then, the skeleton association unit 502n pairs skeleton points indicating the positions of the same joints of the same person between the skeleton points extracted by the skeleton extraction unit 302n and the skeleton points extracted by the skeleton extraction unit 302a. do. As a result, the second corresponding point (pair of skeleton points) between the two images taken by the cameras CAMn and CAMa adjacent to each other is specified.

The plurality of skeleton association units 502 is an example of a skeleton association unit that determines a second corresponding point between two images for each pair of two images among a plurality of images acquired from a plurality of camera CAMs. .. Similar to the estimation device 10, the estimation device 12 specifies a second correspondence point (pair of skeleton points) in addition to the first correspondence point (pair of natural feature points) as a correspondence point between the two images. Therefore, similarly to the estimation device 10, the estimation device 12 can suppress erroneous association of feature points and the like as compared with the case where the skeleton points are not used to identify the same person.

The estimation unit 602 has an external parameter estimation unit 620 (620a, 620b, 620c, ..., 620n) provided corresponding to a pair of camera CAMs adjacent to each other. Therefore, each of the plurality of external parameter estimation units 620 corresponds to each of the plurality of feature point association units 402 and corresponds to each of the plurality of skeleton association units 502.

Each of the plurality of external parameter estimation units 620 receives information indicating a first corresponding point (a pair of natural feature points) between images from the corresponding feature point mapping unit 402, and receives information from the corresponding skeleton mapping unit 502 between images. Receives information indicating the second corresponding point (pair of skeletal points) of. Then, each of the plurality of external parameter estimation units 620 estimates the external parameters of the corresponding camera CAM based on the first corresponding point and the second corresponding point between the two images.

For example, the external parameter estimation unit 620a receives information indicating the first corresponding point between the images captured by the cameras CAMa and CAMb from the feature point associating unit 402a, and the second external parameter estimation unit 620a receives information between the images captured by the cameras CAMa and CAMb. Information indicating the corresponding point is received from the skeleton association unit 502a. Then, the external parameter estimation unit 620a sets the external parameters of the camera CAM including the rotation matrix and the translation vector indicating the positional relationship between the cameras CAMa and CAMb as the first corresponding points between the images captured by the cameras CAMa and CAMb. Estimate based on the second correspondence point.

Similarly, the external parameter estimation unit 620b receives information indicating the first corresponding point between the images captured by the cameras CAMb and CAMc from the feature point associating unit 402b, and receives information indicating the first corresponding point between the images captured by the cameras CAMb and CAMc. 2 Information indicating the corresponding point is received from the skeleton association unit 502b. Then, the external parameter estimation unit 620b sets the external parameters of the camera CAM including the rotation matrix and the translation vector indicating the positional relationship between the cameras CAMb and the CAMc as the first corresponding points between the images captured by the camera CAMb and the CAMc and the first corresponding points. Estimate based on the second correspondence point.

Further, the external parameter estimation unit 620n receives information indicating the first corresponding point between the images taken by the cameras CAMn and CAMa from the feature point associating unit 402n, and the second external parameter estimation unit 620n receives the information between the images taken by the cameras CAMn and CAMa. Information indicating the corresponding point is received from the skeleton association unit 502n. Then, the external parameter estimation unit 620n sets the external parameters of the camera CAM including the rotation matrix and the translation vector indicating the positional relationship between the cameras CAMn and CAMa as the first corresponding points between the images captured by the cameras CAMn and CAMa. Estimate based on the second correspondence point.

Then, the estimation unit 602 optimizes the external parameters of the camera CAM estimated by each external parameter estimation unit 620 by bundle adjustment and transfers them to the free viewpoint video generation unit 20. In this way, the estimation unit 602 estimates the external parameters of the camera CAM based on the first correspondence point and the second correspondence point between the two images.

The configuration of the estimation device 12 is not limited to the example shown in FIG. For example, the estimation device 12 may include an interface that receives an image taken by the camera CAM and transfers the image received from the camera CAM to the feature point extraction unit 202 and the skeleton extraction unit 302 via the bus BUS. Further, for example, the processor 102 may realize the function of the free viewpoint image generation unit 20 by executing a program for generating a free viewpoint image.

FIG. 4 shows an example of the skeleton extracted by the skeleton extraction unit 302 shown in FIG. The skeleton extraction unit 302 extracts the skeleton point BP (black circle shown in FIG. 4) indicating the position of the joint of the person PN in the image IMG by deep learning using the learning data LDP. The method of estimating the position of the joint of the person PN in the image is, for example, Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, “Realtime Multi-Person 2D Pose Optimization using Part Affinity Fields”, in arXiv 1611.08050. , 2016, [Search on September 4, 2017], Internet <URL:
1511929379642_0.pdf
> Is disclosed.

FIG. 5 shows an example of the correspondence between the images IMGa and IMGb by the estimation device 12 shown in FIG. Note that FIG. 5 shows an example in which the estimation device 12 executes the association between the images IMGa and IMGb when the basketball game is photographed by a plurality of camera CAMs to generate a free viewpoint image. The image IMGa is an image taken by the camera CAMa, and the image IMGb is an image taken by the camera CAMb.

The feature point associating unit 402a converts the natural feature points that show the features of the same object between the natural feature points extracted from the image IMGa and the natural feature points extracted from the image IMGb into paired PCs (PC1, PC2). do. For example, the pair PC1 of natural feature points (first correspondence point) between the images IMGa and IMGb indicates the corner of the free throw lane drawn by a white line in the court, and the pair PC2 of natural feature points (first correspondence point). Indicates the corner of the backboard.

Further, the skeleton association unit 502a pairs PP skeleton points BP indicating the positions of the same joints of the same person PN between the skeleton point BP extracted from the image IMGa and the skeleton point BP extracted from the image IMGb. (PP1, PP2, PP3, PP4). For example, the pair PP1 (second correspondence point) of the skeleton point BP between the images IMGa and IMGb indicates the knee of the person PN1 (PN1a, PN1b), and the pair PP2 (second correspondence point) of the skeleton point BP is the person PN2. The elbow of (PN2a, PN2b) is shown. Further, the pair PP3 (second correspondence point) of the skeleton point BP between the images IMGa and IMGb indicates the shoulder of the person PN3 (PN3a, PN3b), and the pair PP4 (second correspondence point) of the skeleton point BP is the person PN4. The knees of (PN4a, PN4b) are shown. The person PN5a in the image IMGa is not associated with the image IMGb because it is not shown in the image IMGb.

Whether or not the person PN in the image IMGa and the person PN in the image IMGb are the same person is determined based on the person information including the joint angle θ of each person PN and the length L between the joints. For example, the skeleton association unit 502a calculates the joint angle θ and the length L between the joints of each person PN in each image IMGa and IMGb based on the skeleton point BP extracted from each image IMGa and IMGb. Then, the skeleton association unit 502a is an image having elements (angle θ, length L) having the same or substantially the same values as the joint angle θ and the length L between the joints of the person PN of interest in the image IMGa. It is determined that the person PN in the IMGb is the same as the person PN of interest.

In the brackets shown in FIG. 5, the knee angle θ0 (θ0a, θ0b) of the person PN1 (PN1a, PN1b), the length L1 (L1a, L1b) from the knee to the hip joint, and the length L2 from the knee to the ankle. An example of calculating (L2a, L2b) is shown. The skeletal point BP0a corresponds to the knee of the person PN1a in the image IMGa. Further, the skeleton point BP1a is a skeleton point BP corresponding to the hip joint of the person PN1a and adjacent to the skeleton point BP0a. The skeleton point BP2a is a skeleton point BP corresponding to the ankle of the person PN1a and adjacent to the skeleton point BP0a. Similarly, the skeletal point BP0b corresponds to the knee of the person PN1b in the image IMGb. Further, the skeleton point BP1b is a skeleton point BP corresponding to the hip joint of the person PN1b and adjacent to the skeleton point BP0b. The skeleton point BP2b is a skeleton point BP corresponding to the ankle of the person PN1b and adjacent to the skeleton point BP0b. The angle θ0a with the line connecting the two is calculated as the knee angle θ0a of the person PN1a. Further, the skeleton association unit 502a calculates the length L1a between the skeleton points BP0a and BP1a as the length L1a from the knee to the hip joint of the person PN1a, and calculates the length L2a between the skeleton points BP0a and BP2a as the person PN1a. It is calculated as the length L2a from the knee to the ankle. Similarly, the skeleton association unit 502a uses the skeleton points BP0b, BP1b, and BP2b to calculate the knee angle θ0b of the person PN1b, the length L1b from the knee to the hip joint, and the length L2b from the knee to the ankle.

The angles θ of the joints other than the knees and the length L between the joints of the persons PN1a and PN1b are also the same as the knee angle θ0, the knee-to-hip length L1 and the knee-to-ankle length L2. It is calculated. As a result, a plurality of angles θ and a plurality of lengths L are calculated for each person PN. The skeleton association unit 502a is, for example, an element (angle) corresponding to each other between the person information having a plurality of angles θ and a plurality of lengths L of the person PN1a as elements and the person information of each person PN in the image IMGb. The sum of the absolute values of the differences of θ and length L) is calculated as the associated evaluation value. Then, the skeleton association unit 502a has a person PN (in the example shown in FIG. 5, which is the smallest association evaluation value among the association evaluation values of the plurality of person PNs in the image IMGb with respect to the person PN1a in the image IMGa). The person PN1b) is determined to be the same as the person PN1a.

Then, the skeleton association unit 502a sets a pair PP of skeleton points BP indicating the corresponding parts among the skeleton point BPs of the persons PN1a and PN1b determined to be the same person between the images IMGa and IMGb, between the images IMGa and IMGb. The pair PP of the skeletal point BP of. That is, the skeleton association unit 502a associates the skeleton point BP extracted from one of the two images IMGa and IMGb with the skeleton point BP extracted from the other of the two images IMGa and IMGb using the person information. A second correspondence point between the two images IMGa and IMGb is determined.

In this way, the skeleton association unit 502a identifies the person PN having a similar joint angle θ and the length L between the joints of the person PN between the two images IMGa and IMGb, thereby between the images IMGa and IMGb. The same person can be identified. Therefore, when a wider range than the space for one person is to be photographed, for example, even in a sports scene such as basketball, the estimation device 12 is the same person as compared with the case where the skeleton point BP is not used to identify the same person. The specific accuracy of can be improved. For example, when the skeleton point BP is not used to identify the same person, since the person PN1 and PN3 wear the same uniform, similar characteristics are extracted between the person PN1 and the person PN3, and the person PN1a and the person PN3b become It may be determined that they are the same person. On the other hand, the estimation device 12 identifies the person PN1a and PN1b having similar joint angles θ and the length L between the joints of the person PN between the two images IMGa and IMGb, so that the person PN1a and the person PN3b are the same. It is possible to prevent erroneous determination as a person. That is, the estimation device 12 can suppress erroneous association of feature points and the like as compared with the case where the skeleton point BP is not used to identify the same person.

The method of associating the images IMGa and IMGb by the estimation device 12 is not limited to the example shown in FIG. For example, in the skeleton association unit 502a, when the minimum association evaluation value of the plurality of person PNs in the image IMGb with respect to the person PN1a in the image IMGb is equal to or more than a predetermined threshold value, the person PN same as the person PN1a is an image. It may be determined that it does not exist in IMGb. Further, the person information may include at least one of the joint angle θ of the person PN and the length L between the joints. Further, the determination of whether or not the person is the same person using the person information is not limited to the method using the absolute value sum of the differences between the elements (angle θ, length L) corresponding to each other between the person information.

FIG. 6 shows an example of the operation of the estimation device 12 shown in FIG. The operation shown in FIG. 6 is an example of a method of estimating the external parameters of the camera CAM. Further, the program for causing the estimation device 12 such as a computer to execute the operation shown in FIG. 6 is an example of an estimation program for external parameters of the camera CAM. The operation shown in FIG. 6 is executed for each frame of the image captured by the camera CAM. The operation shown in FIG. 6 may be executed every few frames.

In step S100, each feature point extraction unit 202 and each skeleton extraction unit 302 acquire an image IMG captured by the corresponding camera CAM among the plurality of camera CAMs. For example, the feature point extraction unit 202a and the skeleton extraction unit 302a acquire the image IMGa taken by the camera CAMa.

Next, in step S200, each feature point extraction unit 202 extracts natural feature points from the image IMG captured by the corresponding camera CAM among the plurality of camera CAMs. For example, the feature point extraction unit 202a extracts natural feature points from the image IMGa.

Next, in step S300, each skeleton extraction unit 302 extracts the skeleton point BP of the person PN from the image IMG captured by the corresponding camera CAM among the plurality of camera CAMs. For example, the skeleton extraction unit 302a extracts the skeleton point BP of the person PN from the image IMGa by deep learning using the learning data LDP.

Next, in step S400, the estimation device 12 executes the first calculation process for calculating the external parameters of the camera CAM for each pair of camera CAMs adjacent to each other. The details of the first calculation process will be described with reference to FIG. By the first calculation process, the external parameters of the camera CAM are calculated for each pair of camera CAMs adjacent to each other. In the first calculation process shown in FIG. 7, the external parameters of the camera CAM that does not capture the person PN are not calculated.

Next, in step S500, the estimation device 12 determines whether or not the first calculation process has been completed for all pairs of camera CAMs adjacent to each other. When the first calculation process is completed for all the pairs of camera CAMs adjacent to each other, the operation of the estimation device 12 shifts to step S600. On the other hand, if the first calculation process is not completed in any pair of all the pairs of camera CAMs adjacent to each other, the operation of the estimation device 12 returns to step S500. That is, the estimation device 12 waits for the execution of the process of step S600 until the first calculation process is completed for all the pairs of camera CAMs adjacent to each other.

In step S600, the estimation unit 602 executes bundle adjustment that optimizes the external parameters of each camera CAM calculated by the first calculation process of step S400.

Next, in step S700, the estimation unit 602 determines the final external parameters of the camera CAM. For example, the estimation unit 602 determines the external parameter of each camera CAM optimized by the bundle adjustment in step S600 as the final external parameter. Then, the estimation unit 602 transfers the final external parameter of the camera CAM to the free viewpoint image generation unit 20. In this way, the estimation device 12 estimates the external parameters of the camera CAM for each frame of the image captured by the camera CAM, and transfers the estimated external parameters of the camera CAM to the free viewpoint image generation unit 20.

Here, in a sports scene, when generating a free-viewpoint image, a calibration board installed in the court is photographed in advance to estimate the positional relationship between the camera CAMs, and the external parameters of the camera CAM are measured. Conventional methods are known. In this type of conventional method, it is difficult to remeasure the external parameters of the camera CAM during the match because the calibration board is removed before the start of the match after the external parameters of the camera CAM are measured. .. Therefore, in the conventional method, when the floor vibrates due to the movement of the player during the game and the posture of the camera CAM changes, the external parameter of the camera CAM is a value measured in advance, that is, a free viewpoint image is generated. It changes from the value of the external parameter used. In this case, the quality of the free-viewpoint image is lower than that in the case where the value of the external parameter of the camera CAM during the game does not change from the value measured in advance.

On the other hand, as described above, the estimation device 12 estimates the external parameters of the camera CAM for each frame of the image captured by the camera CAM. Alternatively, the estimation device 12 estimates the external parameters of the camera CAM every few frames. Therefore, for example, in a sports scene, even if the floor vibrates due to the movement of a player during a match and the posture of the camera CAM changes, the estimation device 12 uses an external parameter of the camera CAM according to the change of the posture or the like. Can be estimated. That is, the external parameters of the camera CAM used to generate the free-viewpoint image can be estimated more accurately than the conventional method. As a result, the quality of the free-viewpoint video can be improved as compared with the conventional method.

The operation of the estimation device 12 is not limited to the example shown in FIG. For example, the process of step S300 may be executed before the process of step S200.

FIG. 7 shows an example of the first calculation process (step S400) shown in FIG. The first calculation process shown in FIG. 7 is the first calculation process for one pair of camera CAMs adjacent to each other. For example, the first calculation process shown in FIG. 7 is executed in parallel for each pair of camera CAMs adjacent to each other. The first calculation process shown in FIG. 7 may be sequentially executed for each pair of camera CAMs adjacent to each other. FIG. 7 describes the first calculation process for the pair of cameras CAMa and CAMb.

In step S410, the skeleton association unit 502a determines whether or not the skeleton point BP of the person PN is extracted from both the images IMGa and IMGb acquired from the pair of cameras CAMa and CAMb. When the skeleton point BP of the person PN is extracted from both the paired images IMGa and IMGb, the operation of the estimation device 12 shifts to step S420. On the other hand, when the skeleton point BP of the person PN is not extracted from at least one of the paired images IMGa and IMGb, the first calculation process for the pair of cameras CAMa and CAMb ends. That is, when the skeleton point BP of the person PN is not extracted from at least one of the paired images IMGa and IMGb, the estimation device 12 does not extract the external parameter indicating the positional relationship between the cameras CAMa and CAMb.

Here, from the viewpoint of generating a free-viewpoint image in a sports scene, it suffices if the free-viewpoint image of the area in which the player (person PN) is captured can be generated accurately. Therefore, in the sports scene, if the area in which the player (person PN) is captured is biased, if calibration (estimation of the external parameter of the camera CAM) can be performed based on the feature points of the area in which the player (person PN) exists. good. That is, the calibration based on the feature points of the region where the person PN does not exist (estimation of the external parameters of the camera CAM) may be omitted. In this case, the amount of calculation for estimating the external parameters of the camera CAM can be reduced as compared with the case where the estimation accuracy of the external parameters of the camera CAM is evenly improved over the entire shooting range of the plurality of camera CAMs.

In step S420, the feature point associating unit 402a associates the natural feature points extracted from the paired images IMGa and IMGb among the natural feature points extracted in step S200 shown in FIG. 6 with the images IMGa and IMGb. Determine the first correspondence point between. For example, as described with reference to FIG. 5, the feature point associating unit 402a shows the features of the same object between the natural feature points extracted from the image IMGa and the natural feature points extracted from the image IMGb. The pair PC of points is determined as the first corresponding point between the images IMGa and IMGb. As a result, the first corresponding point (pair PC of natural feature points) is specified as the corresponding point between the images IMGa and IMGb.

Next, in step S430, the skeleton association unit 502a associates the paired image IMGa and the skeleton point BP extracted from the IMGb among the skeleton point BPs extracted in step S300 shown in FIG. , Determine the second correspondence point between IMGb. For example, as described in FIG. 5, the skeleton association unit 502a is located between the skeleton point BP extracted from the image IMGa and the skeleton point BP extracted from the image IMGb, and the positions of the same joints of the same person PN. The pair PP of the skeletal point BP showing the above is specified. Then, the skeleton association unit 502a determines the pair PP of the specified skeleton point BP as the second corresponding point between the images IMGa and IMGb. As a result, the second correspondence point is specified as the correspondence point between the images IMGa and IMGb separately from the first correspondence point (pair of natural feature points). In this way, the estimation device 12 identifies the corresponding points between the images IMGa and IMGb using the natural feature points and the skeleton point BP. Therefore, even in a sports scene in which a plurality of person PNs (athletes) wear the same uniform, the estimation device 12 can suppress erroneous association of person PNs as compared with the case where the skeleton point BP is not extracted.

Next, in step S440, the external parameter estimation unit 620a calculates the basic matrix using an 8-point algorithm. The basic matrix consists of information on the internal parameters of the camera CAM (focal length of the camera CAM, center coordinates of the image IMG, etc.) and external parameters of the camera CAM (rotation matrix and translation vector indicating the positional relationship between the two camera CAMs). Includes information about.

For example, the external parameter estimation unit 620a selects eight corresponding points from the first corresponding point between the images IMGa and IMGb determined in step S420 and the second corresponding point between the images IMGa and IMGb determined in step S430. .. Then, the external parameter estimation unit 620a calculates the basic matrix including the internal parameters of the camera CAM and the information of the external parameters by executing the 8-point algorithm using the eight selected corresponding points. The external parameter estimation unit 620a may calculate the basic matrix by changing the combination of the eight corresponding points and executing the eight-point algorithm a plurality of times.

Next, in step S450, the external parameter estimation unit 620a decomposes the basic matrix by singular value decomposition (SVD) to calculate the external parameters (rotation matrix, translation vector) of the camera CAM. The elementary matrix contains information on only the external parameters of the internal and external parameters of the camera CAM and indicates the relative position and orientation between the two camera CAMs.

For example, the external parameter estimation unit 620a calculates the basic matrix based on the internal parameters of the known camera CAM and the basic matrix calculated in step S440. Then, the external parameter estimation unit 620a decomposes the basic matrix by singular value decomposition (SVD), and calculates a rotation matrix and a translation vector indicating the relative positions and orientations between the cameras CAMa and CAMb. As a result, the external vector of the camera CAM is calculated. With the end of the process in step S450, the first calculation process for the pair of cameras CAMa and CAMb ends.

As described above, the external parameter estimation unit 620a is outside the camera CAM based on the first correspondence point between the images IMGa and IMGb determined in step S420 and the second correspondence point between the images IMGa and IMGb determined in step S430. Calculate the parameters. Since the external parameter estimation unit 620a accurately identifies the corresponding points between the images IMGa and IMGb as compared with the case where the skeleton point BP is not used for associating the feature points and the like, the external parameter of the camera CAM is set for each frame or for each frame. It can be calculated accurately every few frames.

The first calculation process is not limited to the example shown in FIG. 7. For example, the external parameter estimation unit 620a may calculate the basic matrix using nine or more corresponding points in step S440.

As described above, even in the embodiments shown in FIGS. 3 to 7, the same effects as those in the embodiments shown in FIGS. 1 and 2 can be obtained. For example, the estimation device 12 extracts the skeleton point BP of the person PN from each of the image IMGs acquired from the plurality of camera CAMs in addition to the natural feature points. Then, the estimation device 12 associates the natural feature points extracted from the two image IMGs to determine the first corresponding point between the two image IMGs. Further, the estimation device 12 generates person information including at least one of the joint angle θ of the person PN and the length L between the joints based on the skeleton point BP extracted from the image IMG. Then, the estimation device 12 uses the person information to associate the skeleton point BP extracted from one of the two image IMGs with the skeleton point BP extracted from the other of the two image IMGs, and the second image IMG between the two image IMGs. 2 Determine the corresponding points. As a result, the first correspondence point and the second correspondence point are specified as the correspondence points between the two image IMGs. Since the estimation device 12 determines the corresponding points between the image IMGs using the natural feature points and the skeleton point BP, it is possible to prevent the feature points and the like from being erroneously associated with each other, and the wrong corresponding points are used outside the camera CAM. It is possible to suppress the estimation of parameters. As a result, it is possible to suppress a decrease in the estimation accuracy of the external parameters of the camera CAM.

Further, for example, in the sports scene, even if the posture or position of the camera CAM changes during the game, the estimation device 12 obtains the external parameters of the camera CAM from the image IMG acquired after the posture or position of the camera CAM changes. Can be estimated. Therefore, even if the posture or position of the camera CAM changes, the estimation device 12 can accurately estimate the external parameters of the camera CAM according to the change in the posture or position of the camera CAM. That is, the estimation device 12 can accurately estimate the external parameters of the camera CAM even when the posture or position of the camera CAM changes during the shooting period.

FIG. 8 shows another embodiment of a method of estimating external parameters of a camera, an estimation device and an estimation program. Elements that are the same as or similar to the elements described with reference to FIGS. 1 to 7 are designated by the same or similar reference numerals, and detailed description thereof will be omitted. The estimation device 14 shown in FIG. 8 is included in the free viewpoint image generation system SYS together with, for example, a plurality of camera CAMs and a free viewpoint image generation unit 20. The estimation device 14 performs calibration for estimating the external parameters of the camera CAM in the same manner as the estimation device 12 described with reference to FIG. For example, the estimation device 14 estimates the external parameters of a plurality of camera CAMs that shoot the subject from different positions, and transfers the estimated external parameters to the free viewpoint image generation unit 20. The free-viewpoint image generation unit 20 generates a free-viewpoint image by using the image IMG captured by the plurality of camera CAMs and the external parameters of the camera CAM estimated by the estimation device 14. That is, the external parameters of the camera CAM estimated by the estimation device 14 are used to generate the free viewpoint image. In the example shown in FIG. 8, the plurality of camera CAMs are arranged so as to surround the subject, similarly to the plurality of camera CAMs shown in FIG. In FIG. 8, the description of the camera CAMn and the like shown in FIG. 3 is omitted in order to make the figure easier to see.

The estimation device 14 is the same as or similar to the estimation device 12 shown in FIG. 3, except that it has a processor 104 instead of the processor 102 shown in FIG. For example, the estimation device 14 is realized by an information processing device such as a computer, and has a processor 104 and a memory 1000. The processor 104 and the memory 1000 are connected to the bus BUS. In addition to the learning data LDP, the memory 1000 stores the learning data LDA used in the additional feature extraction units 320 (320a, 320b, ...), Which will be described later.

In FIG. 8, as in FIG. 3, a part of the data path between each skeleton extraction unit 302 and the bus BUS is shown by a broken line. Further, in FIG. 8, since the training data LDA is transferred from the memory 1000 to each additional feature extraction unit 320 via the bus BUS, a part of the data path between each additional feature extraction unit 320 and the bus BUS is used. Shown by a broken line. The learning data LDP and LDA may be stored in a storage device other than the memory 1000 among the storage devices of the estimation device 14, or may be stored in a storage device outside the estimation device 14.

The processor 104 executes, for example, an estimation program (an estimation program for external parameters of the camera CAM) stored in the memory 1000, and controls the operation of the estimation device 14. The estimation program may be stored in a storage device other than the memory 1000 among the storage devices of the estimation device 14, or may be stored in a storage device outside the estimation device 14. Further, the estimation program may be stored in a recording medium REC that can be read by a computer such as a CD-ROM, a DVD, or a USB memory. In this case, the estimation program stored in the recording medium REC is transferred from the recording medium REC to the memory 1000 or the like via an input / output interface (not shown) provided in the estimation device 14. The estimation program may be transferred from the recording medium REC to a hard disk (not shown) and then transferred from the hard disk to the memory 1000.

Similar to the processor 102 shown in FIG. 3, the processor 104 realizes functions such as a plurality of feature point extraction units 202 by executing an estimation program. The processor 104 realizes the functions of the plurality of skeleton association units 504 (504a, 504b, ...) Instead of the plurality of skeleton association units 502 shown in FIG. Further, the processor 104 adds a plurality of additional feature extraction units 320 (320a, 320b, ...) And a plurality of association auxiliary units 520 (520a, 520b) in addition to the functions realized by the processor 102 shown in FIG. , ...) to realize the function. A plurality of feature point extraction units 202, a plurality of skeleton extraction units 302, a plurality of additional feature extraction units 320, a plurality of feature point association units 402, a plurality of skeleton association units 504, a plurality of association auxiliary units 520, and the like. The estimation unit 602 may be realized only by hardware.

As described above, in the estimation device 14, a plurality of skeleton association units 504 are provided in place of the plurality of skeleton association units 502 shown in FIG. 3, a plurality of additional feature extraction units 320 and a plurality of association auxiliary units 520. Is added to the estimation device 12 shown in FIG. Other configurations of the estimation device 14 are the same as or similar to those of the estimation device 12 shown in FIG. For example, the estimation device 14 includes a plurality of feature point extraction units 202, a plurality of skeleton extraction units 302, a plurality of additional feature extraction units 320, a plurality of feature point association units 402, a plurality of skeleton association units 504, and a plurality of correspondences. It has an attachment auxiliary unit 520 and an estimation unit 602.

Each of the plurality of feature point extraction units 202 is the same as or similar to each of the plurality of feature point extraction units 202 shown in FIG. Each of the plurality of skeleton extraction units 302 is the same as or similar to each of the plurality of skeleton extraction units 302 shown in FIG. Each of the plurality of feature point mapping units 402 is the same as or similar to each of the plurality of feature point association units 402 shown in FIG.

Each of the plurality of additional feature extraction units 320 is provided corresponding to each of the plurality of camera CAMs, and sequentially acquires an image IMG from the corresponding camera CAMs. That is, each of the plurality of additional feature extraction units 320 is provided corresponding to each of the plurality of feature point extraction units 202, and acquires an image IMG to be transferred to the corresponding feature point extraction unit 202. In this way, the image IMG captured by each camera CAM is transferred to the corresponding feature point extraction unit 202, the skeleton extraction unit 302, and the additional feature extraction unit 320. Then, each additional feature extraction unit 320 extracts additional feature points indicating the features of the person PN other than the skeleton from the image IMG acquired from the corresponding camera CAM.

The additional feature points include, for example, at least one of a facial feature point indicating the face of the person PN and a clothing feature point indicating the clothing worn on the person PN. Further, the clothing feature point indicates, for example, at least one of the identification information provided on the clothing and identifying the person PN and the shoes worn by the person PN. For example, the identification information is either a number or the name of a person PN. When the ball game is photographed by a plurality of camera CAMs, the additional feature points may include a ball feature point indicating the ball.

Each additional feature extraction unit 320 receives, for example, learning data LDA including image data of the face of the person PN, image data of clothes worn by the person PN, and the like from the memory 1000 via the bus BUS. Then, each additional feature extraction unit 320 includes at least one of a facial feature point indicating the face of the person PN and a clothing feature point indicating the clothing worn on the person PN in the image IMG based on the learning data LDA. Extract feature points.

When the ball game is photographed by a plurality of cameras CAM, each additional feature extraction unit 320 may extract a ball feature point indicating the ball as an additional feature point. For example, each additional feature extraction unit 320 uses the learning data LDA including the image data of the face of the person PN, the image data of the clothing, the image data of the ball, and the like, and the face feature points and the clothing features of the person PN in the image IMG. Points, ball feature points, and the like may be extracted as additional feature points.

Here, the method of recognizing the face of the person PN in the image is, for example, Peiyun Hu, Deva Ramanan, “Finding Tiny Faces”, in arXiv pre-print 1612.04402, 2016, [Search on September 4, 2017], Internet <URL:
1511929379642_1.pdf
> Is disclosed.

Each of the plurality of association auxiliary units 520 is provided corresponding to a pair of camera CAMs adjacent to each other. That is, each of the plurality of association auxiliary units 520 is provided corresponding to the pair of the additional feature extraction units 320. For example, each matching auxiliary unit 520 has additional feature points of the person PN in the image IMG (for example, the face of the person PN, the clothing worn on the person PN, etc.) from each of the pairs of the corresponding additional feature extraction units 320. Receive additional feature information indicating. Then, each association auxiliary unit 520 has an additional feature point indicated by additional information received from one of the pairs of the additional feature extraction unit 320 and an additional feature indicated by the additional information received from the other of the pair of the additional feature extraction unit 320. An additional corresponding point between the two image IMGs is determined by associating the points with each other. For example, the association auxiliary unit 520a has similar features (similar faces, similar features) between the additional feature points extracted by the additional feature extraction unit 320a and the additional feature points extracted by the additional feature extraction unit 320b. Clothes, etc.) are paired. As a result, additional corresponding points (pairs of additional feature points) between the two image IMGs taken by the cameras CAMa and CAMb adjacent to each other are specified. Each of the plurality of association auxiliary units 520 transfers information indicating an additional correspondence point to the corresponding skeleton association unit 504.

Each of the plurality of skeleton mapping units 504 is shown in FIG. 3, except that the additional corresponding points determined by the corresponding mapping auxiliary unit 520 are used to determine the second corresponding point between the two image IMGs. It is the same as or similar to each of the plurality of skeleton association units 502. That is, each of the plurality of skeleton association units 504 was extracted from the skeleton point BP extracted from one of the two image IMGs and the other of the two image IMGs by using the additional correspondence points between the two image IMGs. The second corresponding point between the two image IMGs is determined in association with the skeleton point BP. For example, for each skeleton association unit 504, for the person PN associated with the additional correspondence point between the two image IMGs, the association evaluation value between the associated person PNs (joint angle θ, etc.) Based on the absolute value sum of the differences), it is determined whether or not the person is the same person. In addition, in each skeleton association unit 504, for a person PN in which the association destinations of the additional feature points do not exist with each other, the association evaluation value between the person PNs in which the association destinations of the additional feature points do not exist with each other (joint). It may be determined whether or not the person is the same person based on the absolute value sum of the differences such as the angle θ).

For example, when the uniform number of the person PN5a in the image IMGa and the uniform number of the person PN3b in the image IMGb are different from each other, the additional feature points of the person PN5a and the additional feature points of the person PN3b are not paired. The additional feature points of the person PN3b in the image IMGb are associated with the additional feature points of the person PN3a in the image IMGa having the same number. Even when the joint angle θ and the length L between the joints are similar between the person PN5a and the person PN3b, the additional feature points of the person PN3b are associated with the additional feature points of the person PN3a other than the person PN5a, so that the person PN5a And the person PN3b are not determined to be the same person. In this case, in the skeleton association unit 504a, the additional feature points of the person PN3a and the additional feature points of the person PN3b are associated with each other, and the joint angle θ and the length L between the joints are similar between the person PN3a and PN3b. Therefore, it is determined that the person PN3a and the person PN3b are the same person. By using the additional correspondence points between the images IMGa and IMGb, the skeleton association unit 504a can suppress erroneous determination that the person PN5a and the person PN3b are the same person.

Further, for example, when the ball feature points are extracted in the images IMGa and IMGb shown in FIG. 5, the association auxiliary unit 520a is held by the ball held by the person PN1a in the image IMGa and the person PN1b in the image IMGb. Associate with the ball. As a result, the pair of ball feature points between the images IMGa and IMGb is specified as additional corresponding points between the images IMGa and IMGb. As a result, for example, even if there is a person PN similar to the body shape and posture of the person PN1, the skeleton association unit 504a and the person PN1a in the image IMGa are based on the pair of ball feature points (additional corresponding points). It is possible to associate with the person PN1b in the image IMGb.

In this way, the estimation device 14 uses the additional feature points to determine the second corresponding point between the two image IMGs.

The estimation unit 602 is the same as or similar to the estimation unit 602 shown in FIG. That is, each of the plurality of external parameter estimation units 620 included in the estimation unit 602 is the same as or similar to each of the plurality of external parameter estimation units 620 shown in FIG.

The configuration of the estimation device 14 is not limited to the example shown in FIG. For example, an interface that receives an image IMG taken by the camera CAM and transfers the image IMG received from the camera CAM to the feature point extraction unit 202, the skeleton extraction unit 302, and the additional feature extraction unit 320 via the bus BUS is an estimation device. 14 may be included. Further, the additional feature extraction unit 320 may execute a process for extracting a ball feature point indicating the ball regardless of whether or not the ball game is photographed by the plurality of camera CAMs. Further, for example, the processor 104 may realize the function of the free viewpoint image generation unit 20 by executing a program for generating a free viewpoint image.

FIG. 9 shows an example of the operation of the estimation device 14 shown in FIG. The operation shown in FIG. 9 is an example of a method of estimating the external parameters of the camera CAM. Further, the program for causing the estimation device 14 such as a computer to execute the operation shown in FIG. 9 is an example of an external parameter estimation program of the camera CAM. The operation shown in FIG. 9 is executed for each frame of the image captured by the camera CAM. The operation shown in FIG. 9 may be executed every few frames. Steps that are the same as or similar to those described in FIG. 6 are designated by the same or similar reference numerals, and detailed description thereof will be omitted.

The operation shown in FIG. 9 is the same as or similar to the operation shown in FIG. 6, except that the first calculation process of step S402 is executed instead of the first calculation process of step S400 shown in FIG. .. The first calculation process in step S402 is the same as or similar to the first calculation process in step S400 shown in FIG. 6, except that additional feature points are used to determine the second corresponding point between the two image IMGs. .. Details of the first calculation process in step S402 will be described with reference to FIG.

FIG. 10 shows an example of the first calculation process (step S402) shown in FIG. The first calculation process shown in FIG. 10 is the first calculation process for one pair of camera CAMs adjacent to each other. For example, the first calculation process shown in FIG. 10 is executed in parallel for each pair of camera CAMs adjacent to each other. The first calculation process shown in FIG. 10 may be sequentially executed for each pair of camera CAMs adjacent to each other. Steps that are the same as or similar to the steps described with reference to FIG. 7 are designated by the same or similar reference numerals, and detailed description thereof will be omitted.

In the first calculation process shown in FIG. 10, the process of step S412 is added to the first calculation process shown in FIG. 7. Then, the first calculation process shown in FIG. 10 includes the process of step S432 instead of the process of step S430 shown in FIG. 7. Other processes of the first calculation process shown in FIG. 10 are the same as or similar to the first calculation process shown in FIG. 7. In FIG. 10, the first calculation process for the pair of cameras CAMa and CAMb will be described in the same manner as in FIG.

In step S410, when the skeleton association unit 504a determines that the skeleton point BP of the person PN has been extracted from both the images IMGa and IMGb acquired from the pair of cameras CAMa and CAMb, the estimation device 14 performs the process of step S412. Run. When the skeleton point BP of the person PN is not extracted from at least one of the paired images IMGa and IMGb, the first calculation process for the pair of cameras CAMa and CAMb ends as described with reference to FIG.

In step S412, the additional feature extraction unit 320a extracts additional feature points (for example, a face of a person PN, a feature point indicating clothing worn on the person PN, etc.) from the image IMGa taken by the camera CAMa. For example, the additional feature extraction unit 320a includes at least one of the facial feature points indicating the face of the person PN and the clothing feature points indicating the clothes worn by the person PN in the image IMGa based on the learning data LDA. Extract points. After the process of step S412 is executed, the operation of the estimation device 14 shifts to step S420. Then, after the process of step S420 is executed, the operation of the estimation device 14 shifts to step S432.

In step S432, the skeleton association unit 504a associates the skeleton points BP extracted from each of the paired images IMGa and IMGb with the additional feature points extracted in step S412, and associates the skeleton points BP between the images IMGa and IMGb. 2 Determine the corresponding points. For example, as described in FIG. 8, the skeleton association unit 504a uses the additional correspondence points between the image IMGa and the IMGb to obtain the skeleton point BP extracted from the image IMGa and the skeleton point BP extracted from the image IMGb. Is associated with each other to determine the second corresponding point between the images IMGa and IMGb. As a result, the second corresponding point is specified separately from the first corresponding point as the corresponding point between the images IMGa and IMGb. Since the skeleton association unit 504a determines the association of the skeleton points BP using the pair of additional feature points between the images IMGa and IMGb, the additional feature points are set even if there is a person PN having a similar body shape and posture. Compared with the case where it is not used, the skeleton point BP can be associated more accurately.

After the process of step S432 is executed, the operation of the estimation device 14 shifts to step S440. Then, after the process of step S440 is executed, the operation of the estimation device 14 shifts to step S450. With the end of the process in step S450, the first calculation process for the pair of cameras CAMa and CAMb ends.

Since the estimation device 14 can accurately identify the second corresponding point between the images IMGa and IMGb as compared with the case where the additional feature point is not used for determining the second corresponding point, the external parameter of the camera CAM is set for each frame or the number. It can be calculated accurately every frame. The first calculation process is not limited to the example shown in FIG. For example, the process of step S420 may be executed before the process of step S412.

As described above, even in the embodiments shown in FIGS. 8 to 10, the same effects as those in the embodiments shown in FIGS. 1 to 7 can be obtained. For example, since the estimation device 14 determines the corresponding points between the image IMGs by using the skeleton point BP of the person PN in addition to the natural feature points, it is possible to suppress erroneous association of the feature points and the like, and the erroneous corresponding points. Can be used to suppress the estimation of external parameters of the camera CAM.

Further, the estimation device 14 extracts, in addition to the natural feature point and the skeleton point BP, additional feature points indicating the features of the person PN other than the skeleton point BP from each of the image IMGs acquired from the plurality of camera CAMs. Then, the estimation device 14 uses the person information including at least one of the joint angle θ of the person PN and the length L between the joints and the additional feature points of the person PN to use the skeleton point BP between the two image IMGs. To determine the second corresponding point between the two image IMGs. As a result, the estimation device 14 can accurately identify the second corresponding point between the two image IMGs as compared with the case where the additional feature point is not used for determining the second corresponding point. That is, the estimation device 14 can suppress the estimation of the external parameters of the camera CAM by using an erroneous second corresponding point or the like. As a result, it is possible to suppress a decrease in the estimation accuracy of the external parameters of the camera CAM.

Further, for example, in the sports scene, even if the posture or position of the camera CAM changes during the game, the estimation device 14 obtains the external parameters of the camera CAM from the image IMG acquired after the posture or position of the camera CAM changes. Can be estimated. Therefore, even if the posture or position of the camera CAM changes, the estimation device 14 can accurately estimate the external parameters of the camera CAM according to the change in the posture or position of the camera CAM. That is, the estimation device 12 can accurately estimate the external parameters of the camera CAM even when the posture or position of the camera CAM changes during the shooting period.

The inventions described in the above embodiments will be organized and disclosed as an appendix as follows.
(Appendix 1)
In an estimation method that estimates the external parameters of multiple cameras that shoot subjects from different positions.
From the images acquired from each of the plurality of cameras, the feature points of the captured space are extracted.
The skeleton of a person is extracted from the images acquired from each of the plurality of cameras.
For each pair of two images among the plurality of images acquired from the plurality of cameras, the feature points extracted from one of the two images and the feature points extracted from the other of the two images are associated with each other. The first corresponding point between the two images is determined.
For each pair of the two images, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine a second corresponding point between the two images. ,
An estimation method characterized in that an external parameter of a camera is estimated based on the first correspondence point and the second correspondence point between the two images.
(Appendix 2)
In the estimation method described in Appendix 1,
Based on the skeleton extracted from the image, person information including at least one of the joint angle and the length between the joints of the person is generated.
Using the person information, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine the second corresponding point between the two images. A method of estimating external parameters of a camera, characterized by.
(Appendix 3)
In the estimation method described in Appendix 1 or Appendix 2,
From the images acquired from each of the plurality of cameras, additional feature points indicating the characteristics of a person other than the skeleton are extracted.
Using the additional feature points, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine the second corresponding point between the two images. A method of estimating external parameters of a camera, characterized in that.
(Appendix 4)
In the estimation method described in Appendix 3,
The estimation method, wherein the additional feature point includes at least one of a facial feature point indicating a person's face and a clothing feature point indicating the clothing worn by the person.
(Appendix 5)
In the estimation method described in Appendix 4,
The clothing feature point is an estimation method provided on the clothing and showing at least one of identification information for identifying a person and shoes worn by the person.
(Appendix 6)
In the estimation method described in Appendix 5,
An estimation method characterized in that the identification information is either a uniform number or a person's name.
(Appendix 7)
In the estimation method described in any one of Appendix 3 to Appendix 6,
When the ball game is photographed by the plurality of cameras, the additional feature point is a method of estimating an external parameter of the camera, which includes a ball feature point indicating a ball.
(Appendix 8)
In the estimation method described in any one of Appendix 1 to Appendix 7,
The estimation method is characterized in that the external parameters are used to generate a free-viewpoint image of an arbitrary viewpoint.
(Appendix 9)
In an estimation device that estimates the external parameters of multiple cameras that shoot subjects from different positions.
A feature point extraction unit that extracts feature points in the captured space from images acquired from each of the plurality of cameras, and a feature point extraction unit.
A skeleton extraction unit that extracts the skeleton of a person from images acquired from each of the plurality of cameras, and a skeleton extraction unit.
For each pair of two images among the plurality of images acquired from the plurality of cameras, the feature points extracted from one of the two images and the feature points extracted from the other of the two images are associated with each other. A feature point mapping unit that determines the first corresponding point between the two images, and
For each pair of the two images, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine a second corresponding point between the two images. Skeleton mapping part and
An estimation device including an estimation unit that estimates an external parameter of a camera based on the first correspondence point and the second correspondence point between the two images.
(Appendix 10)
In an external parameter estimation program for multiple cameras that shoot subjects from different positions
From the images acquired from each of the plurality of cameras, the feature points of the captured space are extracted.
The skeleton of a person is extracted from the images acquired from each of the plurality of cameras.
For each pair of two images among the plurality of images acquired from the plurality of cameras, the feature points extracted from one of the two images and the feature points extracted from the other of the two images are associated with each other. The first corresponding point between the two images is determined.
For each pair of the two images, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine a second corresponding point between the two images. ,
An estimation program for causing a computer to perform a process of estimating an external parameter of a camera based on the first correspondence point and the second correspondence point between the two images.
(Appendix 11)
A recording medium that records programs for estimating external parameters of multiple cameras that shoot subjects from different positions.
From the images acquired from each of the plurality of cameras, the feature points of the captured space are extracted.
The skeleton of a person is extracted from the images acquired from each of the plurality of cameras.
For each pair of two images among the plurality of images acquired from the plurality of cameras, the feature points extracted from one of the two images and the feature points extracted from the other of the two images are associated with each other. The first corresponding point between the two images is determined.
For each pair of the two images, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine a second corresponding point between the two images. ,
A recording medium on which an estimation program for causing a computer to execute a process of estimating an external parameter of a camera based on the first correspondence point and the second correspondence point between the two images is recorded.

The above detailed description will clarify the features and advantages of the embodiments. It is intended that the claims extend to the features and advantages of the embodiments as described above, without departing from their spirit and scope of rights. Also, anyone with ordinary knowledge in the art should be able to easily come up with any improvements or changes. Therefore, there is no intention to limit the scope of the embodiments having invention to those described above, and it is possible to rely on suitable improvements and equivalents included in the scope disclosed in the embodiments.

10, 12, 14 ... Estimator; 20 ... Free viewpoint image generator; 100, 102, 104 ... Processor; 200, 202 ... Feature point extraction unit; 300, 302 ... Skeleton extraction unit; 320 ... Additional feature extraction unit; 400 , 402 ... feature point mapping unit; 500, 502, 504 ... skeleton mapping unit; 520 ... mapping auxiliary unit; 600, 602 ... estimation unit; 620 ... external parameter estimation unit; 1000 ... memory; BUS ... bus; CAM Camera; REC Recording medium; SYS Free-viewpoint video generation system

Claims (10)

In an estimation method that estimates the external parameters of multiple cameras that shoot subjects from different positions.
From the images acquired from each of the plurality of cameras, the feature points of the captured space are extracted.
The skeleton of a person is extracted from the images acquired from each of the plurality of cameras.
For each pair of two images of the plurality of images acquired from the plurality of cameras, the feature points extracted from one of the two images and the feature points extracted from the other of the two images are associated with each other. The first corresponding point between the two images is determined.
For each pair of the two images, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine a second corresponding point between the two images. ,
An estimation method characterized in that an external parameter of a camera is estimated based on the first correspondence point and the second correspondence point between the two images. In the estimation method according to claim 1,
Based on the skeleton extracted from the image, person information including at least one of the joint angle and the length between the joints of the person is generated.
Using the person information, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine the second corresponding point between the two images. A method of estimating external parameters of a camera, characterized by. In the estimation method according to claim 1 or 2.
From the images acquired from each of the plurality of cameras, additional feature points indicating the characteristics of a person other than the skeleton are extracted.
Using the additional feature points, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine the second corresponding point between the two images. A method of estimating external parameters of a camera, characterized in that. In the estimation method according to claim 3,
The estimation method, wherein the additional feature point includes at least one of a facial feature point indicating a person's face and a clothing feature point indicating the clothing worn by the person. In the estimation method according to claim 4,
The clothing feature point is an estimation method provided on the clothing and showing at least one of identification information for identifying a person and shoes worn by the person. In the estimation method according to claim 5,
An estimation method characterized in that the identification information is either a uniform number or a person's name. In the estimation method according to any one of claims 3 to 6,
When the ball game is photographed by the plurality of cameras, the additional feature point is a method of estimating an external parameter of the camera, which includes a ball feature point indicating a ball. In the estimation method according to any one of claims 1 to 7.
The estimation method is characterized in that the external parameters are used to generate a free-viewpoint image of an arbitrary viewpoint. In an estimation device that estimates the external parameters of multiple cameras that shoot subjects from different positions.
A feature point extraction unit that extracts feature points in the captured space from images acquired from each of the plurality of cameras, and a feature point extraction unit.
A skeleton extraction unit that extracts the skeleton of a person from images acquired from each of the plurality of cameras, and a skeleton extraction unit.
For each pair of two images among the plurality of images acquired from the plurality of cameras, the feature points extracted from one of the two images and the feature points extracted from the other of the two images are associated with each other. A feature point mapping unit that determines the first corresponding point between the two images, and
For each pair of the two images, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine a second corresponding point between the two images. Skeleton mapping part and
An estimation device including an estimation unit that estimates an external parameter of a camera based on the first correspondence point and the second correspondence point between the two images. In an external parameter estimation program for multiple cameras that shoot subjects from different positions
From the images acquired from each of the plurality of cameras, the feature points of the captured space are extracted.
The skeleton of a person is extracted from the images acquired from each of the plurality of cameras.
For each pair of two images among the plurality of images acquired from the plurality of cameras, the feature points extracted from one of the two images and the feature points extracted from the other of the two images are associated with each other. The first corresponding point between the two images is determined.
For each pair of the two images, the skeleton extracted from one of the two images and the skeleton extracted from the other of the two images are associated with each other to determine a second corresponding point between the two images. ,
An estimation program for causing a computer to perform a process of estimating an external parameter of a camera based on the first correspondence point and the second correspondence point between the two images.

Images