JP7278720B2 - Generation device, generation method and program - Google Patents

Description

The present invention relates to a generation device, generation method, and program for generating three-dimensional shape data.

For various purposes such as terrain analysis and street viewing, there is a demand for analyzing images captured by cameras and acquiring three-dimensional position information (three-dimensional shape data) of terrain. In Patent Literature 1, three-dimensional shape data is acquired from aerial photographs of terrain taken from a plurality of directions using a stereo matching method.

International Publication WO08/152740

However, since the stereo matching method as disclosed in Patent Document 1 uses matching for each pixel between images, it may not be possible to acquire three-dimensional shape data with high accuracy depending on the pattern on the object plane. . For example, for a subject plane with a pattern drawn that has almost the same color and that extends in a certain direction, such as lines drawn on a competition field, the accuracy of the above matching does not improve, and the three-dimensional shape of the subject plane Data cannot be obtained with high accuracy.

An object of the present invention is to obtain highly accurate three-dimensional shape data regardless of the pattern on the object surface.

The generating apparatus of the present invention includes first obtaining means for obtaining a plurality of photographed images obtained by photographing a subject plane from a plurality of directions, and second obtaining means for obtaining information indicating the position and shape of a pattern on the subject plane. for each of a plurality of areas on the subject plane based on the acquisition means, the plurality of photographed images acquired by the first acquisition means, and the information indicating the position and shape of the pattern acquired by the second acquisition means; generating means for generating three-dimensional shape data corresponding to the object plane based on the three-dimensional position information for each region determined by the determining means; and the first projection means for generating a projection image on a projection plane having a different distance from the object plane for each area and for each distance from the object plane, based on the plurality of captured images acquired by the acquisition means; synthesizing means for synthesizing a plurality of projected images having the same distance from the object plane to generate a synthetic image for each distance from the object plane, wherein the determining means is generated by the synthesizing means; Further, the three-dimensional position information is determined for each area based on the synthesized image for each area and for each distance from the object plane.

According to the present invention, highly accurate three-dimensional shape data can be acquired regardless of the pattern on the object surface.

FIG. 2 is a diagram showing an example of the configuration of a generation device that generates three-dimensional shape data of an object plane; FIG. 4 is a diagram showing an example of each camera that constitutes a camera group; FIG. 2 is a diagram for explaining the concept of the first embodiment; FIG. Schematic diagram of the shape of the line drawn in the field. FIG. 2 is a block diagram showing an example of the logical configuration of the generation device according to the first embodiment; FIG. 4 is a flowchart showing the flow of processing of the generation device according to the first embodiment; 4A and 4B are schematic diagrams showing how lines on a projected image appear differently depending on the height of the projection plane; FIG. FIG. 4 is a diagram for explaining camera reliability; FIG. 4 is a schematic diagram showing a composite image obtained by combining projection images for each height of a projection plane; FIG. 10 is a block diagram showing an example of the logical configuration of a generation device according to the second embodiment; 9 is a flow chart showing the flow of processing of the generating device according to the second embodiment; FIG. 11 is a diagram showing a configuration example of an image processing system according to a third embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the embodiments are essential for the solution of the present invention. In addition, the same configuration will be described by attaching the same reference numerals.

In addition, in the following embodiments, generation of three-dimensional shape data of a stadium field (ground) will be described as an example, but application of the present invention is not limited to this. The present invention can also be applied to obtain three-dimensional information on general topography, road shape, wall surface shape, painting shape, and wall painting shape.

In the present embodiment, the three-dimensional shape data is data representing the three-dimensional shape of a substantially planar object plane. , z with position information. In addition, the three-dimensional shape data is not limited to being represented by a point cloud, and may be represented by other data formats. may be represented by polygon mesh data, voxels, or the like.

Also, the image in the present embodiment is image data, and does not necessarily have to be a visible image that is generated to be displayed on a display device such as a display.

[Embodiment 1]
FIG. 1 is a diagram showing an example of the configuration of a generation device 100 that generates three-dimensional shape data according to this embodiment. The generation device 100 includes a CPU (Central Processing Unit) 101 , a main memory 102 , a storage section 103 , an input section 104 , a display section 105 and an external I/F section 106 , which are connected via a bus 107 . First, the CPU 101 is an arithmetic processing device that controls the generation device 100 in an integrated manner, and executes various programs stored in the storage unit 103 or the like to perform various processes. The main memory 102 temporarily stores data and parameters used in various processes, and also provides a work area to the CPU 101 . The storage unit 103 is a large-capacity storage device that stores various programs and various data necessary for GUI (Graphical User Interface) display, and uses a non-volatile memory such as a hard disk or a silicon disk.

The input unit 104 is a device such as a keyboard, mouse, electronic pen, touch panel, etc., and receives operation input from the user. A display unit 105 is configured by a liquid crystal panel or the like, and performs GUI display of analysis results. The external I/F unit 106 is connected to each camera constituting the camera group 109 via the LAN 108, and transmits and receives video data and control signal data. A bus 107 connects the above-described units and performs data transfer.

The generation device 100 is connected to the camera group 109 via the LAN 108 . Based on the control signal from the generation device 100, the camera group 109 starts and stops shooting, changes camera settings (shutter speed, aperture, etc.), and transfers captured image data.

Note that the configuration of the generation device 100 includes various components other than those described above, but they are not the main focus of the present embodiment, so description thereof will be omitted.

FIG. 2 is a diagram showing an arrangement example of each camera constituting the camera group 109. As shown in FIG. Here, an example in which 10 cameras are installed in the stadium is shown, but the number and installation positions of the cameras are not limited to this. A player 202 and a ball are present on a field 201 on which a game is played, and ten cameras 203a to 203j are arranged around the field 201. FIG. In each of the cameras 203a to 203j that make up the camera group 109, appropriate camera orientation, focal length, exposure control parameters, etc. are set so that the entire field 201 or the attention area of the field 201 fits within the angle of view. there is

FIG. 3 is a diagram showing an outline of this embodiment. In this embodiment, STEP. 1, the field is divided into a plurality of regions based on the two-dimensional positions and shapes of the lines drawn in the field 201 . FIG. 4 shows the shape of lines drawn in the field. A line refers to a pattern that extends in a direction and has a finite width in the direction perpendicular to that direction. The length and width of the line are determined by competition standards. In this embodiment, in order to obtain the height of the field for each region based on a specific pattern such as lines, the field 201 is virtually divided into a plurality of divided regions so as to include the lines. . This division may be made in the photographed image obtained by photographing with a camera. FIG. 3(a) shows an example of division into six divided areas (S1 to S6). Note that the divided regions may be virtually set with respect to the captured image, or may be virtually set with respect to the projection image described later.

Information about the line that is the specific pattern (for example, information indicating the shape such as the position on the field, the length in the extending direction, the width in the direction perpendicular to the extending direction, etc.) 100 is entered. However, the photographed image and the subsequent STEP. 2 may be determined by the generation device 100 from the projection image output in step 2. FIG. In this embodiment, a case will be described in which information about lines is input in advance and acquired by the generation device 100 .

Next, STEP. 2, projection images on projection planes with different distances (heights) from the field, which is the object plane, are generated based on the images captured by the camera. FIG. 3B shows an example in which all the cameras 1 to 10 are used to capture the divided area S1. In this case, for the area S1, the photographed images acquired by the camera 1 are projected onto a plurality of projection planes having different heights from the field to generate a plurality of projected images. For example, when the height is set in increments of 5 cm and the height of the central point of the field is set to 0 cm, and the projection image is generated by projecting onto the projection plane from −15 cm to +15 cm, the captured image acquired by the camera 1 is: Seven projection images are generated. When this projected image is generated from the ten cameras 1 to 10 photographing the divided area S1, a total of 70 (=7×10) projected images are generated for each distance of the object plane in the divided area S1. be done. A plurality of projected images are similarly generated in other divided areas. However, for example, in a certain divided area, if the image is not captured by the camera 3, the projection image is generated from the image captured by the camera other than the camera 3.

Also, the height of this projection plane may be based on an arbitrary point on the field. For example, the center point of the field may be used as a reference point, and a plane parallel to a parallel plane containing the reference point of the field may be set as the projection plane. Specifically, the reference point is set to (0, 0, 0) in three-dimensional coordinates, the region on the object plane near the reference point is set to a reference plane that is an xy plane, and the distance in the z direction from the reference plane is The projection plane may be set differently.

STEP. By 2, changing the height from the field changes the position of the line in the projected image, like the multiple projected images of camera 1 in FIG. 3(b). This is because the camera 1 photographs the divided area S1 in a direction perpendicular to the line extending direction. Then, in projected images with different heights from the field, when the height of the projection plane changes, from the correspondence relationship between the pixel positions of the captured image and the positions on the projection plane, changes the position of the line in each projection image.

On the other hand, in the multiple projection images of the camera 10 in FIG. 3B, the positions of the lines in the projection images appear to be unchanged. This is because the camera 10 photographs the divided area S1 in a direction parallel to the direction in which the lines are extended. In this case, due to the correspondence relationship between the pixel positions of the captured image and the positions on the projection plane, the position of the line in the projected image changes in the direction parallel to the stretching direction when the projection plane changes. looks unchanged.

It should be noted that STEP. 1 and STEP. The order of 2 can be changed. Specifically, projection images of the cameras 203a to 203j may be generated for each height from the field, and the projection images for each height may be divided into regions.

Next, for this embodiment, STEP. 3, the projection images are synthesized for each set area and for each height from the field to generate a synthesized image. As shown in FIGS. 3B and 3C, in the region S1, for example, at a height of +15 cm, ten projection images 301Aa to 301Aj generated from the captured images of the cameras 1 to 10 are synthesized. Then, a composite image 302A is generated. Other synthetic images (for example, synthetic images 302D and 302G) are similarly generated. Here, "A" in 301Aa represents height, and the same "A" represents that the height from the field is the same. Also, "a" indicates a camera number, and the same "a" means that the projection image is based on the captured image acquired by the same camera.

Next, STEP. 4, the synthesized image with the sharpest line is determined from the synthesized images 302A to 302G for each height from the field of the divided area S1. Then, the determined height from the field corresponding to the synthesized image is determined as the height from the field in the divided area S1. This STEP. 4 is performed for each set area, the height from the field in each divided area is determined. Note that the height from the field in each divided area may be determined based on another evaluation value instead of the sharpness of the image.

Finally STEP. 5, the three-dimensional shape data of the field is generated based on the determined height from the field in each divided area. Processing performed by the generation device 100 will be described in detail below.

FIG. 5 is a block diagram showing the functional configuration of the generation device 100. As shown in FIG. In this embodiment, the generation device 100 generates three-dimensional shape data of a stadium field. The generation device 100 includes an image acquisition unit 501, a camera parameter acquisition unit 502, a projection unit 503, an area setting unit 504, a reliability calculation unit 505, a synthesis unit 506, a distance determination unit 507, a generation unit 508, and a specific pattern acquisition unit 509. have

An image acquisition unit 501 acquires a plurality of captured images captured by the camera group 109 . The camera group 109 is composed of ten cameras 203a to 203j shown in FIG. Then, the image acquisition unit 501 acquires captured images from each of the cameras 203a to 203j. The image acquisition unit 501 outputs the captured image to the camera parameter acquisition unit 502 and the projection unit 503 .

A camera parameter acquisition unit 502 performs camera calibration from the captured image output from the image acquisition unit 501, and acquires camera parameters including extrinsic parameters, intrinsic parameters, and distortion parameters of the camera. The extrinsic parameters are parameters representing the position and orientation of the camera, such as rotation matrices and position vectors. Intrinsic parameters are camera-specific parameters such as focal length and image center. The camera parameter acquisition unit 502 outputs camera parameters to the projection unit 503 and reliability calculation unit 505 .

The projection unit 503 projects images based on a plurality of captured images output from the image acquisition unit 501, camera parameters output from the camera parameter acquisition unit 502, and information indicating a set area output from an area setting unit 504, which will be described later. , to generate the projection image. A projection image is generated by projecting each photographed image onto a projection plane having a different distance from the object plane, and generating a projection image for each set area and for each distance from the object plane. The projection unit 503 outputs a projection image for each set area and for each distance from the object plane to the synthesis unit 506 .

A specific pattern acquisition unit 509 acquires information about a specific pattern from the outside. The information about the specific pattern includes information about the position of the specific pattern on the object plane, information indicating the shape of the specific pattern, information indicating the color difference between the specific pattern and another area on the object plane, and the like. In the case of a line in which a specific pattern is drawn in a field, the information indicating the shape of the line is the extending direction of the line, its length, and the width in the direction perpendicular to the extending direction of the line. Specific pattern acquisition section 509 outputs information about the specific pattern to area setting section 504 and distance determination section 507 .

Based on the information about the specific pattern output from the specific pattern acquisition unit 509, the area setting unit 504 virtually sets a plurality of areas on the object plane for determining the three-dimensional position information of the object plane. Specifically, the region setting unit 504 virtually sets regions such that at least part of the specific pattern is included in each of the plurality of set regions. Here, an example of setting the area to be set on the projected image is shown, but the area may be set on the captured image. The area setting unit 504 outputs to the projection unit 503 and the reliability calculation unit 505 information indicating a plurality of setting areas for determining the distance of the reference point on the object plane.

The reliability calculation unit 505 calculates camera reliability for each region set by the region setting unit 504 and for each camera (each projection image). The camera reliability is calculated using the camera parameters output from the camera parameter acquisition unit 502, information indicating a plurality of set areas output from the area setting unit 504, and the position and shape of the specific pattern output from the specific pattern acquisition unit 509. is calculated based on the information indicating The camera reliability is used when the synthesizing unit 506, which will be described later, synthesizes projected images. The reliability calculation unit 505 outputs camera reliability to the combining unit 506 .

Based on the camera reliability output from the reliability calculation unit 505, the synthesizing unit 506 synthesizes projection images for each set area on a projection plane having the same distance from the object plane to generate a synthetic image. This camera reliability is used as a weight for each projection image when synthesizing by weighted averaging processing. The weighted averaging process referred to here means calculating a pixel value by weighting and averaging pixel values of corresponding pixels of each of a plurality of projection images. The synthesizing unit 506 outputs the synthetic image for each set area and for each distance from the object plane to the distance determining unit 507 .

The distance determining unit 507 is based on the synthesized image for each set area and for each distance from the object plane output from the synthesizing unit 506 and the information indicating the position and shape of the specific pattern output from the specific pattern acquiring unit 509. to determine the distance from the object plane. Specifically, the distance determination unit 507 calculates the evaluation value of the specific pattern in the composite image for each set area and each distance from the object plane, and determines the distance from the object plane based on the evaluation value. For example, the distance determination unit 507 can use sharpness as the evaluation value. In this case, the distance from the subject plane corresponding to the composite image in which the sharpness of the specific pattern is the highest is determined as the distance from the subject plane in the set area in the synthesized images for each distance from the subject plane in a certain set area. do. The distance determination unit 507 outputs to the generation unit 508 the distance from the reference point on the object plane determined for each set area. The distance determination unit 507 determines three-dimensional position information of the set area. That is, the xy coordinates of the set area are coordinates based on the area setting unit 504, and the distance from the reference point on the object plane determined by the distance determination unit 507 corresponds to the z coordinate.

Based on the distance from the reference point of the object plane determined for each set area output from the distance determination unit 507, that is, based on the three-dimensional position information determined for each set area, the generating unit 508 generates 3D images of the object plane. Generate dimensional shape data. A generation unit 508 outputs three-dimensional shape data of the object plane.

Next, the processing performed by the generation device 100 will be described in detail with reference to the flowchart shown in FIG. This series of processing is realized by the CPU 101 reading a predetermined program from the storage unit 103, developing it in the main memory 102, and executing the program by the CPU 101. FIG.

In S601, camera parameters are obtained through calibration processing. First, the image acquisition unit 501 sends a photographing instruction to the camera group 109 via the LAN 108 . The captured image is obtained by the image obtaining unit 501 . The camera group 109, as shown in FIG. 2, is composed of a plurality of cameras 203a to 203j with different photographing directions. A camera parameter acquisition unit 502 calculates parameters of each camera of the camera group 109 from the image acquired by the image acquisition unit 501 . The camera parameters are calculated by camera calibration processing using a plurality of images captured by different camera positions as input. Below is an example of a simple camera calibration procedure.

First, a planar pattern such as a square grid is photographed from multiple viewpoints. Second, the feature points of the captured image are detected, and the coordinates of the feature points are obtained in the image coordinate system. Here, the feature points of the square grid are points of intersection of straight lines. Third, the initial values of the internal parameters of the camera are calculated using the calculated feature point coordinates. Here, the internal parameters of the camera are parameters representing the focal length and the optical center called the principal point. Also, the initial values of the internal parameters of the camera do not necessarily have to be calculated from the feature points in the image, and the design values of the camera may be used. Fourthly, the camera's intrinsic parameters, extrinsic parameters, and distortion aberration coefficients are calculated by nonlinear optimization processing called bundle adjustment. Here, the extrinsic parameters of the camera are parameters representing the position of the camera, the line-of-sight direction, and the angle of rotation about the line-of-sight direction. The distortion aberration coefficient is a coefficient that expresses radial image distortion caused by the difference in refractive index of the lens and circumferential distortion caused by non-parallelism between the lens and the image plane. There are many other methods of camera calibration, but they are not the focus of this embodiment, so the details are omitted.

In S602, the image acquisition unit 501 sends a photographing instruction to the camera group 109 so as to photograph the field. The image acquiring unit 501 receives the captured images acquired by capturing the object plane by the plurality of cameras 203a to 203j that configure the camera group 109 and have different capturing directions.

In S603, the specific pattern acquisition unit 509 acquires information about the specific pattern. Specifically, the specific pattern acquisition unit 509 extracts information including the shape of the line that is the specific pattern and the position of the line on the object plane based on the captured image acquired in S602.

In S604, based on the information about the specific pattern acquired in S603, the area setting unit 504 sets a plurality of predetermined areas for determining the distance on the object plane. Specifically, the region setting unit 504 sets regions such that each set region includes at least part of the specific pattern. Also, the area setting unit 504 may divide the area so that one line of the specific pattern is equally divided. When dividing, it is preferable to determine the area width so that the line is included in one area when the height of the projection plane changes. The set areas may be set so that the entire object plane is covered by the plurality of set areas. That is, in order to calculate the height more densely, the set areas may be set so as to overlap. Also, the set regions may be set so as to divide the object plane without overlapping. The set regions may not have the same size or the same shape, and may have different sizes and shapes as long as they include lines of a specific pattern.

In S605, based on the camera parameters acquired in S601, the projection unit 503 uses a plurality of captured images for each of the plurality of set regions set in S604 to project images onto a plurality of projection planes having different distances from the object plane. Generate a projected projection image. Note that the projection unit 503 may generate a projection image by projecting the captured image onto a different projection plane, and then extract a projection image corresponding to the set region from the projection image. Alternatively, the projection unit 503 may extract an image for each set area from the captured image, and then project the image onto the projection plane to generate a projection image corresponding to the set area.

When performing camera calibration, the field 201 in FIG. 2 is used as a reference for a plane with a height of approximately 0 m. and set the origin to the center of the field. The projection plane is a plane horizontal to the field, which is the object plane. The horizontal range to be projected onto the projection plane is determined using information indicating the position and shape of the line so as to cover the entire field. For example, based on the shape of the line in FIG. 4, projection is performed over a range of 80 m long and 120 m wide. Of course, the projection may be performed with a margin of a few percent to account for the error between the actual field and the shape of the line.

In order to calculate the height of the entire field, multiple projection planes with different heights are used, but the range of heights to be projected is determined in accordance with the field gradient standards of the stadium. For example, as a field standard, there is a standard such as a maximum gradient of 0.3% from the center of the field to the edge of the field. In this case, if the distance from the origin of the field to the edge of the field is 40 m, the allowable height variation is up to 12 cm. Therefore, the range of projection height is set to -15 cm to +15 cm to cover this range. Within this range, the height increments can be set arbitrarily. By increasing the number of increments, that is, the number of projection planes, highly accurate three-dimensional shape data can be obtained.

When generating projection images, distortion correction is first performed on images captured by each camera in accordance with the camera's internal parameters and distortion parameters. The parameters used for image distortion correction are the internal parameters calculated in S601 and the distortion parameters.

Next, a conversion matrix between the coordinates of the projected image and the coordinates of the captured image is calculated. Let V be a transformation matrix from the world coordinate system in which the projection plane exists to the camera coordinate system. Here, the camera coordinate system is set so that the origin of the coordinate system is the starting point, the x-axis and y-axis are the horizontal and vertical directions of the image, respectively, and the z-axis is the line-of-sight direction of the camera. Further, P is defined as a transformation matrix from the camera coordinate system to the screen coordinate system. This is a transformation matrix that projects an object plane having three-dimensional coordinates existing with respect to the camera coordinate system onto a two-dimensional plane. That is, the equation (1 ) are as follows:

Here, in order to apply a translational transformation, the coordinates w and w' were added to make the four-dimensional coordinates. Using this equation (1), the images captured by each camera are projected onto projection planes with different heights z to generate projection images. Specifically, the projection image is generated by setting the pixel values of the coordinates of the captured image to the pixel values of the coordinates of the projection image corresponding to the coordinates of the captured image.

In S606, the reliability calculation unit 505 calculates camera reliability from the camera parameters obtained in S601 and the specific pattern obtained in S603 for each set area set in S604. Here, the camera reliability is an index indicating how useful the photographed images obtained from each camera are for determining the distance to the object plane for each set area. A simple example is shown in FIG.

FIG. 7 is a schematic diagram of shooting a field with four cameras (701 to 704). Here, it is assumed that the height of the rectangular area 705 set in the center of the field is obtained. When the height of the rectangular area 705 changes, when projection images are generated from images captured by the cameras 701 and 703 having a line-of-sight vector (optical axis) perpendicular to the extending direction of the line, the position of the line in each projection image is changes significantly. On the other hand, when projection images are generated by changing the height of the projection plane from the captured images of the cameras 702 and 704 having line-of-sight vectors (optical axes) parallel to the extending direction of the lines, the position of the line in each projection image is little change.

In the present embodiment, as will be described later, a synthesized image is generated by synthesizing projection images of each camera for each set area and for each height from the field surface. In order to calculate evaluation values such as line displacement and degree of blur in the composite image, the composite image is processed by selectively using images in which such characteristics are likely to appear, or weighting the composite image before compositing. is desirable. A camera reliability is defined for weight calculation when selecting and combining images.

Therefore, as shown in FIG. 8, when a camera 802 is installed with respect to a line 801, and the position of the camera with respect to the center of the line is defined by a horizontal angle φ and an elevation angle θ, for example, the camera reliability ω can be expressed by the formula ( 2).

This means that a camera whose line-of-sight vector is nearly perpendicular to the direction of the line and whose elevation angle θ is small has a higher camera reliability. That is, when determining the height of the rectangular area 705, the camera reliability of the cameras 701 and 703 shown in FIG. 7 is increased. As is clear from Equation (2), the camera reliability of each camera changes for each set area, so the camera reliability is calculated for each set area and for each camera. In addition, the camera reliability can be said to be a reliability indicating how useful the photographed image or projected image corresponding to each camera is for determining the distance from the object plane, that is, the three-dimensional position information of the object plane. . The camera reliability is an index for determining the distance from the reference point of the object plane for each set area, that is, the three-dimensional position information of the object plane. This is an index that indicates the degree of usefulness when determining the distance.

As can be seen from equation (2), if the specific pattern is like a line, the angle formed by the extending direction of the line and the direction of a straight line when the optical axis of the camera is projected onto the plane in the extending direction of the line. A camera with a larger (90°-φ) has a higher camera reliability. Also, the camera having a smaller elevation angle θ has a higher camera reliability.

The camera reliability determination method is not limited to the above method. Depending on the physical distance d from the camera to the line, the focal length f, the number of pixels p, etc., there is a difference in whether or not the line is clearly captured by the camera, so the reliability of the camera is increased according to these parameters. You may Alternatively, the camera reliability ω may be calculated by combining a camera capable of clearly capturing a line and increasing the reliability of a camera with a small horizontal angle and elevation angle. The camera reliability ω obtained by this combination is given by Equation (3). where α and β are weighting parameters.

Threshold processing may also be performed, such as setting camera reliability lower than a predetermined threshold to 0, for example.

In S<b>607 , based on the camera reliability obtained in S<b>606 , the synthesizing unit 506 synthesizes the plurality of projection images obtained in S<b>605 on the same projection plane for each set region. Specifically, since the areas captured by each camera are different, it is determined whether or not each camera can capture the set area. Specifically, in the projected image for each set area, if there is a pixel value obtained by projecting the captured image on all the pixels in each area, it is assumed that the set area can be captured by the corresponding camera. This determination may be performed by the region setting unit 504 in S604, or may be performed by the projection unit 503 in S605.

Next, for each set area, projected images are synthesized for each projection plane using the projected images captured by each of the plurality of cameras. Specifically, a weighted averaging process based on the camera reliability ω for each set area calculated in S606 is performed to generate a composite image. That is, the projected image is an rgb image, and the synthesized image of each set area Bj at the height h of the projection plane is expressed by Equation (4). Note that the camera number is k.

As described in the processing of S606, it is not necessary to generate the weighted average image using all the camera reliability. For example, the projected image corresponding to the camera whose camera reliability is equal to or less than a predetermined threshold is not used, or only the projected image corresponding to the camera whose reliability is equal to or greater than the predetermined threshold is used to perform the averaging process to generate a composite image. may In addition, even if a synthetic image is not generated by performing averaging processing using the camera reliability, a synthetic image can be generated using a simple average value or an intermediate value of the pixel values of corresponding pixels in the projected image. You may

In S607, the distance determination unit 507 determines the distance from the reference point of the object plane for each set area, that is, the distance of 3 Determine dimensional position information. When determining the distance from the reference point of the object plane, the evaluation value of the specific pattern for each set area of the synthesized image is evaluated. The evaluation value is specifically described below using sharpness, but is not limited to this.

FIG. 9 shows an example of a composite image for each distance (height) from a reference point on the object plane in a certain set area. Specifically, FIG. 9 shows an example in which the projection plane is set by changing the height from −15 cm to +15 cm with respect to the object plane in increments of 0.5 cm, and the composite image is generated at each height. ing. As shown in FIG. 9, in the synthesized image at an appropriate height (0 cm), lines are clearly visible. This is because the lines included in each of the plurality of projection images used for synthesis overlap at substantially the same positions. Overlapping position means that the actual line height is in its projection plane. On the other hand, in the synthesized images at other heights, the lines included in each of the plurality of projection images used for synthesis are out of alignment, and the lines are blurred.

Therefore, the actual height of the line can be determined by evaluating the sharpness of the line. A filter such as the Laplacian filter L is used to evaluate the sharpness of the image. The sharpness Sj,h of the synthesized image of a certain set area Bj at the height h is represented by Equation (5).

Note that the filter for evaluating the sharpness of an image is not limited to the Laplacian filter, and a primary differential filter, Prewitt filter, Sobel filter, or the like may be used. Also, the difference between the synthesized image to which the smoothing filter is applied and the original synthesized image to which the filter is not applied is calculated, and the synthesized image having a height that makes the difference large is determined as the appropriate height of the setting area. You can do it.

There are also several ways to determine height after evaluating sharpness. For example, the height Hj that maximizes the sharpness is determined as the appropriate height of the setting area Bj (see equation (6)).

As an index of whether or not the appropriate height is calculated correctly, for example, it may be determined whether or not the sharpness changes smoothly when the height is changed. Specifically, it is determined using equation (7) whether or not the sharpness changes smoothly around the height at which the sharpness is maximized. Here, let a be the height width (increment) at which the projection image is generated. If the expression (7) is satisfied. The height is judged to be an appropriate height for the setting region Bj, and if it does not satisfy the height, the height corresponding to the synthesized image with the next highest sharpness is similarly judged.

Further, when a plurality of sharpnesses close to the maximum sharpness are calculated, the height of the projection plane corresponding to the sharpnesses may be averaged to calculate the appropriate height of the set area.

Further, the constraint condition may be the continuity of the height of adjacent set areas. Since the height of the object plane changes smoothly, the height difference between adjacent set areas should be small. By combining this constraint condition and the above sharpness, the likelihood Mj,h of the height hj in the region Bj can be calculated by Equation (8). where α and β are weighting parameters. Then, the height at which this likelihood is maximized may be determined as the optimum height of the set area.

The first term on the right side of Equation (8) is an index that indicates the continuity of the distance from the object plane and indicates the change in the distance from the object plane to two adjacent set areas.

In this way, the composite image is used to determine the appropriate height for each set area. In addition, the distance determination unit 507 first determines the distance from the reference point of the object plane to the apex of each set area in the set area set by the area setting unit 504 and including the portions where the set areas overlap each other. Determined by the method above. Then, the distance determination unit 507 determines the median value or the average value of the distances of the vertices of the respective set regions from the reference point of the object plane as the distances of the vertices of the overlapping portion from the reference point of the object plane. good too.

In S609, the generation unit 508 generates three-dimensional shape data of the object plane according to the height of each set area determined in S608. Specifically, assuming that a vertex exists at the geometric center position of the set area, the vertex coordinates are changed according to the height determined in S608. In this case, processing may be performed such as not using a region in which only a camera with a low reliability is used for height calculation as a vertex when generating three-dimensional shape data. For areas other than the line, the vertex coordinates of the line portion can be used to generate the vertex coordinates. For example, the height vz of a vertex v in an area not on a line can be calculated by a weighted average of the distances from the vertex v' existing in the neighborhood area Ω of the vertex (equation (9), equation (10 ), see formula (11)).

As a method of expressing three-dimensional shape data, the shape may be expressed as a point group using only the calculated vertices. In this case, it is represented by a point group having x, y, and z positional information in a three-dimensional space in the world coordinate space that uniquely indicates the imaging space. Also, the three-dimensional shape data may be expressed as polygon mesh data as a set of a plurality of planes by generating planes connecting the geometric center positions of the set regions. Also, the three-dimensional shape data may be represented by voxels.

As described above, in the present embodiment, the area setting unit 504 sets an area for determining the three-dimensional position information so as to include at least part of the specific pattern (pattern), and uses the specific pattern (pattern). determines the three-dimensional position information of the object plane. Therefore, three-dimensional shape data can be generated with high accuracy.

Although the pattern in the present embodiment has been described as an example of lines drawn on the field, it is not limited to this. For example, patterns may include graphics, signs, paintings, and the like. Moreover, the pattern may be an artificial pattern or a naturally formed pattern. In addition, it is desirable that the pattern has a different color from the colors of other regions different from the pattern on the subject plane.

[Embodiment 2]
In the first embodiment, the height of each set area is determined after synthesizing the projection images. In the present embodiment, a configuration will be described in which the height of each set area is determined without synthesizing projection images. FIG. 10 is a block diagram showing the functional configuration of the generation device 1000 according to this embodiment. FIG. 11 is a flow chart of processing performed by the generation device 1000 . 10 and 11, the same reference numerals are given to the same configurations as in the first embodiment.

The generation device 1000 includes an image acquisition unit 501, a camera parameter acquisition unit 502, a projection unit 503, an area setting unit 504, a reliability calculation unit 1001, a synthesis unit 506, a distance determination unit 1002, a generation unit 508, and a specific pattern acquisition unit 509. have The reliability calculation unit 505 of the first embodiment outputs the camera reliability to the synthesizing unit 506, but the reliability calculation unit 1001 of the present embodiment outputs the camera reliability to the distance determination unit 1002. different.

In FIG. 11, the processing (S601 to S606) up to calculation of the camera reliability and the processing (S609) for generating the three-dimensional shape data are the same as those in the first embodiment, so description thereof will be omitted. The processing of S1101 will be specifically described below.

In S1101, the distance determination unit 1002 determines an appropriate distance from the camera reliability calculated in S605 and the projection image for each set area and each distance from the object plane acquired in S605. First, as described in the first embodiment, since the areas captured by each camera are different, it is determined whether or not each camera can capture the set area.

Next, lines existing in the set area are detected. Since the lines are clearly drawn on the lawn or the ground according to a predetermined standard, they can be easily extracted by color detection or processing for extracting areas with high brightness. When the height of the projection plane is changed, the position of the line greatly changes between projection images corresponding to cameras with high camera reliability. However, when projected onto an appropriate height of the projection plane, the positions of the lines of the projected images corresponding to any cameras are substantially the same. That is, for the height h, the height H of the projection plane that maximizes the intersection of the existing regions Wi,j of the lines in the arbitrary regions Bi,j,h of the projection image of the camera i is set to the optimum height of the set region. height (see equation (12)). Note that the method for determining the appropriate height of the setting area is not limited to this method. For example, the height that minimizes the union of the line existence areas may be calculated (see equation (13)). Note that the line existence area is an area where the line is drawn in the projection image.

When the continuity of adjacent set regions is a constraint condition, it is sufficient to consider the continuity of the existing regions of the lines of the projection image with high camera reliability. For example, when the regions are divided so that the regions partially overlap, a constraint condition may be added such that the height that maximizes the intersection of existing regions of adjacent lines is adopted.

[Embodiment 3]
An image processing system for generating a virtual viewpoint image according to this embodiment will be described below. The 3D shape data of the field generated in the above-described embodiment is used when generating a virtual viewpoint image.

A system in which a plurality of cameras and microphones are installed in a facility such as a stadium or a concert hall to photograph and collect sound will be described with reference to the system configuration diagram of FIG. 12 . Image processing system 1200 includes sensor systems 1210 a - 1210 j , image computing server 1300 , controller 1400 , switching hub 1280 and end user terminal 1290 .

Controller 1400 has control station 1410 and virtual camera operation UI 1430 . The control station 1410 manages the operation state and controls parameter setting for each block constituting the image processing system 1200 through the networks 1410a to 1410c, 1391, 1280a, 1280b, and 1270a to 1270i. Here, the network may be GbE (gigabit Ethernet) or 10GbE conforming to the IEEE standard, which is Ethernet (registered trademark, hereinafter omitted), or may be configured by combining interconnect Infiniband, industrial Ethernet, and the like. Moreover, it is not limited to these, and other types of networks may be used.

First, the operation of transmitting ten sets of images and sounds for sensor systems 1210a-1210j from sensor system 1210j to image computing server 1300 will be described. In the image processing system 1200 of this embodiment, sensor systems 1210a to 1210j are connected by a daisy chain.

In this embodiment, the ten sets of sensor systems 1210a to 1210j are referred to as the sensor system 1210 without distinction unless otherwise specified. Similarly, devices in each sensor system 1210 will be referred to as a microphone 1211, a camera 1212, a camera platform 1213, an external sensor 1214, and a camera adapter 1220 without distinction unless otherwise specified. Although the number of sensor systems is described as 6 sets, it is only an example and the number is not limited to this. Each camera 1212a-1212j of the imaging system is located at a position other than the symmetrical position of the different cameras.

Also, the plurality of sensor systems 1210 may not have the same configuration, and for example, may be composed of different models of devices. In this embodiment, unless otherwise specified, the term "image" will be described as including the concepts of moving images and still images. That is, the image processing system 1200 of this embodiment can process both still images and moving images. Also, in the present embodiment, an example in which the virtual viewpoint content provided by the image processing system 1200 includes the virtual viewpoint image and the virtual listening point sound will be mainly described, but the present invention is not limited to this. For example, virtual viewpoint content does not have to include audio. Further, for example, the sound included in the virtual viewpoint content may be sound collected by a microphone closest to the virtual viewpoint. In addition, in this embodiment, for the sake of simplification of explanation, the description of audio is partially omitted, but it is assumed that basically both images and audio are processed.

Sensor systems 1210a-1210j each have one camera 1212a-1212j. That is, the image processing system 1200 has a plurality of cameras 1212 for photographing a subject from a plurality of directions. Note that the plurality of cameras 1212 will be described using the same reference numerals, but may differ in performance and model. A plurality of sensor systems 1210 are connected by a daisy chain. With this connection configuration, it is possible to reduce the number of connection cables and save wiring work when the resolution of captured images is increased to 4K or 8K and the capacity of image data is increased due to the increase in frame rate. I will mention it here.

Note that the connection configuration is not limited to this, and a star-shaped network configuration in which the sensor systems 1210a to 1210j are connected to the switching hub 1280 and data is transmitted and received between the sensor systems 1210 via the switching hub 1280 may be used.

Also, FIG. 12 shows a configuration in which all of the sensor systems 1210a to 1210j are cascaded to form a daisy chain, but this is not a limitation. For example, the plurality of sensor systems 1210 may be divided into several groups, and the sensor systems 1210 may be daisy-chained for each divided group. Then, the camera adapter 1220 that is the end of the division unit may be connected to the switching hub to input the image to the image computing server 1300 . Such a configuration is particularly effective in stadiums. For example, a stadium may have multiple floors and the sensor system 1210 may be deployed on each floor. In this case, it is possible to input to the image computing server 1300 for each floor or for each half circumference of the stadium. flexibility can be achieved.

In addition, control of image processing in the image computing server 1300 is switched depending on whether one or more camera adapters 1220 are daisy-chained to input images to the image computing server 1300. . That is, control is switched depending on whether the sensor system 1210 is divided into multiple groups. When there is one camera adapter 1220 for image input, an image of the entire circumference of the stadium is generated while performing image transmission through a daisy chain connection. is taken. That is, if the sensor system 1210 is not divided into groups, it will be synchronized.

However, when there are a plurality of camera adapters 1220 that perform image input, it is conceivable that the delay from when an image is captured to when it is input to the image computing server 1300 differs for each lane (path) of the daisy chain. That is, if the sensor system 1210 is divided into groups, the timing at which the image data for the entire circumference is input to the image computing server 1300 may not be synchronized. Therefore, in the image computing server 1300, it is necessary to check the collection of the image data and perform the subsequent image processing by synchronizing control that waits until all the image data are collected.

In this embodiment, sensor system 1210a includes microphone 1211a, camera 1212a, camera platform 1213a, external sensor 1214a, and camera adapter 1220a. Note that the configuration is not limited to this, and at least one camera adapter 1220a and one camera 1212a or one microphone 1211a may be provided. Further, for example, the sensor system 1210a may be configured with one camera adapter 1220a and multiple cameras 1212a, or may be configured with one camera 1212a and multiple camera adapters 1220a. That is, the plurality of cameras 1212 and the plurality of camera adapters 1220 in the image processing system 1200 correspond to N to M (both N and M are integers equal to or greater than 1). Also, the sensor system 1210 may include devices other than the microphone 1211a, the camera 1212a, the platform 1213a, and the camera adapter 1220a. Furthermore, at least a portion of the functionality of camera adapter 1220 may reside in front-end server 1330 . In this embodiment, the sensor systems 1210b to 1210j have the same configuration as the sensor system 1210a, so description thereof will be omitted. Note that the configuration is not limited to the same configuration as the sensor system 1210a, and each sensor system 1210 may have a different configuration.

The sound collected by the microphone 1211a and the image captured by the camera 1212a are subjected to various processing in the camera adapter 1220a, and then transmitted to the camera adapter 1220b of the sensor system 1210b through the daisy chain 1270a. be. Similarly, sensor system 1210b transmits the collected sound and the captured image together with the image and sound obtained from sensor system 1210a to sensor system 1210c.

The camera adapter 1220 performs processes such as foreground/background separation processing, foreground three-dimensional shape data information generation processing, and dynamic calibration on image data captured by the camera 1212 and image data received from other camera adapters 1220. . A silhouette image of the dynamic object is generated by the camera adapter 1220 based on foreground/background separation processing on the captured image. Also, based on a plurality of silhouette images received from other camera adapters 1220, three-dimensional shape data corresponding to the dynamic object is generated by the visual volume intersection method or the like. A plurality of three-dimensional shape data are integrated by an image computing server 1300, which will be described later. Note that the camera adapter 1220 does not generate three-dimensional shape data corresponding to dynamic objects, and the image computing server 1300 collectively generates three-dimensional shape data corresponding to a plurality of dynamic objects. good too. Note that the three-dimensional shape data referred to here is three-dimensional shape data corresponding to dynamic objects, unlike the three-dimensional shape data generated in the first and second embodiments described above. A dynamic object refers to an object that moves (its absolute position can change) when photographed from the same direction in time series, that is, a moving object. A dynamic object refers to, for example, a person or a ball in a ball game.

Continuing with the operations described above, images and sounds acquired by sensor systems 1210 a - 1210 j are transmitted to switching hub 1280 using sensor systems 1210 j - 1280 b and then to image computing server 1300 .

Although the cameras 1212a to 1212j and the camera adapters 1220a to 1220j are separated in this embodiment, they may be integrated in the same housing. In that case, the microphones 1211 a to 1211 j may be built into the integrated camera 1212 or may be connected to the outside of the camera 1212 .

Next, the configuration and operation of image computing server 1300 will be described. The image computing server 1300 of this embodiment processes the data obtained from the sensor system 1210j. The image computing server 1300 has a front end server 1330 , a database 1350 (hereinafter also referred to as DB), a back end server 1370 and a time server 1390 . The three-dimensional shape data corresponding to the field, which is the object plane generated in the first and second embodiments, is stored in the DB 1350 in advance.

The time server 1390 has a function of distributing time and synchronization signals, and distributes the time and synchronization signals to the sensor systems 1210a to 1210j via the switching hub 1280. FIG. The camera adapters 1220a to 1220j that have received the time and synchronization signal Genlock the cameras 1212a to 1212j based on the time and the synchronization signal to perform image frame synchronization. That is, the time server 1390 synchronizes the shooting timings of the multiple cameras 1212 . As a result, since the image processing system 1200 can generate a virtual viewpoint image based on a plurality of captured images captured at the same timing, it is possible to suppress deterioration in the quality of the virtual viewpoint image due to the deviation of the capturing timing. In this embodiment, the time server 1390 manages the time synchronization of a plurality of cameras 1212, but this is not limitative, and each camera 1212 or each camera adapter 1220 independently performs time synchronization processing. may

The front-end server 1330 reconstructs segmented transmission packets from the images and sounds acquired from the sensor system 1210j, converts the data format, and writes them to the DB 1350 according to the camera identifier, data type, and frame number. .

Next, the back-end server 1370 receives a viewpoint designation from the virtual camera operation UI 1430, reads data such as corresponding image and audio data from the DB 1350 based on the received viewpoint, and performs rendering processing to generate a virtual viewpoint image. to generate The read data includes three-dimensional shape data corresponding to stadiums, three-dimensional shape data corresponding to fields, and the like.

Note that the configuration of the image computing server 1300 is not limited to this. For example, at least two of the front-end server 1330, database 1350, and back-end server 1370 may be integrated. Also, at least one of the front-end server 1330, the database 1350, and the back-end server 1370 may be included in multiple numbers. Additionally, devices other than those described above may be included at any location within the Image Computing Server 1300 . Furthermore, at least part of the functions of the image computing server 1300 may be included in the end-user terminal 1290 or the virtual camera operation UI 1430 .

The rendered image is transmitted from the back-end server 1370 to the end-user terminal 1290, and the user operating the end-user terminal 1290 can view images and listen to audio according to the specified viewpoint. That is, the backend server 1370 generates virtual viewpoint content based on captured images (multi-viewpoint images) captured by a plurality of cameras 1212 and viewpoint information. More specifically, the back-end server 1370 generates a virtual viewpoint based on image data of a predetermined region extracted from images captured by the cameras 1212 by the camera adapters 1220 and a viewpoint specified by user operation. Generate content. The backend server 1370 then provides the generated virtual viewpoint content to the end user terminal 1290 . Note that in the present embodiment, the virtual viewpoint content is generated by the image computing server 1300, and the case where it is generated by the backend server 1370 will be mainly described. However, not limited to this, the virtual viewpoint content may be generated by a device other than the backend server 1370 included in the image computing server 1300 , or may be generated by the controller 1400 or the end user terminal 1290 .

The virtual viewpoint content in this embodiment is content that includes a virtual viewpoint image as an image obtained when a subject is photographed from a virtual viewpoint. In other words, it can also be said that the virtual viewpoint image is an image representing the appearance at the designated viewpoint. A virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to a viewpoint arbitrarily designated by the user. The virtual viewpoint image also includes an image corresponding to a viewpoint specified by the user from among a plurality of candidates and an image corresponding to a viewpoint automatically specified by the device.

In this embodiment, an example in which voice data (audio data) is included in the virtual viewpoint content will be mainly described, but voice data does not necessarily have to be included. Also, the back-end server 1370 may store virtual viewpoint images in H.264, for example. 264 or HEVC, and then transmitted to the end user terminal 1290 using the MPEG-DASH protocol. The virtual viewpoint images may also be sent to the end user terminal 1290 uncompressed. In particular, the former that performs compression encoding assumes a smartphone or tablet as the end user terminal 1290, and the latter assumes a display capable of displaying uncompressed images. That is, it should be clearly stated that the image format can be switched according to the type of end user terminal 1290 . Also, the image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or other transmission methods may be used.

Thus, the image processing system 1200 has three functional domains: an image collection domain, a data storage domain, and an image generation domain. The image collection domain includes sensor systems 1210-1210j. Also, the data storage domain includes database 1350 , front-end server 1330 and back-end server 1370 . Also, the image generation domain includes a virtual camera operation UI 1430 and an end user terminal 1290 . For example, the virtual camera operation UI 1430 can directly acquire images from the sensor systems 1210a to 1210j without being limited to this configuration. However, in this embodiment, the data storage function is placed in the middle rather than acquiring images directly from the sensor systems 1210a-1210j. Specifically, the front-end server 1330 converts the image data and audio data generated by the sensor systems 1210a to 1210j and the meta information of these data into the common schema and data type of the database 1350. FIG. As a result, even if the cameras 1212 of the sensor systems 1210a to 1210j are changed to cameras of other models, the front-end server 1330 can absorb the differences and register them in the database 1350. FIG. This reduces the risk that the virtual camera operation UI 1430 will not operate properly when the camera 1212 is changed to a camera of another model.

Also, the virtual camera operation UI 1430 is configured to access via the backend server 1370 without directly accessing the database 1350 . The backend server 1370 performs common processing related to image generation processing, and the virtual camera operation UI 1430 performs the difference part of the application related to the operation UI. As a result, in the development of the virtual camera operation UI 1430, it is possible to focus on the development of the functional requirements of the UI operation device and the UI for operating the virtual viewpoint image to be generated. Also, the backend server 1370 can add or delete common processing related to image generation processing in response to a request from the virtual camera operation UI 1430 . This makes it possible to flexibly respond to requests from the virtual camera operation UI 1430 .

As described above, in the image processing system 1200, the backend server 1370 generates a virtual viewpoint image based on image data captured by a plurality of cameras 1212 for capturing images of a subject from a plurality of directions. Note that the image processing system 1200 in this embodiment is not limited to the physical configuration described above, and may be configured logically.

<Other embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100 generation device 501 image acquisition unit 504 area setting unit 507 determination unit 508 generation unit 509 specific pattern acquisition unit

Claims (12)

a first obtaining means for obtaining a plurality of photographed images obtained by photographing an object plane from a plurality of directions;
a second acquiring means for acquiring information indicating the position and shape of the pattern on the object plane;
a three-dimensional position of each of the plurality of areas on the object plane based on the plurality of photographed images acquired by the first acquisition means and the information indicating the position and shape of the pattern acquired by the second acquisition means; a determining means for determining information;
generating means for generating three-dimensional shape data corresponding to the object plane based on the three-dimensional position information for each region determined by the determining means;
Projection means for generating projection images on projection planes having different distances from the subject plane for each region and for each distance from the subject plane, based on the plurality of captured images acquired by the first acquisition means;
synthesizing means for synthesizing a plurality of projection images having the same distance from the object plane to generate a synthetic image for each region and each distance from the object plane;
The determining means determines three-dimensional position information for each area based on the synthesized image for each area and for each distance from the object plane generated by the synthesizing means. Data generator. further comprising setting means for setting a plurality of areas on the subject plane based on the information indicating the position and shape of the pattern acquired by the second acquisition means;
2. The generating apparatus according to claim 1, wherein said determining means determines three-dimensional position information for each of a plurality of areas on said object plane set by said setting means. 3. The generating apparatus according to claim 2, wherein said setting means sets said plurality of areas such that each of said plurality of areas includes at least part of said pattern. The determination means determines three-dimensional position information corresponding to a composite image in which the pattern is sharp among the composite images for each region and for each distance from the object plane generated by the composite device. 4. The generation device according to claim 1, wherein the three-dimensional position information is determined as . further comprising third acquisition means for acquiring parameters of the imaging device;
for determining three-dimensional position information of the plurality of regions based on the information indicating the position and shape of the pattern acquired by the second acquiring means and the parameters acquired by the third acquiring means; further comprising calculating means for calculating the index of
5. The synthesizing unit, based on the index calculated by the calculating unit, synthesizes the plurality of projection images having the same distance from the object plane. The generator according to . When synthesizing the plurality of projected images having the same distance from the object plane, the synthesizing means uses the index calculated by the calculating means as a weight to determine the pixel value of the corresponding pixel in each projected image. 6. The generator according to claim 5, wherein weighted averaging is performed on the data. 7. The generator according to any one of claims 1 to 6, wherein the pattern is a line drawn on a field for playing a game. 8. The generation device according to claim 1, wherein the pattern has a color different from that of other areas on the object plane. a first obtaining step of obtaining a plurality of photographed images obtained by photographing an object plane from a plurality of directions;
a second acquisition step of acquiring information indicating the position and shape of the pattern on the object plane;
three-dimensionally for each of the plurality of regions on the object plane based on the plurality of photographed images obtained by the first obtaining step and the information indicating the position and shape of the pattern obtained by the second obtaining step; a determining step of determining location information;
a generation step of generating three-dimensional shape data corresponding to the object plane based on the three-dimensional position information for each region determined by the determination step;
a projecting step of generating a projection image on a projection plane having a different distance from the subject plane for each region and for each distance from the subject plane, based on the plurality of captured images acquired by the first acquiring step;
synthesizing a plurality of projected images having the same distance from the object plane to generate a synthetic image for each region and each distance from the object plane;
The determination step determines three-dimensional position information for each region based on the synthesized image for each region and for each distance from the object plane generated in the synthesis step. How the data is generated. further comprising a setting step of setting a plurality of areas on the subject plane based on the information indicating the position and shape of the pattern acquired in the second acquiring step;
10. The generating method according to claim 9, wherein said determining step determines three-dimensional position information for each of a plurality of areas on said object plane set by said setting step. 11. The generating method according to claim 10, wherein in the setting step, the plurality of areas are set such that each of the plurality of areas includes at least part of the pattern. A program for causing a computer to function each means of the generation device according to any one of claims 1 to 8.

Images