JP2020004053A - Generation apparatus, generation method, and program - Google Patents

Abstract

To acquire high precision three-dimensional shape data regardless of a pattern on a subject surface.SOLUTION: A generation apparatus includes: first acquisition means of acquiring multiple captured images acquired by imaging a subject surface from multiple directions; second acquisition means of acquiring information indicating position and shape of a pattern on the subject surface; determination means of determining three-dimensional position information for each area in the subject surface, on the basis of the captured images acquired by the first acquisition means and the information indicating position and shape of the pattern acquired by the second acquisition means; and generation means of generating three-dimensional shape data corresponding to the subject surface on the basis of the three-dimensional position information of each area determined by the determination means.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a generation device, a generation method, and a program for generating three-dimensional shape data.

  For various uses such as terrain analysis and street viewing, there is a demand to analyze images captured by a camera and acquire three-dimensional position information (three-dimensional shape data) of the terrain. In Patent Literature 1, three-dimensional shape data is obtained from aerial photographs obtained by photographing terrain from a plurality of directions using a stereo matching method.

International Publication WO08 / 152740

  However, in the stereo matching method as disclosed in Patent Document 1, since matching between pixels between images is used, it may not be possible to accurately obtain three-dimensional shape data depending on a pattern on a subject surface. . For example, for a subject surface having a pattern that is almost the same in color and extends in a certain direction, such as a line drawn on a competition field, the accuracy of the above-described matching is not improved, and the three-dimensional shape of the subject surface is not improved. Data cannot be acquired with high accuracy.

  An object of the present invention is to acquire high-precision three-dimensional shape data regardless of a pattern on a subject surface.

  The generating apparatus according to the present invention includes a first obtaining unit configured to obtain a plurality of captured images obtained by capturing an image of a subject surface from a plurality of directions, and a second obtaining unit configured to obtain information indicating a position and a shape of a pattern on the subject surface. Acquiring means, based on the plurality of captured images acquired by the first acquiring means, and information indicating the position and shape of the pattern acquired by the second acquiring means, for each of a plurality of regions on the object plane Determining means for determining three-dimensional position information; and generating means for generating three-dimensional shape data corresponding to the subject plane based on the three-dimensional position information for each area determined by the determining means. It is characterized by.

  According to the present invention, highly accurate three-dimensional shape data can be obtained regardless of a pattern on a subject surface.

FIG. 3 is a diagram illustrating an example of a configuration of a generation device that generates three-dimensional shape data of a subject surface. The figure which showed an example of each camera which comprises a camera group. FIG. 3 is a diagram for explaining the concept of the first embodiment. The schematic diagram of the shape of the line drawn in the field. FIG. 2 is a block diagram illustrating an example of a logical configuration of the generation device according to the first embodiment. 5 is a flowchart illustrating a flow of processing of the generation device according to the first embodiment. FIG. 9 is a schematic diagram illustrating a difference in how a line on a projected image looks due to a difference in height of a projection surface. FIG. 4 is a diagram for explaining camera reliability. FIG. 9 is a schematic diagram illustrating a composite image obtained by combining the projection images for each height of the projection plane. FIG. 9 is a block diagram showing an example of a logical configuration of a generation device according to the second embodiment. 9 is a flowchart illustrating a flow of processing of the generation device according to the second embodiment. FIG. 9 is a diagram illustrating a configuration example of an image processing system according to a third embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all combinations of the features described in the present embodiments are necessarily essential to the solution of the present invention. The same components will be described with the same reference numerals.

  Further, in the following embodiment, description will be made by taking an example of generation of three-dimensional shape data of a stadium field (ground), but application of the present invention is not limited to this. The present invention can also be applied to the acquisition of three-dimensional information on general terrain, road shapes, wall shapes, painting shapes, and mural shapes.

  In the present embodiment, the three-dimensional shape data is data representing a three-dimensional shape of a substantially flat subject surface. , Z represented by a point group having position information. In addition, the three-dimensional shape data is not limited to data represented by a point group, but may be represented by another data format. For example, a surface of a simple convex polygon (called a polygon) such as a triangle or a quadrangle may be used. May be represented by voxels or polygon mesh data.

  Further, the image in the present embodiment is image data, and need not necessarily be a viewable image generated for display on a display device such as a display.

[Embodiment 1]
FIG. 1 is a diagram illustrating an example of a configuration of a generation device 100 that generates three-dimensional shape data according to the present embodiment. The generation device 100 includes a CPU (Central Processing Unit) 101, a main memory 102, a storage unit 103, an input unit 104, a display unit 105, and an external I / F unit 106, and each unit is connected via a bus 107. First, the CPU 101 is an arithmetic processing unit that controls the generation apparatus 100 in an integrated manner, and performs various processes by executing various programs stored in the storage unit 103 and the like. The main memory 102 temporarily stores data, parameters, and the like used in various processes, and also provides the CPU 101 with a work area. The storage unit 103 is a large-capacity storage device that stores various programs and various data necessary for displaying a GUI (Graphical User Interface). For example, a non-volatile memory such as a hard disk or a silicon disk is used.

  The input unit 104 is a device such as a keyboard, a mouse, an electronic pen, and a touch panel, and receives an operation input from a user. The display unit 105 is configured by a liquid crystal panel or the like, and performs a GUI display of an analysis result and the like. The external I / F unit 106 is connected to each camera constituting the camera group 109 via the LAN 108, and transmits and receives video data and control signal data. The bus 107 connects the above-described units and performs data transfer.

  The generation device 100 is connected to a camera group 109 via a LAN 108. The camera group 109 starts and stops shooting, changes camera settings (shutter speed, aperture, etc.), and transfers captured video data based on a control signal from the generation device 100.

  The configuration of the generation device 100 includes various components other than the above, but is not the main focus of the present embodiment, and a description thereof will be omitted.

  FIG. 2 is a diagram showing an example of the arrangement of each camera constituting the camera group 109. Here, an example in which ten cameras are installed in a stadium is shown, but the number and installation positions of the cameras are not limited to this. A player 202 and a ball are present on a field 201 where a game is to be played, and ten cameras 203 a to 203 j are arranged around the field 201. In each of the cameras 203a to 203j constituting the camera group 109, appropriate camera orientation, focal length, exposure control parameters, and the like are set so that the entire field 201 or the attention area of the field 201 falls within the angle of view. I have.

  FIG. 3 is a diagram showing an outline of the present embodiment. In the present embodiment, first, in STEP. 1, the field is divided into a plurality of regions based on the two-dimensional position and shape of the line drawn in the field 201. FIG. 4 shows the shape of the line drawn in the field. This line refers to a pattern that extends in a certain direction and has a finite width in a direction perpendicular to the direction. The length and width of the line are determined by competition standards. In the present embodiment, the field 201 is virtually divided into a plurality of divided regions so as to include the line in order to acquire the height of the field for each region based on a specific pattern (pattern) such as a line. . This division may be performed on a photographed image obtained by photographing with a camera. FIG. 3A shows an example in which the image is divided into six divided areas (S1 to S6). Note that the divided area may be set virtually for a captured image or may be set virtually for a projected image described later.

  The information regarding the line as the specific pattern (for example, information indicating the position on the field, the shape such as the length in the stretching direction, and the width in the direction perpendicular to the stretching direction) is generated in advance by the generation device via the input unit 104. 100 is input. However, the captured image and the subsequent STEP. The generation apparatus 100 may determine the projection image from the projection image output in step 2. In the present embodiment, a case will be described in which information about a line is input in advance and the generation apparatus 100 acquires the information.

  Next, STEP. A projection image on a projection plane having a different distance (height) from a field, which is a subject plane, as shown in FIG. 2, is generated based on an image captured by a camera. FIG. 3B shows an example in which the divided region S1 is photographed by all the cameras 1 to 10. In this case, for the area S1, the captured images acquired by the camera 1 are projected onto a plurality of projection planes having different heights from the field, and a plurality of projection images are generated. For example, when the height is set at 5 cm intervals and the height of the center point of the field is set to 0 cm, and projected onto a projection surface from -15 cm to +15 cm to generate a projection image, a captured image obtained by the camera 1 Seven projection images are generated. When this projection image is generated from ten cameras 1 to 10 capturing the divided area S1, a total of 70 (= 7 × 10) projection images are generated for each distance of the subject surface in the divided area S1. Is done. Similarly, a plurality of projection images are generated in other divided regions. However, for example, if a certain divided area has not been photographed by the camera 3, a projection image is generated from a photographed image of a camera other than the camera 3.

  The height of the projection plane may be based on an arbitrary point on the field. For example, a plane parallel to a parallel plane including the reference point of the field may be set as the projection plane, with the center point of the field as the reference point. Specifically, the reference point is defined as (0, 0, 0) in three-dimensional coordinates, an area on the subject plane near the reference point is defined as a reference plane which is an xy plane, and a distance in the z direction from the reference plane is determined. Alternatively, the projection plane may be set.

  STEP. By changing the height from the field according to 2, the position of the line in the projection image changes like the plurality of projection images of the camera 1 in FIG. This is because the camera 1 captures the divided area S1 from a direction perpendicular to the line extending direction. Then, in a projected image having a different height from the field in the projected image, when the height of the projected surface changes from the correspondence relationship between the pixel position of the captured image and the position on the projection surface, the projection image is moved in a direction perpendicular to the stretching direction. The position of the line in each projection image changes.

  On the other hand, in the plurality of projection images of the camera 10 of FIG. 3B, the position of the line in the projection image does not seem to change. This is because the camera 10 photographs the divided area S1 from a direction parallel to the line extending direction. In this case, the position of the line in the projected image changes in a direction parallel to the stretching direction when the projection surface changes, due to the correspondence between the pixel position of the captured image and the position on the projection surface. Seems unchanged.

  Note that STEP. 1 and STEP. The order of 2 may be changed. Specifically, the projection images of the cameras 203a to 203j may be generated for each height from the field, and the projection image for each height may be divided into regions.

  Next, in this embodiment, STEP. As shown in 3, the projection image is synthesized for each set area and for each height from the field to generate a synthesized image. As shown in FIGS. 3B and 3C, in the region S1, for example, at a height of +15 cm, ten images of the projection images 301Aa to 301Aj generated from the captured images of the cameras 1 to 10 are combined. Thus, a composite image 302A is generated. Other composite images (for example, composite images 302D and 302G) are similarly generated. Here, “A” of 301Aa represents the height, and if the “A” is the same, it indicates that the height from the field is the same. “A” indicates a camera number, and the same “a” means a projection image based on a captured image obtained by the same camera.

  Next, STEP. As shown in FIG. 4, from among the composite images 302A to 302G for each height from the field of the divided area S1, the composite image with the sharpest line is determined. Then, the height from the field corresponding to the determined composite image is determined as the height from the field in the divided area S1. This STEP. By performing the process 4 for each set area, the height from the field in each divided area is determined. The height from the field in each divided area may be determined based on another evaluation value instead of the sharpness of the image.

  Finally, STEP. As shown in 5, the three-dimensional shape data of the field is generated based on the determined height from the field in each divided area. Hereinafter, the processing performed by the generation device 100 will be described in detail.

  FIG. 5 is a block diagram illustrating a functional configuration of the generation device 100. In the present embodiment, the generation device 100 generates three-dimensional shape data of a stadium field. The generation device 100 includes an image acquisition unit 501, a camera parameter acquisition unit 502, a projection unit 503, an area setting unit 504, a reliability calculation unit 505, a synthesis unit 506, a distance determination unit 507, a generation unit 508, and a specific pattern acquisition unit 509. Having.

  The image acquisition unit 501 acquires a plurality of captured images captured by the camera group 109. The camera group 109 includes ten cameras 203a to 203j shown in FIG. Then, the image obtaining unit 501 obtains a captured image from each of the cameras 203a to 203j. The image acquisition unit 501 outputs a captured image to the camera parameter acquisition unit 502 and the projection unit 503.

  The camera parameter acquisition unit 502 performs camera calibration from the captured image output from the image acquisition unit 501, and acquires camera parameters including external parameters, internal parameters, and distortion parameters of the camera. The external parameters are parameters representing the position and orientation of the camera, such as a rotation matrix and a position vector. The internal parameters are parameters unique to the camera, such as the focal length and the image center. The camera parameter acquisition unit 502 outputs camera parameters to the projection unit 503 and the reliability calculation unit 505.

  The projection unit 503 is based on a plurality of captured images output from the image acquisition unit 501, camera parameters output from the camera parameter acquisition unit 502, and information indicating a setting area output from an area setting unit 504 described below. And generate a projection image. As the projection image, each of the captured images is projected onto a projection plane having a different distance from the object plane, and a projection image is generated for each set area and each distance from the object plane. The projection unit 503 outputs a projection image for each set area and for each distance from the subject plane to the synthesis unit 506.

  The specific pattern acquisition unit 509 acquires information on the specific pattern from outside. The information on the specific pattern includes position information of the specific pattern on the subject surface, information indicating the shape of the specific pattern, information indicating a color difference between another region of the subject surface and the specific pattern, and the like. In the case where the specific pattern is a line drawn in a field, the extending direction and length of the line and the width in the direction perpendicular to the extending direction of the line are information indicating the shape of the line. The specific pattern acquisition unit 509 outputs information on the specific pattern to the area setting unit 504 and the distance determination unit 507.

  The region setting unit 504 virtually sets a plurality of regions on the subject plane for determining three-dimensional position information of the subject plane based on the information on the specific pattern output from the specific pattern acquisition unit 509. Specifically, the region setting unit 504 virtually sets the region such that at least a part of the specific pattern is included in each of the plurality of setting regions. Here, an example is shown in which the set area is set on the projection image, but the area may be set in the captured image. The region setting unit 504 outputs information indicating a plurality of setting regions for determining the distance of the reference point on the subject plane to the projection unit 503 and the reliability calculation unit 505.

  The reliability calculation unit 505 calculates the camera reliability for each area set by the area setting unit 504 and for each camera (each projection image). The camera reliability is calculated based on the camera parameters output from the camera parameter obtaining unit 502, the information indicating the plurality of setting areas output from the area setting unit 504, and the position and shape of the specific pattern output from the specific pattern obtaining unit 509. Is calculated based on the information indicating The camera reliability is used when a synthesizing unit 506 described later synthesizes a projected image. The reliability calculation unit 505 outputs the camera reliability to the synthesis unit 506.

  The synthesizing unit 506 synthesizes a projection image for each set area on the projection plane having the same distance from the subject plane based on the camera reliability output from the reliability calculation unit 505, and generates a synthesized image. The camera reliability is used as a weight of each projected image when performing synthesis by weighted averaging processing. Here, the weighted averaging process refers to calculating a pixel value by averaging the pixel values of the corresponding pixels of each of the plurality of projection images with weight. The combining unit 506 outputs a combined image for each set area and for each distance from the subject plane to the distance determination unit 507.

  The distance determining unit 507 is based on the synthesized image output from the synthesizing unit 506 for each set area and each distance from the subject plane, and information indicating the position and shape of the specific pattern output from the specific pattern acquisition unit 509. To determine the distance from the object plane. More specifically, the distance determination unit 507 calculates an evaluation value of a specific pattern in a composite image for each set area and each distance from the subject plane, and determines a distance from the subject plane based on the calculated evaluation value. For example, the distance determination unit 507 can use the sharpness as the evaluation value. In this case, in a composite image for each distance from the object plane in a certain setting area, the distance from the object plane corresponding to the composite image having the highest sharpness of the specific pattern is determined as the distance from the object plane in the setting area. I do. The distance determining unit 507 outputs, to the generating unit 508, the distance from the reference point on the subject plane determined for each set area. The three-dimensional position information of the setting area is determined by the distance determining unit 507. That is, the xy coordinates of the setting area are coordinates based on the area setting unit 504, and the distance from the reference point on the subject plane determined by the distance determining unit 507 corresponds to the z coordinate.

  Based on the distance from the reference point of the object plane determined for each set area output from the distance determination unit 507, that is, the three-dimensional position information determined for each set area, the generation unit 508 generates Generate dimensional shape data. The generation unit 508 outputs three-dimensional shape data of the subject surface.

  Next, a process performed by the generation device 100 will be described in detail with reference to a flowchart illustrated in FIG. This series of processing is realized by the CPU 101 reading a predetermined program from the storage unit 103, expanding the program in the main memory 102, and executing the program.

  In step S601, camera parameters are obtained by a calibration process. First, the image acquisition unit 501 sends a shooting instruction to the camera group 109 via the LAN 108. The captured image is acquired by the image acquisition unit 501. As shown in FIG. 2, the camera group 109 includes a plurality of cameras 203a to 203j having different photographing directions. The camera parameter acquisition unit 502 calculates parameters of each camera of the camera group 109 from the image acquired by the image acquisition unit 501. The camera parameters are calculated by a camera calibration process in which a plurality of images at different photographing positions of the camera are input. Hereinafter, an example of a simple camera calibration procedure will be described.

  First, a plane pattern such as a square grid is photographed from multiple viewpoints. Second, feature points of the captured image are detected, and the coordinates of the feature points are determined in the image coordinate system. Here, the feature points of the square grid are the intersections of the straight lines. Third, an initial value of an internal parameter of the camera is calculated using the calculated feature point coordinates. Here, the internal parameters of the camera are parameters representing a focal length and an optical center called a principal point. Further, the initial values of the internal parameters of the camera do not always need to be calculated from the feature points in the image, and the design values of the camera may be used. Fourth, the camera calculates internal parameters, external parameters, and distortion coefficients by a non-linear optimization process called bundle adjustment. Here, the external parameters of the camera are parameters representing the position of the camera, the line-of-sight direction, and the rotation angle about the line-of-sight direction as an axis. Further, the distortion aberration coefficient is a coefficient representing distortion of an image in a radial direction caused by a difference in refractive index of a lens, and distortion in a circumferential direction caused by a lens and an image plane not being parallel. There are many other camera calibration methods, but the details are omitted because they are not the main focus of the present embodiment.

  In step S602, the image acquisition unit 501 sends a shooting instruction to the camera group 109 to shoot a field. The image acquisition unit 501 receives images captured by the plurality of cameras 203a to 203j that form the camera group 109 and have different shooting directions by shooting the subject surface.

  In S603, the specific pattern acquisition unit 509 acquires information on the specific pattern. Specifically, the specific pattern acquisition unit 509 extracts information including the shape of the line as the specific pattern and the position of the line on the subject surface based on the captured image acquired in S602.

  In step S604, based on the information on the specific pattern acquired in step S603, the region setting unit 504 sets a plurality of predetermined regions for determining a distance on the subject surface. Specifically, the area setting unit 504 sets the area so that each set area includes at least a part of the specific pattern. Further, the area setting unit 504 may divide the area so as to equally divide one line as the specific pattern. In the case of division, when the height of the projection plane changes, it is preferable to determine the area width so that the line is included in one area. The setting area may be set so that the object surface is entirely covered by the plurality of setting areas. That is, in order to calculate the height more densely, the setting areas may be set so as to overlap. Further, the setting area may be set so as to divide the object plane without overlapping. The set areas do not have to be the same size or the same shape as each other, and may have different sizes and shapes as long as they include a line that is a specific pattern.

  In step S605, based on the camera parameters acquired in step S601, the projection unit 503 uses a plurality of captured images for each of the plurality of setting areas set in step S604 to project a plurality of projection planes having different distances from the subject plane. Generate a projected image. Note that the projection unit 503 may project the captured image onto a different projection plane to generate a projection image, and then extract a projection image corresponding to the setting area from the projection image. Further, the projection unit 503 may extract an image from the captured image for each setting area, and then project the image on a projection plane to generate a projection image corresponding to the setting area.

  When performing camera calibration, the field 201 in FIG. 2 is used as a reference for a plane having a height of approximately 0 m, the direct axis direction of the field is the x axis, the short axis direction is the y axis, and the vertical direction of the field is the z axis. And set the origin at the center of the field. The projection plane is a plane that is horizontal to the field that is the subject plane. The range in the horizontal direction projected on the projection plane is determined using information indicating the position and shape of the line so as to cover the entire field. For example, based on the line shape in FIG. 4, projection is performed on a range of 80 m in length and 120 m in width. Of course, the projection may be performed with a surplus of several percent in consideration of the error between the actual field and the line shape.

  The calculation is performed for a plurality of projection planes having different heights so that the height of the entire field can be calculated. The range of the height to be projected is determined in accordance with the standard regarding the field gradient of the stadium. For example, it is assumed that there is a field standard such that the gradient from the center of the field to the edge of the field is up to 0.3%. In this case, if the distance from the origin of the field to the end of the field is 40 m, the allowable variation in height is up to 12 cm. Therefore, the range of the projected height is set to -15 cm to +15 cm or the like so as to cover this range. Within this range, the height increment can be set arbitrarily. By increasing the number of notches, that is, the number of projection surfaces, highly accurate three-dimensional shape data can be obtained.

  When generating a projection image, first, distortion correction of an image captured by each camera is performed in accordance with internal parameters and distortion parameters of the camera. The parameters used for the image distortion correction are the internal parameters calculated in S601 and the distortion parameters.

  Next, a transformation matrix between the coordinates of the projected image and the coordinates of the captured image is calculated. A transformation matrix from the world coordinate system where the projection plane exists to the camera coordinate system is defined as V. Here, the camera coordinate system is set such that the origin of the coordinate system is the starting point, the x-axis and the y-axis are the horizontal and vertical directions of the image, respectively, and the z-axis is the line of sight of the camera. Further, a transformation matrix from the camera coordinate system to the screen coordinate system is defined as P. This is a transformation matrix for projecting a subject plane having three-dimensional coordinates existing on the camera coordinate system onto a two-dimensional plane. That is, the equation (1) that projects the homogeneous coordinates (x, y, z, w) of the point X on the projected image onto the homogeneous coordinates (x ′, y ′, z ′, w ′) of the point U on the captured image. ) Is as follows.

  Here, coordinates w and w 'were added to perform the translational transformation, and the coordinates were set to four-dimensional coordinates. Using this equation (1), the images captured by the cameras are respectively projected onto projection planes having different heights z to generate projection images. Specifically, a projected image is generated by setting the pixel value of each coordinate of the captured image to the pixel value of the coordinate of the projected image corresponding to each coordinate of the captured image.

  In S606, the reliability calculation unit 505 calculates the camera reliability from the camera parameters acquired in S601 and the specific pattern acquired in S603 for each of the setting areas set in S604. Here, the camera reliability is an index indicating how useful a captured image obtained from each camera is in determining the distance when determining the distance of the object plane for each set area. A simple example is shown in FIG.

  FIG. 7 is a schematic diagram in which a field is photographed by four cameras (701 to 704). Here, the height of the rectangular area 705 set at the center of the field is determined. When the height of the rectangular area 705 changes, and a projection image is generated from images captured by the cameras 701 and 703 having a line-of-sight vector (optical axis) perpendicular to the line extending direction, the position of the line in each projection image Changes greatly. On the other hand, when the projection images are generated by changing the height of the projection plane from the images captured by the cameras 702 and 704 having a line-of-sight vector (optical axis) parallel to the line extending direction, the position of the line in each projection image is Hardly change.

  In the present embodiment, as described later, a composite image is generated by synthesizing the projection images of the cameras for each set area and for each height from the field plane. In order to calculate an evaluation value such as a line shift or a degree of blur in the composite image, processing such as selectively using an image in which such characteristics are likely to appear in the composite image, or performing weighting and then compositing is performed. It is desirable. The camera reliability is defined for calculating the weight at the time of selecting and combining the images.

  Therefore, as shown in FIG. 8, when the camera 802 is installed with respect to the line 801 and the position of the camera with respect to the center of the line is defined by the horizontal angle φ and the elevation angle θ, for example, the camera reliability ω becomes It is represented by 2).

  This means that a camera having a line of sight vector that is close to perpendicular to the direction of the line and having a smaller elevation angle θ has higher camera reliability. That is, when determining the height of the rectangular area 705, the camera reliability of the cameras 701 and 703 shown in FIG. 7 increases. As is clear from equation (2), the camera reliability of each camera changes for each set area, and therefore the camera reliability is calculated for each set area and each camera. In addition, the camera reliability is a reliability indicating a distance from a subject plane, that is, how useful a captured image or a projected image corresponding to each camera is to determine three-dimensional position information of the subject plane. . Further, the camera reliability is an index for determining the distance from the reference point of the object plane for each set area, that is, the three-dimensional position information of the object plane. This is an index indicating the degree of usefulness when determining the distance.

  As can be seen from equation (2), if the specific pattern is a line, the angle between the line extending direction and the linear direction when the optical axis of the camera is projected into the plane in the line extending direction. A camera with a larger (90 ° −φ) has a higher camera reliability. In addition, a camera having a smaller elevation angle θ has a higher camera reliability.

  The method for determining the camera reliability is not limited to the above method. Depending on the physical distance d from the camera to the line, the focal length f, the number of pixels p, etc., whether or not the line can be clearly captured by the camera will differ, and the reliability of the camera will increase according to those parameters. May be. Alternatively, the camera reliability ω may be calculated by combining a camera capable of clearly capturing a line and a camera having a small horizontal angle and a small elevation angle to increase the reliability. The camera reliability ω by this combination is as shown in Expression (3). Here, α and β are weight parameters.

  Also, threshold processing may be performed, such as setting the camera reliability whose camera reliability is lower than a predetermined threshold to 0.

  In step S607, the combining unit 506 combines the plurality of projection images acquired in step S605 with the plurality of projection images on the same projection plane for each set area based on the camera reliability acquired in step S606. More specifically, first, since the area where each camera can shoot is different, it is determined whether or not each camera can shoot the set area. Specifically, in the projection image for each set area, if there is a pixel value obtained by projecting the captured image to all the pixels in each area, it is assumed that the set area can be shot by the corresponding camera. This determination may be performed by the region setting unit 504 in S604, or may be performed by the projection unit 503 in S605.

  Next, the projection images are synthesized for each projection plane using the projection images of the plurality of cameras that can be photographed for each set area. Specifically, a weighted averaging process based on the camera reliability ω for each set area calculated in S606 is performed to generate a composite image. That is, with the projection image as an rgb image, a composite image of each setting area Bj at the height h of the projection plane is expressed by Expression (4). Note that the camera number is k.

  As described in the processing of S606, it is not always necessary to generate a weighted average image using all camera reliability. For example, the averaging process is performed using only the projection image corresponding to the camera whose camera reliability is equal to or less than the predetermined threshold or using only the projection image corresponding to the camera having the predetermined threshold or more to generate a composite image. May be. In addition, even if an average process is not performed by using the camera reliability to generate a composite image, a composite image is generated by using a simple average value or an intermediate value of pixel values of corresponding pixels of a projection image. May be.

  In step S607, the distance determination unit 507 determines the distance from the reference point of the object plane for each set area, that is, 3 for each set area, based on the synthesized image for each set area and each projection plane synthesized in S607. Determine dimensional position information. When determining the distance from the reference point of the object plane, the evaluation value of the specific pattern for each set area of the composite image is evaluated. The evaluation value is specifically described below using the sharpness, but is not limited to this.

  FIG. 9 shows an example of a composite image for each distance (height) from the reference point on the subject plane in a certain setting area. Specifically, FIG. 9 shows an example in which the projection plane is set by changing the height in steps of 0.5 cm from −15 cm to +15 cm with respect to the subject plane, and a composite image is generated at each height. ing. As shown in FIG. 9, in the composite image at an appropriate height (0 cm) among the composite images, the lines look clear. This is because the positions of the lines included in each of the plurality of projection images used for the composition overlap at substantially the same position. Overlapping positions mean that the actual line height is at its projection plane. On the other hand, in the composite image at other heights, the positions of the lines included in each of the plurality of projection images used for the composition are shifted, so that the lines are blurred.

  Therefore, the actual height of the line can be determined by evaluating the sharpness of the line. In order to evaluate the sharpness of the image, for example, a filter such as a Laplacian filter L is used. The sharpness Sj, h of the composite image of a certain setting area Bj at the height h is represented by Expression (5).

  Note that the filter for evaluating the sharpness of the image is not limited to the Laplacian filter, and a primary differential filter, a Prewitt filter, a Sobel filter, or the like may be used. Further, a difference between the combined image subjected to the smoothing filter and the original combined image not subjected to the filter is calculated, and a combined image having a height such that the difference is increased is determined as an appropriate height of the setting area. You may do so.

  After evaluating the sharpness, there are several ways to determine the height. For example, the height Hj at which the sharpness is maximized is determined as an appropriate height of the setting area Bj (see Expression (6)).

  As an index of whether or not an appropriate height has been correctly calculated, for example, it may be determined whether or not the sharpness changes smoothly when the height is changed. Specifically, it is determined using Equation (7) whether or not the sharpness changes smoothly around the height where the sharpness is maximum. Here, the height width (step) at which the projection image is generated is defined as a. If equation (7) is satisfied. It is determined that the height is the appropriate height of the setting area Bj. If the height is not satisfied, the same determination is performed for the height corresponding to the synthesized image having the next highest sharpness.

  Further, when a plurality of sharpness levels close to the maximum sharpness level are calculated, the heights of the projection planes corresponding to the sharpness levels may be averaged and calculated as an appropriate height of the setting area.

  Furthermore, the continuity of the height with the adjacent setting area may be used as the constraint condition. Since the height of the object plane changes smoothly, the difference in height between adjacent setting areas should be small. By combining this constraint condition and the sharpness described above, the likelihood Mj, h of the height hj in the region Bj can be calculated by Expression (8). Here, α and β are weight parameters. Then, the height at which the likelihood becomes maximum may be determined as the optimum height of the setting area.

  The first term on the right side of Expression (8) is an index indicating the continuity of the distance from the object plane and indicating the change in the distance between two adjacent setting areas from the object plane.

  In this way, an appropriate height in each setting area is determined using the composite image. In addition, in the setting areas set by the area setting unit 504 and including the portions where the setting areas overlap with each other, the distance determining unit 507 first calculates the distance from the reference point of the vertex of each setting area to the subject plane. Determined by the above method. Then, the distance determination unit 507 determines an intermediate value or an average value of the distances of the vertexes of the respective setting areas from the reference point of the object plane as the distances from the reference points of the object plane at the vertices of the overlapping portion. Is also good.

  In step S609, the generation unit 508 generates three-dimensional shape data of the subject surface in accordance with the height of each setting area determined in step S608. Specifically, assuming that a vertex exists at the geometric center position of the setting area, the vertex coordinates are changed according to the height determined in S608. In this case, for an area in which only cameras with low reliability are used for height calculation, processing such as not using it as a vertex when generating three-dimensional shape data may be performed. For regions other than those on the line, vertex coordinates can be generated using the vertex coordinates of the line portion. For example, the height vz of the vertex v of the region not on the line can be calculated by a weighted average of the distance from the vertex v ′ existing in the neighboring region Ω of the vertex (expression (9), expression (10) ), Equation (11)).

  In the method of expressing the three-dimensional shape data, the shape may be expressed as a point group using only the calculated vertices. In this case, it is expressed by a point group having x, y, and z position information in a three-dimensional space in a world coordinate space that uniquely indicates an imaging space. In addition, the three-dimensional shape data may generate a surface connecting the geometric center positions of the setting area, and may be expressed as polygon mesh data as a set of a plurality of surfaces. Further, the three-dimensional shape data may be represented by voxels.

  As described above, in the present embodiment, the area setting unit 504 sets the area for determining the three-dimensional position information so as to include at least a part of the specific pattern (pattern), and uses the specific pattern (pattern). Thus, three-dimensional position information of the object plane is determined. Therefore, three-dimensional shape data can be generated with high accuracy.

  Although the pattern in the present embodiment has been described by taking the line drawn in the field as an example, the pattern is not limited to this. For example, the pattern may include a figure, a sign, a painting, and the like. In addition, the pattern may be an artificially made pattern or a naturally made pattern. Further, it is desirable that the pattern has a different color from the color of another area different from the pattern on the subject surface.

[Embodiment 2]
In the first embodiment, the height of each set area is determined after the projection images are combined. In the present embodiment, an embodiment will be described in which the height of each set area is determined without combining projected images. FIG. 10 is a block diagram illustrating a functional configuration of the generation device 1000 according to the present embodiment. FIG. 11 is a flowchart of a process performed by the generation device 1000. 10 and 11, the same components as those of the first embodiment are denoted by the same reference numerals.

  The generation apparatus 1000 includes an image acquisition unit 501, a camera parameter acquisition unit 502, a projection unit 503, an area setting unit 504, a reliability calculation unit 1001, a synthesis unit 506, a distance determination unit 1002, a generation unit 508, and a specific pattern acquisition unit 509. Having. Although the reliability calculation unit 505 of the first embodiment outputs the camera reliability to the combining unit 506, the reliability calculation unit 1001 of the present embodiment is different from the first embodiment only in that the camera reliability is output to the distance determination unit 1002. different.

  In FIG. 11, the processing (S601 to S606) until the camera reliability is calculated and the processing (S609) for generating the three-dimensional shape data are the same as those in the first embodiment, and thus description thereof will be omitted. Hereinafter, the process of S1101 will be specifically described.

  In step S1101, the distance determination unit 1002 determines an appropriate distance from the camera reliability calculated in step S605 and the projection images for each set area and each distance from the subject plane acquired in step S605. First, as described in the first embodiment, since the area where each camera can shoot is different, it is determined whether each camera can shoot the set area.

  Next, a line existing in the set area is detected. Since the line is clearly drawn on the lawn or the ground in a predetermined standard, the line can be easily extracted by color detection or a process of extracting a region having a large luminance. When the height of the projection plane is changed, the position of the line greatly changes between projection images corresponding to cameras with high camera reliability. However, when the image is projected at an appropriate height of the projection plane, the positions of the lines are almost the same regardless of the projection image corresponding to any camera. That is, for the height h, the height H of the projection plane that maximizes the intersection of the existing areas Wi, j of the lines of the arbitrary areas Bi, j, h of the projection image of the camera i is determined as the optimal setting area. The height can be determined (see equation (12)). Note that a method for determining an appropriate height of the setting area is not limited to this method. For example, the height may be calculated so that the union of the existing areas of the lines is minimized (see Expression (13)). Note that the region where the line exists is the region where the line in the projected image is drawn.

  When the continuity of the adjacent set areas is used as the constraint condition, the continuity of the existing area of the line of the projected image with high camera reliability may be considered. For example, when the area is divided so that the area partially overlaps, a constraint condition that employs a height at which the intersection of the existing areas of the adjacent lines becomes the maximum may be added.

[Embodiment 3]
Hereinafter, an image processing system that generates a virtual viewpoint image according to the present embodiment will be described. The three-dimensional shape data of the field generated in the above-described embodiment is used when generating a virtual viewpoint image.

  A system in which a plurality of cameras and microphones are installed in a facility such as a stadium or a concert hall to capture and collect sound will be described with reference to a system configuration diagram in FIG. The image processing system 1200 includes sensor systems 1210a to 1210j, an image computing server 1300, a controller 1400, a switching hub 1280, and an end user terminal 1290.

  The controller 1400 has a control station 1410 and a virtual camera operation UI 1430. The control station 1410 performs operation state management and parameter setting control for each block constituting the image processing system 1200 through the networks 1410a to 1410c, 1391, 1280a, 1280b, and 1270a to 1270i. Here, the network may be GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard, which is Ethernet (registered trademark, hereinafter abbreviated), or may be configured by combining interconnect Infiniband, industrial Ethernet, and the like. The network is not limited to these, and may be another type of network.

  First, an operation of transmitting ten sets of images and sounds of the sensor systems 1210a to 1210j from the sensor system 1210j to the image computing server 1300 will be described. In the image processing system 1200 of the present embodiment, the sensor systems 1210a to 1210j are connected by a daisy chain.

  In the present embodiment, unless otherwise specified, the sensor system 1210 is described without distinguishing ten sets of the sensor systems 1210a to 1210j. Similarly, the devices in each sensor system 1210 will be described as a microphone 1211, a camera 1212, a camera platform 1213, an external sensor 1214, and a camera adapter 1220 without distinction unless otherwise specified. Although the number of sensor systems is described as six sets, this is merely an example, and the number is not limited to this. Each of the cameras 1212a to 1212j of the imaging system is arranged at a position other than the symmetric position of a different camera.

  Further, the plurality of sensor systems 1210 do not have to have the same configuration, and for example, each may be configured by a device of a different model. In the present embodiment, unless otherwise specified, the term image is described as including the concept of a moving image and a still image. That is, the image processing system 1200 of the present embodiment can process both still images and moving images. In the present embodiment, an example will be mainly described in which the virtual viewpoint content provided by the image processing system 1200 includes a virtual viewpoint image and a virtual listening point sound, but the present invention is not limited to this. For example, sound may not be included in the virtual viewpoint content. Also, for example, the sound included in the virtual viewpoint content may be sound collected by a microphone closest to the virtual viewpoint. In the present embodiment, for simplicity of description, description of audio is partially omitted, but it is assumed that both image and audio are basically processed.

  Each of the sensor systems 1210a to 1210j has one camera 1212a to 1212j. That is, the image processing system 1200 has a plurality of cameras 1212 for photographing a subject from a plurality of directions. The plurality of cameras 1212 will be described using the same reference numerals, but may have different performances and models. The plurality of sensor systems 1210 are connected by a daisy chain. With this connection form, it is possible to reduce the number of connection cables and to save labor for wiring work in increasing the resolution of a captured image to 4K or 8K and increasing the capacity of image data accompanying a higher frame rate. It is specified here.

  The connection configuration is not limited to this, and a star-type network configuration in which the sensor systems 1210a to 1210j are connected to the switching hub 1280 and data is transmitted and received between the sensor systems 1210 via the switching hub 1280 may be used.

  Further, FIG. 12 shows a configuration in which all of the sensor systems 1210a to 1210j are cascaded to form a daisy chain, but the present invention is not limited to this. For example, a plurality of sensor systems 1210 may be divided into some groups, and the sensor systems 1210 may be daisy-chained in divided groups. Then, the camera adapter 1220 which is the end of the division unit may be connected to the switching hub to input an image to the image computing server 1300. Such a configuration is particularly effective in a stadium. For example, a case is considered in which a stadium is configured with a plurality of floors, and a sensor system 1210 is provided for each floor. In this case, the input to the image computing server 1300 can be performed for each floor or for every half turn of the stadium, and the installation and simplification of the system can be achieved even in a place where wiring is difficult to connect all the sensor systems 1210 with one daisy chain. Can be made more flexible.

  In addition, control of image processing in the image computing server 1300 is switched according to whether there is one or more camera adapters 1220 connected in a daisy chain and inputting an image to the image computing server 1300. . That is, the control is switched according to whether the sensor system 1210 is divided into a plurality of groups. When the number of camera adapters 1220 for inputting an image is one, an image of the entire circumference of the stadium is generated while performing image transmission by daisy chain connection. Has been taken. That is, if the sensor system 1210 is not divided into groups, synchronization is achieved.

  However, when a plurality of camera adapters 1220 perform image input, the delay from when an image is captured to when it is input to the image computing server 1300 may be different for each daisy chain lane (path). That is, when the sensor system 1210 is divided into groups, the timing at which the image data of the entire circumference is input to the image computing server 1300 may not be synchronized. Therefore, it is specified that the image computing server 1300 needs to perform image processing in the subsequent stage while checking the aggregation of the image data by the synchronization control that waits until the image data of the entire circumference is prepared and synchronizes.

  In the present embodiment, the sensor system 1210a has a microphone 1211a, a camera 1212a, a camera platform 1213a, an external sensor 1214a, and a camera adapter 1220a. Note that the present invention is not limited to this configuration, and it is sufficient that at least one camera adapter 1220a and one camera 1212a or one microphone 1211a are provided. Further, for example, the sensor system 1210a may be configured with one camera adapter 1220a and a plurality of cameras 1212a, or may be configured with one camera 1212a and a plurality of camera adapters 1220a. That is, the plurality of cameras 1212 and the plurality of camera adapters 1220 in the image processing system 1200 correspond to N to M (N and M are both integers of 1 or more). Further, the sensor system 1210 may include devices other than the microphone 1211a, the camera 1212a, the camera platform 1213a, and the camera adapter 1220a. Further, at least a part of the functions of the camera adapter 1220 may be included in the front-end server 1330. In the present embodiment, the sensor systems 1210b to 1210j have the same configuration as that of the sensor system 1210a, and a description thereof will be omitted. Note that the configuration is not limited to the same as the sensor system 1210a, and each sensor system 1210 may have a different configuration.

  The sound collected by the microphone 1211a and the image captured by the camera 1212a are subjected to various processes in the camera adapter 1220a, and then transmitted to the camera adapter 1220b of the sensor system 1210b through the daisy chain 1270a. You. Similarly, the sensor system 1210b transmits the collected sound and the captured image to the sensor system 1210c together with the image and sound obtained from the sensor system 1210a.

  The camera adapter 1220 performs processing such as foreground / background separation processing, foreground three-dimensional shape data information generation processing, and dynamic calibration on image data captured by the camera 1212 and image data received from another camera adapter 1220. . The camera adapter 1220 generates a silhouette image of the dynamic object based on the foreground / background separation process on the captured image. Further, based on a plurality of silhouette images received from another camera adapter 1220, three-dimensional shape data corresponding to the dynamic object is generated by a volume intersection method or the like. A plurality of three-dimensional shape data is integrated by an image computing server 1300 described later. Note that the camera adapter 1220 does not generate three-dimensional shape data corresponding to a dynamic object, but instead generates three-dimensional shape data corresponding to a plurality of dynamic objects collectively by the image computing server 1300. Is also good. The three-dimensional shape data referred to here is different from the three-dimensional shape data generated in the first and second embodiments, and is three-dimensional shape data corresponding to a dynamic object. A dynamic object refers to an object that moves (its absolute position may change) when images are taken in the same direction in a time series, that is, a moving object. The dynamic object refers to, for example, a person or a ball in a ball game.

  By continuing the above-described operation, the images and sounds acquired by the sensor systems 1210a to 1210j are transmitted to the switching hub 1280 using the sensor systems 1210j to 1280b, and then transmitted to the image computing server 1300.

  In this embodiment, the cameras 1212a to 1212j and the camera adapters 1220a to 1220j are separated from each other, but they may be integrated in the same housing. In that case, the microphones 1211a to 1211j may be built in the integrated camera 1212, or may be connected to the outside of the camera 1212.

  Next, the configuration and operation of the image computing server 1300 will be described. The image computing server 1300 according to the present embodiment processes data acquired from the sensor system 1210j. The image computing server 1300 includes a front-end server 1330, a database 1350 (hereinafter, also referred to as DB), a back-end server 1370, and a time server 1390. Note that the three-dimensional shape data corresponding to the field, which is the object plane, generated in the first and second embodiments is stored in the DB 1350 in advance.

  The time server 1390 has a function of distributing a time and synchronization signal, and distributes the time and synchronization signal to the sensor systems 1210a to 1210j via the switching hub 1280. The camera adapters 1220a to 1220j that have received the time and the synchronization signal perform Genlock of the cameras 1212a to 1212j based on the time and the synchronization signal, and perform image frame synchronization. That is, the time server 1390 synchronizes the shooting timings of the cameras 1212. Accordingly, since the image processing system 1200 can generate a virtual viewpoint image based on a plurality of captured images captured at the same timing, it is possible to suppress a decrease in the quality of the virtual viewpoint image due to a shift in the capturing timing. In this embodiment, the time server 1390 manages the time synchronization of the plurality of cameras 1212. However, the present invention is not limited to this, and each camera 1212 or each camera adapter 1220 performs processing for time synchronization independently. You may.

  The front-end server 1330 reconstructs a segmented transmission packet from the image and sound acquired from the sensor system 1210j to convert the data format, and then writes the data in the DB 1350 according to the camera identifier, data type, and frame number. .

  Next, the back-end server 1370 accepts the designation of a viewpoint from the virtual camera operation UI 1430, reads out corresponding data such as image and audio data from the DB 1350 based on the accepted viewpoint, performs rendering processing, and performs a virtual viewpoint image. Generate The read data includes three-dimensional shape data corresponding to a stadium, three-dimensional shape data corresponding to a field, and the like.

  Note that the configuration of the image computing server 1300 is not limited to this. For example, at least two of the front-end server 1330, the database 1350, and the back-end server 1370 may be integrally configured. Further, a plurality of at least one of the front-end server 1330, the database 1350, and the back-end server 1370 may be included. Further, a device other than the above devices may be included at an arbitrary position in the image computing server 1300. Furthermore, at least a part of the functions of the image computing server 1300 may be included in the end user terminal 1290 or the virtual camera operation UI 1430.

  The image subjected to the rendering processing is transmitted from the back-end server 1370 to the end user terminal 1290, and the user operating the end user terminal 1290 can perform image browsing and audio viewing according to the designation of the viewpoint. That is, the back-end server 1370 generates virtual viewpoint content based on the captured images (multiple viewpoint images) captured by the plurality of cameras 1212 and the viewpoint information. More specifically, the back-end server 1370 generates a virtual viewpoint based on image data of a predetermined area extracted from images captured by a plurality of cameras 1212 by a plurality of camera adapters 1220 and a viewpoint specified by a user operation. Generate content. Then, the backend server 1370 provides the generated virtual viewpoint content to the end user terminal 1290. Note that, in the present embodiment, the virtual viewpoint content is generated by the image computing server 1300, and a description will be mainly given of a case where the virtual viewpoint content is generated by the back-end server 1370. However, not limited to this, the virtual viewpoint content may be generated by a device other than the back-end server 1370 included in the image computing server 1300, or may be generated by the controller 1400 or the end-user terminal 1290.

  The virtual viewpoint content according to the present embodiment is a content including a virtual viewpoint image as an image obtained when a subject is photographed from a virtual viewpoint. In other words, it can be said that the virtual viewpoint image is an image representing the appearance at the designated viewpoint. The virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on a result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to the viewpoint arbitrarily specified by the user. In addition, an image corresponding to a viewpoint designated by the user from a plurality of candidates and an image corresponding to a viewpoint automatically designated by the device are also included in the virtual viewpoint image.

  In the present embodiment, an example in which audio data (audio data) is included in the virtual viewpoint content will be mainly described, but audio data may not necessarily be included. Further, the back-end server 1370 converts the virtual viewpoint image into, for example, H.264. After compression-encoding according to an encoding method such as H.264 or HEVC, it may be transmitted to the end user terminal 1290 using the MPEG-DASH protocol. Further, the virtual viewpoint image may be transmitted to the end user terminal 1290 without compression. In particular, the former performing compression encoding assumes a smartphone or a tablet as the end user terminal 1290, and the latter assumes a display capable of displaying an uncompressed image. That is, it is specified that the image format can be switched according to the type of the end user terminal 1290. Further, the image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or another transmission method may be used.

  As described above, the image processing system 1200 has three functional domains: a video collection domain, a data storage domain, and a video generation domain. The image collection domain includes sensor systems 1210-1210j. The data storage domain includes a database 1350, a front-end server 1330, and a back-end server 1370. The video generation domain includes a virtual camera operation UI 1430 and an end user terminal 1290. The present invention is not limited to this configuration. For example, the virtual camera operation UI 1430 can directly acquire images from the sensor systems 1210a to 1210j. However, in the present embodiment, a method of arranging a data storage function in the middle is used instead of a method of directly acquiring images from the sensor systems 1210a to 1210j. Specifically, the front-end server 1330 converts the image data and audio data generated by the sensor systems 1210a to 1210j and the meta information of the data into a common schema and data type of the database 1350. Thus, even if the camera 1212 of the sensor system 1210a to 1210j changes to a camera of another model, the changed difference can be absorbed by the front-end server 1330 and registered in the database 1350. This can reduce the risk that the virtual camera operation UI 1430 will not operate properly when the camera 1212 is changed to another camera.

  Further, the virtual camera operation UI 1430 is configured to access via the back-end server 1370 without directly accessing the database 1350. A common process related to the image generation process is performed by the back-end server 1370, and a difference portion of the application related to the operation UI is performed by the virtual camera operation UI 1430. As a result, in the development of the virtual camera operation UI 1430, it is possible to focus on the development for the UI operation device and the function request of the UI for operating the virtual viewpoint image to be generated. The back-end server 1370 can also add or delete common processing related to image generation processing in response to a request from the virtual camera operation UI 1430. This makes it possible to flexibly respond to the request of the virtual camera operation UI 1430.

  As described above, in the image processing system 1200, the virtual viewpoint image is generated by the back-end server 1370 based on the image data based on the photographing by the cameras 1212 for photographing the subject from a plurality of directions. Note that the image processing system 1200 according to the present embodiment is not limited to the physical configuration described above, and may be configured logically.

<Other embodiments>
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. It can also be realized by the following processing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Reference Signs List 100 generating device 501 image obtaining unit 504 area setting unit 507 determining unit 508 generating unit 509 specific pattern obtaining unit

Claims (20)

First obtaining means for obtaining a plurality of captured images obtained by capturing an image of a subject surface from a plurality of directions;
A second acquisition unit that acquires information indicating a position and a shape of a pattern on the subject surface;
Based on the plurality of captured images obtained by the first obtaining unit and the information indicating the position and shape of the pattern obtained by the second obtaining unit, a three-dimensional position for each of a plurality of regions on the subject surface Determining means for determining information;
Generating means for generating three-dimensional shape data corresponding to the object plane based on the three-dimensional position information for each area determined by the determining means. Setting means for setting a plurality of regions on the subject surface based on information indicating the position and shape of the pattern acquired by the second acquiring means,
The apparatus according to claim 1, wherein the determining unit determines three-dimensional position information for each of a plurality of regions on the subject plane set by the setting unit.   The generation apparatus according to claim 2, wherein the setting unit sets the plurality of regions such that each of the plurality of regions includes at least a part of the pattern. A projection unit configured to generate a projection image on a projection plane having a different distance from the subject plane for each of the regions and each distance from the subject plane based on the plurality of captured images acquired by the first acquisition unit. And
2. The apparatus according to claim 1, wherein the determination unit determines three-dimensional position information for each of the regions based on the projection images generated by the projection unit for each of the regions and for each distance from the object plane. The generating device according to any one of claims 3 to 3. A synthesizing unit for synthesizing a plurality of projection images having the same distance from the object plane for each area and each distance from the object plane to generate a synthesized image,
5. The three-dimensional position information for each of the regions, based on a composite image generated for each of the regions and for each distance from the subject plane, generated by the combining unit. The generating device according to claim 1.   The determining means outputs three-dimensional position information corresponding to a synthesized image in which the pattern becomes sharp among the synthesized images generated by the synthesizing means for each area and each distance from the object plane, The generation apparatus according to claim 5, wherein the three-dimensional position information is determined. A third acquisition unit configured to acquire a parameter of the imaging apparatus;
Determining three-dimensional position information of the plurality of regions based on information indicating a position and a shape of the pattern acquired by the second acquiring unit and the parameter acquired by the third acquiring unit; Further comprising calculating means for calculating the index of
The generating apparatus according to claim 5, wherein the synthesizing unit synthesizes the plurality of projection images having the same distance from the subject plane based on the index calculated by the calculating unit. .   The synthesizing unit, when synthesizing the plurality of projection images having the same distance from the subject plane, uses the index calculated by the calculation unit as a weight, and calculates a pixel value of a pixel corresponding to each projection image. The generating apparatus according to claim 7, wherein a weighted averaging process is performed on the generated data.   The determining means determines three-dimensional position information for each of a plurality of regions on the subject surface based on the region where the pattern is present in the projected image for each of the regions and for each distance from the subject surface. The generating device according to claim 4.   The determining unit may determine three-dimensional position information for each of a plurality of regions on the subject surface based on a product set of the existing regions of the pattern in the projection image for each of the regions and for each distance from the subject surface. The generating device according to claim 9, wherein:   The determining unit may determine three-dimensional position information for each of a plurality of regions on the subject surface based on a union of the existing regions of the pattern in the projection image for each of the regions and for each distance from the subject surface. The generating device according to claim 9, wherein:   The determining means determines three-dimensional position information for each of a plurality of regions on the subject surface based on continuity of a distance from the subject surface between two adjacent regions among the plurality of regions. The generating device according to any one of claims 4 to 11, wherein:   13. The method according to claim 12, wherein the continuity of the distance from the subject plane is represented by using an index indicating a change in a distance from the subject plane between the two adjacent areas of the plurality of areas. The generating device as described.   The generation device according to any one of claims 1 to 13, wherein the pattern is a line drawn in a field for performing a game.   The generation device according to claim 1, wherein the pattern has a color different from a color of another region on a subject plane. A first obtaining step of obtaining a plurality of captured images obtained by capturing an image of a subject surface from a plurality of directions;
A second acquisition step of acquiring information indicating a position and a shape of a pattern on the object plane;
Based on the plurality of captured images obtained in the first obtaining step and the information indicating the position and shape of the pattern obtained in the second obtaining step, a three-dimensional image is formed for each of the plurality of regions on the subject plane. A determination step of determining location information;
Generating a three-dimensional shape data corresponding to the subject plane based on the three-dimensional position information for each of the regions determined in the determining step. A setting step of setting a plurality of regions on the subject surface based on the information indicating the position and the shape of the pattern acquired in the second acquisition step;
17. The generation method according to claim 16, wherein the determining step determines three-dimensional position information for each of a plurality of regions on the subject plane set in the setting step.   18. The generation method according to claim 17, wherein the setting step sets the plurality of regions such that each of the plurality of regions includes at least a part of the pattern. A projection step of generating a projection image on a projection plane having a different distance from the object plane for each of the regions and each distance from the object plane based on the plurality of captured images acquired in the first acquisition step. And
17. The method according to claim 16, wherein the determining step determines three-dimensional position information for each of the regions based on the projection images generated for each of the regions and for each distance from the subject plane, which are generated in the projecting step. 19. The generation method according to any one of claims 18 to 18.   A program for causing a computer to function each unit of the generation device according to claim 1.