JP2008310724A - Three-dimensional shape restoration device, three-dimensional shape restoration method, three-dimensional shape restoration program and recording medium with its program stored - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that calculation costs to be spent on the generation of a dot group which is unnecessary for three-dimensional shape restoration is huge. <P>SOLUTION: An image acquired by picking up the image of a target object by a plurality of imaging devices installed according to multi-aspects is input as an input image (201). Then, a section corresponding to the target object is extracted from the input image, and the outline of the extracted section is calculated, and a polygon approximate outline is extracted by approximating the outline to a polygon (S2021). Then, texture featured dots as two-dimensional dots showing the characteristics of the texture of the target object are extracted from the input image (2022). Then, a three-dimensional polyhedron is restored based on the extracted polygon approximate outline, and featured dots as three-dimensional dots following the texture are added based on the extracted texture featured dots, and a polygon is generated (203). <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for restoring a three-dimensional shape of a subject captured in an image captured or captured by an imaging device such as a camera.

  Much research has been conducted on techniques for describing the geometric structure (three-dimensional shape) of a three-dimensional surface of an object in the three-dimensional measurement field and the CG (Computer Graphic) creation field. Polygon representation is often used to describe the geometric structure of the three-dimensional surface.

  Boligon represents the surface shape of the target object (hereinafter simply the target), but in addition to the surface of the polygon matching the target surface, the vertex and edge positions of the polygon are the feature points of the target surface texture. In other words, the calculation of the polygon along the texture of the target surface has various advantages.

  For example, if a polygon matches the characteristics of the surface texture, it is easy to analyze the color and reflection characteristics for each surface of the polygon, and to correct the color and reflection characteristics. It becomes easy.

  A conventional method for acquiring a polygon that follows the texture as described above from an object in the real world will be described based on an example of the processing flow of FIG. In this conventional method, step S101 for determining whether or not the input is performed by a camera (a kind of photographing apparatus; for example, a digital camera), point shape acquisition step S102 having steps S1021 to S1025, and polygon shape acquisition step S103 having step S1031. , Steps S1041 to S1045 have a polygon correction step S104.

  First, in step S101, it is determined whether or not the input is from a camera. If the input is from a camera, the process proceeds to step S1022. If it is not input by the camera, the process proceeds to step S1021.

  In the point cloud shape acquisition step S102, a known three-dimensional shape in the real world is acquired as a point cloud. There are several typical techniques for this acquisition. For example, a method using image data obtained from a camera (hereinafter simply referred to as an image) as an input (S1022 to S1025), and a target surface without using a camera. Or the method of measuring the three-dimensional position (S1021).

  As a method for measuring the three-dimensional position of the target surface without using the camera (S1021), for example, a method using a three-dimensional laser scanner, a laser range finder, or the like is known, and the target surface is irradiated with laser light. Outputs the position of the surface point of the object in the direction.

  In the method of using an image obtained from a camera as an input (S1022 to S1025), after performing input image processing (S1022), a multi-baseline stereo method (S1024), a visual volume crossing is performed by a user method selection (S1023) One of the methods (S1025) is selected and executed. In either method, an image obtained by capturing an object using a plurality of cameras is used as the input image acquisition process.

  In the multi-baseline stereo method (S1024), a distance image (depth map) is created for each of two cameras constituting a stereo camera pair from images obtained from a large number of cameras, and these are connected together. Compute the 3D point cloud of interest.

  The visual volume intersection method (S1025) is a method for restoring an object in a common visual field of cameras using images obtained from a plurality of cameras, and an example based on Non-Patent Document 1 will be described below.

  The shape restoration by the visual volume intersection method in Non-Patent Document 1 is performed by projecting and evaluating each voxel in the three-dimensional space as a restoration range on the image planes of all cameras, thereby determining whether or not there is an object in the voxel. The data to be obtained is a point cloud.

  Polygon vertices generated by this method are often unrelated to the target texture feature point (two-dimensional point representing the texture feature) or edge, and are shifted from the texture feature point or edge on the surface, A phenomenon occurs in which there are no vertices or edges on the surface texture feature, or on the contrary, a large number of points are generated in a portion having no texture feature. In other words, when the obtained polygon is viewed from the viewpoint of a polygon along the texture, there are excess or deficiency of points and displacement of positions.

  This visual volume intersection method uses a method of restoring the shape of an object with a uniform density of several millimeters (millimeters) to several centimeters (centimeters), and generates an enormous number of points.

  The point group-polygon conversion process (S1031) in the polygon shape acquisition step (S103) is a process of converting the point group obtained in step S102 into a polygon. In Non-Patent Document 1, the point group is converted by the discrete marching cube method. Convert to polygon.

  In general, in order to convert a point cloud into a polygon so that there is no mistake in surface tension, a dense point cloud is desirable. Therefore, it is difficult to reduce the number of points at this stage, and there is a problem that the calculation cost required for processing the enormous points obtained in the point cloud shape acquisition step (S102) is high. Further, since the obtained polygon is a dense mesh, the number of polygons is large and the amount of data is large. From the viewpoint of expressing a texture, there are problems that many unnecessary points are included.

  The input image acquisition process (S1041) in the polygon correction step (S104) is the same process as that in step S1022, and will not be described.

  In the case of Non-Patent Document 1, the surface feature extraction process (S1042) is a process for acquiring the color of the pixel in the acquired image.

  The polygon vertex evaluation process (S1043) sets an energy field that moves with respect to each vertex of the polygon, and is designed based on the constraints based on the color of the pixels, constraints based on silhouettes, constraints that prevent polygon intersection, and the like. Is evaluated and the correction value of the vertex is calculated.

  Also in the polygon vertex evaluation process (S1043), it is necessary to process the large number of polygons, so the calculation amount is enormous, and the vertex evaluation algorithm based on a plurality of constraint conditions is complicated. There is a problem that vertices that deviate from the expected range do not produce stable results.

  The polygon correction process (S1044) is a process of changing the coordinates of the polygon vertices according to the correction value of the vertices, and outputs the polygon with the corrected vertex coordinates.

  Further, a PVH (Polyhederal Visual Hulls) method known as one of the visual volume intersection methods will be described here as a related technique.

  A PVH method, which is one method for realizing the principle of the visual volume intersection method, will be described with reference to FIG. In the PVH method in the following description, a polygon approximated contour (that is, a polygon approximated contour) extracted from a multi-viewpoint image is input.

In Figure 14A, the sign C object, the code P _A projection center of the camera, the code SP imaging surface of the camera, reference numeral D _A is a two-dimensional silhouette obtained on the imaging surface of the camera.

In FIG. 14B, symbol V _A is called a visual volume, and is a cone that extends in the three-dimensional space so as to pass each point on the contour of the silhouette D _A from the projection center P _{A of the} camera. the in 14B, (configured) to have occurred by using a projection center P _a silhouette D _a camera. Reference numerals F ₁ and F ₂ are called faces constituting the visual volume, and each is a face having a visual cone.

In FIG. 14C, the visual volume V _A and the visual volume V _B are Visual Cone obtained from two cameras. Reference symbol VH is a visual hull, and is a three-dimensional shape of a target obtained by obtaining a plurality of visual cones, here, a common region of the visual volume V _A and the visual volume V _B.

  In the PVH method, the principle of calculation for obtaining one surface of the Visual Hull in the visual volume intersection method shown in FIG. 14 will be described based on FIG.

Symbols P ₁ and P _{2 in} FIG. 15 are the projection centers of the plurality of cameras. Reference numerals SP ₁ and SP ₂ are imaging surfaces of the respective cameras. A symbol C indicates a target object. Code D _1, D ₂ is the silhouette of the object C obtained on the imaging plane SP _1, SP ₂ cameras. Reference numeral F is a face formed by using one side having the projection center P ₂ and the silhouette D ₂ of the camera. Reference numeral PF denotes an area obtained by projecting Face F onto the imaging surface SP _{1 of the} camera. Reference numeral OA is an overlapping area of the area PF and the silhouette D ₁ obtained on the imaging surface SP ₁ of the camera. A symbol IA is a three-dimensional surface obtained by back projecting the overlapping area OA onto Face F, and is a part of Face F.

  When there are three or more cameras, the three-dimensional plane IA can be obtained by back projecting the overlapping area obtained on the imaging surfaces of a plurality of cameras onto Face F and obtaining the intersection area (common area). That is, the three-dimensional surface IA indicates one surface having the Visual Hull.

  An example of the processing flow showing the PVH method until the entire Visual Hall of interest is calculated using a plurality of cameras will be described with reference to FIG.

  First, it is a process of generating Visual Cone for all the cameras, and Visual Cone is generated for each camera (S501 to S503).

  Step S501 is processing to repeat for all cameras.

  Step S502 is processing for calculating Visual Cone, and generates Visual Cone by extending a surface from the projection center of the camera so as to pass through each side of the target polygon approximate contour on the imaging surface.

  Step S503 is processing for determining that the repetition has been completed for all the cameras.

  Then, the surface constituting the Visual Cone is cut out and the Visual Hull is calculated on the Visual Cone (S504 to S513). For all Visual Cone, the Visual Hull of the entire object is calculated by repeating steps S505 to S512. Each is calculated in the following steps.

  Step S505 is a process for repeating all the Faces of the Visual Cone to be processed.

  Steps S <b> 506 to S <b> 510 are processes for projecting a face onto an imaging surface of another camera, obtaining a silhouette (polygon approximate contour and a two-dimensional surface constituting the interior) and a projected face overlapping region, and backprojecting to face. It is.

  Step S506 is a process of repeating for cameras other than the camera to which the Visual Cone to be processed belongs.

  Step S507 is processing for projecting the face onto the camera screen, and the face projected onto the screen is calculated by projection calculation.

  Step S508 is a process for calculating the projected face and silhouette overlap area, and calculates the overlap area of the two faces of the face and silhouette projected using the inside / outside determination of the geometric polygon.

  Step S509 is a process of back projecting the superposed area to the face, and the superposed area obtained in step S508 is back projected onto the face by projection calculation to obtain a back projected superposed area.

  Step S510 is processing for determining that the repetition has been completed for all the cameras in the repetition processing from step S506.

  Step S511 is a process for obtaining intersection regions of all the back-projected superimposed regions on the face, and calculates intersection regions using geometric polygon inside / outside determination.

  Step S512 is processing for determining that the repetition has been completed for all the faces in the repetition processing from step S505.

  Step S513 is processing for determining that the repetition has been completed for all Visual Cones in the repetition processing from step S504.

The polyhedron shape constituting the target Visual Hull is calculated by the above-described processing showing an example of the PVH method, and a three-dimensional point representing the shape is obtained as each vertex of the polyhedron.
Nobuhara Shohei, Wada Toshikazu, Matsuyama Takashi, “Highly accurate 3D shape restoration from multi-viewpoint images using elastic mesh models”, Information Processing Society of Japan, Journal of Information Processing Society of Japan, Information Processing Society of Japan, December 2002 , Vol. 43, SIG11 (CVIM5), pp. 53-63. Matusik, W.M. , C.I. Buehler, and L.B. McMillan, “Polyhederal Halls for Real-Time Rendering”, Processeds of Terrors Europe, Workshop on Rendering, June 2001 (pp. 2001). 115225.

In the conventional method as shown in FIG. 13 described above, the calculation cost for generating unnecessary point clouds (for example, step S1031 in FIG. 13).
It is therefore necessary to avoid generating unnecessary points in order to reduce their computational costs.

  Furthermore, the complexity of the algorithm for correcting polygons and the uncertainty of the results are major problems, and an algorithm that can obtain a stable result is required.

  The present invention has been made based on the above-described problem, and calculates a three-dimensional point representing the three-dimensional polyhedron shape of the target object and a three-dimensional point along the texture, and uses them to calculate a polygon along the texture. An object of the present invention is to provide a three-dimensional shape restoration apparatus, a three-dimensional shape restoration method, a three-dimensional shape restoration program, and a recording medium storing the program.

  In order to solve the above problems, the present invention uses a feature point addition type PVH method in addition to the above-described PVH method.

  Hereinafter, the detailed principle of the PVH method and the feature point addition type PVH method will be described.

  A process for obtaining one surface of the Visual Hull in the feature point addition type PVH method proposed in the present invention and a feature point of the target surface projected on the surface will be described with reference to FIG.

Reference numerals P ₁ and P _{2 in} FIG. 17 are projection centers of a plurality of cameras (a type of imaging device or imaging device). Reference numerals SP ₁ and SP ₂ are imaging surfaces of the respective cameras. A symbol C indicates a target object. Code TI ₁ is a foreground object C obtained on the imaging surface SP ₁ of the camera, in which an extracted object C by background subtraction. Code D ₂ is the silhouette of the object C obtained on the imaging surface SP ₂ cameras. Reference numeral F denotes a face configured by using one side of the camera projection centers P ₂ and D ₂ .

Reference numeral PF denotes an area obtained by projecting Face F onto the imaging surface SP _{1 of the} camera. Code OA is the overlapping region of the region PF and D ₁ obtained on the imaging surface SP1 of the camera. Symbol TF is a texture feature point extracted from the foreground TI ₁ . Reference numeral IA is a three-dimensional surface obtained by back projecting the overlapping area OA onto Face F, and is a part of Face F.

  When there are three or more cameras, the three-dimensional plane IA can be obtained by back projecting the overlapping area obtained on the imaging surfaces of a plurality of cameras onto Face F and obtaining the intersection area (common area). That is, the three-dimensional surface IA indicates one surface having the Visual Hull.

  The symbol TV is a texture feature point that is back-projected on the three-dimensional surface IA, and is obtained by back-projecting the texture feature point TF to Face F and selecting the one inside the three-dimensional surface IA.

  In order to solve the above problem based on the above-described principle, the invention according to claim 1 acquires an image captured by a plurality of imaging devices having a target object installed at multiple viewpoints from an input image acquisition unit, and An input image acquisition processing unit that inputs an acquired image as an input image, and a three-dimensional shape restoration device that restores the three-dimensional shape of the target object, the target being obtained from the input image input by the input image acquisition processing unit Extracting a part corresponding to the object, calculating a contour of the extracted part, extracting a polygonal approximate contour obtained by approximating the contour to a polygon, and features of the texture of the target object from the input image A surface feature extraction unit that extracts texture feature points that are two-dimensional points representing the three-dimensional polyhedron using the polygonal approximate contour extracted by the contour information extraction unit, and the surface feature extraction A feature point that is a three-dimensional point along the texture is added on the basis of the texture feature point extracted in step 3, and a polygon is added based on the restored three-dimensional polyhedron and the feature point that is a three-dimensional point along the added texture. And a polygon generation unit for generating.

  According to a second aspect of the present invention, in the first aspect of the invention, the polygon generation unit indicates the shape of the target object based on the PVH method using the polygonal approximate contour extracted by the contour information extraction unit. Reconstructing projection means for projecting the texture feature points extracted by the surface feature extraction unit onto the surface of the restored three-dimensional polyhedron while restoring the shape of a three-dimensional polyhedron composed of three-dimensional points, and the restoring projection means A feature point that is a three-dimensional point projected along the texture is added, and a vertex of the restored three-dimensional polyhedron is connected to a feature point that is the projected three-dimensional point along the texture to connect the three-dimensional polyhedron And a polygon generating means that follows the texture to generate polygons.

  According to a third aspect of the present invention, in the second aspect of the present invention, in the process in which the restoration projection means projects the visual volume in the PVH method, out of the texture feature points extracted by the surface feature extraction unit. Means for calculating a texture feature point existing in a plane formed by projecting a volume onto an image, and calculating a feature point that is a three-dimensional point along the texture by back projecting the calculated texture feature point; It is characterized by providing.

  According to a fourth aspect of the present invention, in the first aspect of the invention, the polygon generation unit indicates the shape of the target object based on the PVH method using the polygonal approximate contour extracted by the contour information extraction unit. PVH processing means for restoring the shape of a three-dimensional polyhedron composed of three-dimensional points, and texture feature points extracted by the surface feature extraction unit are projected onto the restored three-dimensional polyhedron surface, and the projected textures are projected. A feature point addition processing means for adding a feature point that is a three-dimensional point along the line, a vertex of the restored three-dimensional polyhedron, and a feature point that is a projected three-dimensional point along the texture, Means for dividing a polyhedron to generate a polygon.

  According to a fifth aspect of the present invention, in the invention according to the fourth aspect, the feature point addition processing means is extracted by the surface feature extraction unit on a surface constituting the intersection of the visual volumes calculated by the PVH method. And a means for calculating an intersection with the intersection of the viewing volumes by projecting the texture feature point, and regarding the intersection as a feature point that is a three-dimensional point along the texture.

  According to a sixth aspect of the present invention, there is provided an input image acquisition processing unit that acquires an image obtained by capturing a target object with a plurality of photographing devices installed at multiple viewpoints from an input image acquisition unit and inputs the acquired image as an input image. A three-dimensional shape restoration method used in an apparatus for restoring the three-dimensional shape of the target object, wherein a portion corresponding to the target object is extracted from the input image input by the input image acquisition processing unit, and the extraction is performed A contour information extraction step of calculating a contour of the portion that has been approximated and extracting a polygonal approximate contour obtained by approximating the contour to a polygon; and a texture feature point that is a two-dimensional point representing the texture feature of the target object from the input image. A three-dimensional polyhedron is restored using the surface feature extraction step to be extracted and the polygon approximate contour extracted in the contour information extraction step, and is extracted in the surface feature extraction step. A polygon that generates a polygon based on the restored 3D polyhedron and a feature point that is a 3D point along the added texture is added based on the texture feature point. And a generating step.

  According to a seventh aspect of the invention, in the sixth aspect of the invention, the polygon generating step indicates the shape of the target object based on the PVH method using the polygonal approximate contour extracted in the contour information extracting step. A restoration projection step of projecting the texture feature points extracted in the surface feature extraction step onto the plane of the restored three-dimensional polyhedron while restoring the shape of a three-dimensional polyhedron composed of three-dimensional points, and the restoration projection step A feature point that is a three-dimensional point projected along the texture is added, and a vertex of the restored three-dimensional polyhedron is connected to a feature point that is the projected three-dimensional point along the texture to connect the three-dimensional polyhedron And a polygon generation step along the texture for generating polygons by dividing.

  According to an eighth aspect of the present invention, in the seventh aspect of the present invention, among the texture feature points extracted in the surface feature extraction step, the restoration projection step projects the visual volume in the PVH method. Calculating a texture feature point present in a plane constructed by projecting a volume onto an image, back-projecting the calculated texture feature point, and calculating a feature point that is a three-dimensional point along the texture; It is characterized by having.

  The invention according to claim 9 is the invention according to claim 6, wherein the polygon generation step indicates the shape of the target object based on the PVH method using the polygonal approximate contour extracted in the contour information extraction step. A PVH processing step for restoring the shape of a three-dimensional polyhedron composed of three-dimensional points, and a texture feature point extracted in the surface feature extraction step is projected onto the surface of the restored three-dimensional polyhedron, and the projected texture A feature point addition processing step of adding a feature point that is a three-dimensional point along the line, a vertex of the restored three-dimensional polyhedron, and a feature point that is a projected three-dimensional point along the texture, Dividing the polyhedron to generate polygons.

  According to a tenth aspect of the present invention, in the ninth aspect of the invention, the feature point addition processing step is extracted in the surface feature extraction step on a plane that constitutes a view volume intersection calculated by the PVH method. Projecting the texture feature point, calculating an intersection with the intersection of the viewing volumes, and considering the intersection as a feature point that is a three-dimensional point along the texture.

  An eleventh aspect of the invention is a three-dimensional shape restoration program, which causes a computer to function as each unit and means in the three-dimensional shape restoration device according to any one of the first to fifth aspects.

  The invention described in claim 12 is a recording medium, wherein the three-dimensional shape restoration program described in claim 11 is recorded.

  Since the invention according to claim 1 or 6 employs a three-dimensional polyhedron, the shape of the target object can be calculated with a small number of points. Polygons along texture feature points can be generated.

  In the inventions according to the second, fourth, seventh, and ninth aspects, since the three-dimensional points representing the shape of the three-dimensional polyhedron are each vertex of the three-dimensional polyhedron, the shape of the object can be calculated with a small number of points. A polygon along the feature point of the three-dimensional point obtained by projecting the texture feature point onto the surface of the three-dimensional polyhedron can be generated.

  The invention according to the third, fifth, eighth, and tenth aspects can further limit feature points that are three-dimensional points along the texture by back projection. A process for searching on which surface the feature point, which is a three-dimensional point along the texture, is projected later is unnecessary.

  The invention according to claim 11 can describe the three-dimensional shape restoration apparatus according to any one of claims 1 to 5 as a computer program.

  The invention described in claim 12 can record the three-dimensional shape restoration program described in claim 11 on a recording medium.

  As described above, according to the invention of claim 1 or 6, the cost for calculating the shape of the target object can be reduced. A stable polygon can be obtained.

  According to the inventions of claims 2, 4, 7, and 9, the cost for calculating the shape of the target object can be reduced. A stable polygon can be obtained.

  According to the third, fifth, eighth and tenth aspects of the present invention, the cost for calculating the polygon can be reduced.

  According to invention of Claim 11, the computer program which operates a computer as a three-dimensional shape restoration apparatus can be provided.

  According to invention of Claim 12, the recording medium which recorded the computer program which operates a computer as a three-dimensional shape restoration apparatus can be provided.

  These can contribute to the computer vision field.

  The configuration of the three-dimensional shape restoration apparatus in the embodiment of the present invention will be described with reference to FIG.

  The three-dimensional shape restoration apparatus in FIG. 1 includes an input image acquisition processing unit 201 that acquires an image captured from multiple viewpoints (multi-viewpoint imaging), and a target object (hereinafter simply referred to as an object) from the images captured from multiple viewpoints. An outline information extraction unit 2021 that extracts an outline, a surface feature extraction unit 2022 that extracts a texture feature point (feature on a texture) of a target object from a multi-viewpointed image, and an outline and a surface extracted by an outline information extraction unit 2021 A polygon generation unit 203 that generates polygons based on the texture features extracted by the feature extraction unit 2022 is provided.

  The input image acquisition processing unit 201 acquires images taken from multiple viewpoints (taken by installing a plurality of cameras (imaging devices or imaging devices) so as to have multiple viewpoints) from an input image acquisition unit (not shown). And input as an input image. The input image acquisition unit is a functional unit that reads the multi-viewpoint image from a storage device such as a camera or a hard disk device.

  It is assumed that the multi-viewpoint-captured image is taken by a camera arranged so as to have a shared field of view in all cameras. Further, it is desirable that the image is acquired by synchronous shooting.

  The contour information extraction unit 2021 extracts the contour of the target object appearing in the image from the multi-viewpoint image by the contour extraction process.

  In the contour extraction process, the target is extracted from the image by background difference, and the extracted target contour is calculated by boundary line tracking. The contour is a closed curve, and in the present embodiment, the closed curve is approximated to a polygon with an arbitrary accuracy.

  Since the accuracy of the contour is related to the number of contours approximated by a polygon, if the number of strokes is small, the calculation cost can be reduced, but the result becomes rough. On the other hand, when the number of strokes is too large, for example, when the number of strokes of a polygon approximated contour is close to the number of contours before approximation, the result is fine but the calculation cost increases. In the present embodiment, the accuracy is determined in advance by the user according to the fineness of the result to be obtained.

  The surface feature extraction unit 2022 extracts target texture feature points (features on the texture) from the multi-viewpoint captured image by surface feature extraction processing.

  The surface feature extraction processing is processing for performing corner detection on an image and calculating feature points. In addition to the corner detection, a method widely used in image processing such as a general feature point extraction method or an edge extraction method may be used. In the present embodiment, the feature is a point, but may be a side.

  The polygon generation unit 203 uses the polygon approximate contour extracted by the contour information extraction unit 2021 and the texture feature point extracted by the surface feature extraction unit 2022 by the polygon generation processing, and uses the three-dimensional surface of the target. Calculate the shape as a polygon.

  The polygon generation process uses the polygonal approximate contour generated by the contour information extraction unit 2021 and the principle of the above-mentioned PVH (Polyhedral Visual Hulls) method (see Non-Patent Document 2), which is one of the visual volume intersection methods. Based on the above, the target 3D polyhedron (Visual Hull) shape (a 3D polyhedron shape consisting of 3D points indicating the shape of the target object) is restored, and texture feature points are projected onto the restored 3D polyhedron surface. Then, the vertex of the three-dimensional polyhedron and the projected feature point (three-dimensional feature point) are connected (connected), and the three-dimensional polyhedron is divided to generate a polygon.

  Note that a division method such as Delaunay division may be used to divide the surface, and an arbitrary format can be used as an expression format of the polygon data. In the present embodiment, the polygon is represented by a triangular patch.

  In addition, the three-dimensional shape restoration apparatus may include a control unit (for example, a CPU (Central Processing Unit) or an OS (Operating System)) that controls each functional unit.

  The three-dimensional shape restoration device may include a storage unit (for example, a memory or a hard disk device) that stores information or data used in each functional unit.

  The three-dimensional shape restoration apparatus may be realized by a general computer (for example, a personal computer).

  Furthermore, in this embodiment, the feature point addition type PVH method is adopted in the polygon generation processing. Based on this feature point addition type PVH method, the entire Visual Hull of the object is calculated using a plurality of cameras (steps S701 to S715), and the feature points existing on the Visual Hull are calculated (steps S707 to S713). An example of the process flow will be described with reference to FIG.

  Steps S701 to S708 are the same as steps S501 to S508 in FIG.

  Step S709 is a process of calculating feature points in the superimposition area. Among the feature points (texture feature points) obtained in the surface feature extraction process (process in the surface feature extraction unit 2022), those in the superimposition area are calculated. The process to select.

  Step S710 is a process of back-projecting the superimposition region and the selected feature point onto Face, and the superimposition region obtained at Step S708 and the feature point inside the superposition region obtained at Step S709 are projected to Face. This is a process for back projecting and obtaining the back projected superposed region and the back projected feature points (three-dimensional feature points).

  Step S711 is the same processing as step S510 in FIG.

  Step S712 is the same processing as step S511 in FIG.

  Step S713 is a process of calculating the feature points inside the intersection area obtained in step S712, and is a process of selecting the back-projected feature points obtained in step S710 that are inside the intersection area. .

  Step S714 is the same processing as step S512 in FIG.

  Step S715 is the same process as step S513 in FIG.

  With the example of the processing flow of the above-described feature point addition type PVH method, a three-dimensional point representing a shape and a three-dimensional point (feature point) along the texture are obtained.

  The feature point addition type PVH method calculates and projects the feature points on the same plane in the process of calculating the shape of the target polyhedron. Therefore, in the process of projecting the feature points on the surface of the three-dimensional polyhedron, Therefore, it is not necessary to search for which surface is projected onto the surface, and feature points can be arranged on the surface of the three-dimensional polyhedron with low calculation cost.

  A three-dimensional shape restoration method in the three-dimensional shape restoration device in the present embodiment will be described. Note that each step in the following description is executed by a functional unit having the same name as the step name.

  First, an image taken from multiple viewpoints is acquired from an input image acquisition unit (not shown) and input as an input image. (Input image acquisition processing step).

  Next, the contour of the target object shown in the image is extracted from the multi-viewpoint image (contour information extraction step).

  Next, target texture feature points (texture features) are extracted from the multi-viewpoint image (surface feature extraction step).

  Then, using the polygon approximate contour extracted by the contour information extraction step and the texture feature points extracted by the surface feature extraction step, the target three-dimensional surface shape is calculated as a polygon (polygon generation step). .

  Note that the contour information extraction step and the surface feature extraction step may be processed in parallel.

  As described above, the present embodiment is unnecessary by calculating a 3D point representing the target 3D polyhedron shape and a 3D point along the texture, and calculating a polygon along the texture using them. The generation of a large number of such points is avoided, and the calculation cost is greatly reduced.

  Further, in step S1043 in FIG. 13, the calculation cost is enormous because many vertices are evaluated. Regarding evaluation of a large number of vertices, it is desirable to avoid evaluation of unnecessary points and to evaluate only points suitable for evaluation.

  On the other hand, the three-dimensional point representing the shape is based on the vertex of the target polygon approximate contour calculated from the image, and the three-dimensional point along the texture is based on the target texture feature point calculated from the image. Because polygon vertices can be generated at positions along the texture without evaluating polygon vertices or correcting polygons, the algorithm is simple and stable results can be obtained without complicated evaluation. it can.

  Next, an example in which the feature point addition type PVH method in the present embodiment is adopted will be described below.

[Example 1]
The configuration of the three-dimensional shape restoration apparatus in Embodiment 1 will be described with reference to FIG.

  The three-dimensional shape restoration apparatus in FIG. 3 includes an input image acquisition unit 801, a contour information extraction unit 8021, a surface feature extraction unit 8022, and a polygon generation unit 803.

  Note that the input image acquisition unit 801, contour information extraction unit 8021, and surface feature extraction unit 8022 are the same functional units as the input image acquisition unit 201, contour information extraction unit 2021, and surface feature extraction unit 2022 in FIG. The description is omitted.

  The polygon generation unit 803 corresponds to the function of the polygon generation unit 203, and includes a PVH processing unit 8031, a feature point addition processing unit 8032, and a polygon patch generation processing unit 8033.

  The PVH processing unit 8031 calculates Visual Hull by processing based on the PVH method (hereinafter simply referred to as PVH processing). That is, the PVH processing unit 8031 receives the polygonal approximate contour of the target object (hereinafter simply referred to as the target) calculated by the contour information extraction unit 8021 as input, and calculates the target visual hull by PVH processing. Get the 3D point to represent.

  The PVH process is a process of calculating the target Visual Hull according to the principle in FIG. 15 and the PVH method in FIG.

  The feature point addition processing unit 8032 adds feature points on the surface constituting the Visual Hull by feature point addition processing. Then, the feature point addition processing unit 8032 obtains a three-dimensional point along the texture by adding the feature point on the Visual Hull.

  In the feature point addition processing, the texture feature point obtained by the surface feature extraction unit 8022 is projected onto a three-dimensional space, and the intersection point with the Visual Hull is calculated, thereby calculating the feature point projected on the surface constituting the Visual Hull. Is a process for obtaining.

  The polygon patch generation processing unit 8033 receives the Visual Hull to which the feature points have been added, that is, a three-dimensional point representing a three-dimensional polyhedron shape and a three-dimensional point along the texture, as an input, by a polygon patch generation process. Is generated.

  The polygon patch generation process is a process of connecting the vertices constituting the surface of the Visual Hull and the feature points on the same surface of the Visual Hull, and generates a polygon by a vertex connection method such as Delaunay division.

  Table T2 in FIG. 4 is a result of an experiment in which the number of vertices calculated by Example 1 is compared with the number of vertices calculated by the method based on Non-Patent Document 1 in the experimental environment shown in Table T1. When generating polygons with the same resolution from images taken under the same conditions, as shown in Table T2, it can be created with half the number of vertices compared to the method based on Non-Patent Document 1. And the processing time was less than half.

  As described above, the Visual Hull calculated by the PVH processing of the PVH processing unit 8031 in the first embodiment is a polyhedron representation, and the three-dimensional points representing the shape are the vertices of the polyhedron, so that the Visual Hull of the conventional method is used. The shape of the object can be calculated with a smaller number of points than in the case of expressing a point cloud. Thereby, calculation cost can be reduced as compared with the conventional method in which a large number of points are generated.

  In addition, the calculation of the three-dimensional points along the texture simply requires adding the texture feature points necessary on the Visual Hull surface calculated by the polyhedron representation in the feature point addition processing unit 8032 to the projection calculation, thus simplifying the algorithm. it can. Further, since the polygon vertices along the texture can be obtained without evaluating the polygon vertices or correcting the polygons, a stable result can be obtained.

  The polygons along the texture are calculated by connecting the polyhedron vertices constituting the Visual Hull, which is a three-dimensional point representing the shape, and the three-dimensional points along the texture added on the surface of the Visual Hull. This eliminates the need for a calculation cost.

[Example 2]
The configuration of the three-dimensional shape restoration apparatus in the second embodiment will be described with reference to FIG.

  The three-dimensional shape restoration apparatus in FIG. 5 includes an input image acquisition unit 901, a contour information extraction unit 9021, a surface feature extraction unit 9022, and a polygon generation unit 903.

  The input image acquisition unit 901, contour information extraction unit 9021, and surface feature extraction unit 9022 are the same functional units as the input image acquisition unit 201, contour information extraction unit 2021, and surface feature extraction unit 2022 in FIG. Is omitted.

  The polygon generation unit 903 corresponds to the function of the polygon generation unit 203 in FIG. 1, and includes a feature point addition type PVH processing unit 9031 and a polygon patch generation unit 9032.

  The feature point addition type PVH processing unit 9031 performs the feature point addition type PVH processing on the target polygon approximate contour calculated by the contour information extraction unit 9021 and the image of each camera calculated by the surface feature extraction unit 9022. Using the obtained texture feature point, the Visual Hull and the Visual Hull projected on the Visual Hull, that is, the three-dimensional point representing the shape and the three-dimensional point along the texture are calculated.

  The feature point addition type PVH processing is performed by calculating the feature points projected on the target Visual Hull and Visual Hull according to the principle in FIG. 17 and the method of calculating the feature points existing on the entire Visual Hull and Visual Hull in FIG. It is a process to calculate.

  The polygon patch generation unit 9032 is the same functional unit as the polygon patch generation unit 8033, and thus description thereof is omitted.

  A result obtained by implementing Example 1 and Example 2 as a program and using a camera and a computer (computer) will be described below.

  Images p11 to p14 in FIG. 6 are examples of input images, and are images obtained by photographing a target using a multi-viewpoint camera. That is, the input images p11 to p14 are examples of images obtained from four different viewpoints.

  Images ps11 to ps14 in FIG. 7 are examples of silhouette images obtained by the conventional silhouette extraction method. The images are extracted from the input images p11 to p14 in FIG. is there.

  FIG. 8 is an example of an image subjected to a conventional polygon approximation and contour extraction method. The contours of the silhouette images ps11 to ps14 in FIG. 7 are extracted, polygons are approximated, and a polygon is approximated to the silhouette. The displayed contour is superimposed. Images e11 to e13 in FIG. 8A show polygonal approximate contours corresponding to the three silhouette images ps11, ps12, and ps14. FIG. 8B shows polygon approximate contours e22 and e23 obtained with different approximation accuracy for the part e21a in the silhouette image ps14.

  A white point in the image pt11 in FIG. 9 is an example of a texture feature point by a conventional method for extracting a feature on the texture, and a feature point extraction process is performed on the input image p12. An image pt11 in FIG. 9 shows feature points obtained by performing corner detection using a Harris operator. In addition, the image pt11 superimposes the feature points calculated on the input image p12.

  An image vh11 in FIG. 10 is an example of the Visual Hull calculated by the conventional PVH method, and is calculated using the polygonal approximate contours of the objects obtained from seven cameras.

  A black point in the image vt11 in FIG. 11 is an example in which the texture feature points in the present embodiment and the example are projected on the Visual Hull. On the Visual Hull in the image vh11 in FIG. 10, the black point of the image pt11 in FIG. This is a projection of texture feature points as indicated by white dots. Equivalent results are obtained using either Example 1 or Example 2.

  Images pg11 and pg12 in FIG. 12 are examples of polygons obtained by the processing in the second embodiment. Polygon vertices are projected onto an image pt11 in FIG. It is shown that the polygon vertex is located on the feature point (white point in the images pg11 and pg12 in FIG. 12) and is along the texture.

  As described above, according to the second embodiment, the three-dimensional point expressing the shape and the three-dimensional point along the texture can be calculated simultaneously by the feature point addition type PVH. In the first embodiment, after obtaining the Visual Hull, it is necessary to perform a process of back projecting the feature points and determining whether or not each surface of the Visual Hull intersects. In the second embodiment, a process corresponding to this is as follows. Since the feature hull PVH is performed during the calculation of the Visual Hull, the number of projection calculations is small, and the calculation cost is lower than that of the first embodiment.

  Note that the present invention is realized by configuring a part or all of the functions of each unit and each means in the three-dimensional shape restoration apparatus of the present embodiment and examples by a computer program and executing the program using the computer. Of course, the method (procedure) relating to the three-dimensional shape restoration apparatus of the present embodiment and examples can be configured by a computer program, and the program can be executed by the computer. , A computer-readable recording medium (storage medium) such as FD (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact) Dis k), a DVD (Digital Versatile Disk), a removable disk, etc., and can be stored or distributed. It is also possible to provide the above program through a network such as the Internet or electronic mail.

  Further, a computer program describing a method related to the above-described three-dimensional shape restoration apparatus of the present embodiment and examples is implemented so as to access a memory or an external storage unit storing input / output data required for the method. May be.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the described embodiments, and various modifications can be made within the scope described in each claim.

  For example, the target contour extracted by the contour information extraction unit and the texture feature points extracted by the surface feature extraction unit are temporarily stored in a storage unit provided in advance, and the polygon is generated by accessing the storage unit. You can go.

The block diagram of the dimension shape restoration apparatus in this embodiment. The figure which shows an example of the processing flow which calculates the whole Visual Hull of object based on the feature point addition type PVH method in this embodiment, and calculates the feature point on Visual Hull. 1 is a configuration diagram of a three-dimensional shape restoration apparatus in Embodiment 1. FIG. The figure which shows the experimental environment and experimental result of Example 1. FIG. The block diagram of the three-dimensional shape decompression | restoration apparatus in Example 2. FIG. The figure which shows an example of an input image. The figure which shows an example of a silhouette image. The figure which shows an example of a polygon approximate outline. The figure which shows an example of a texture feature point. The figure which shows an example of Visual Hull calculated by the conventional PVH method. The figure which shows an example which projected the texture feature point on Visual Hull. FIG. 10 is a diagram illustrating an example of a polygon obtained by processing in the second embodiment. The figure which shows an example of the processing flow which shows the conventional method which acquires the polygon which follows a texture. The figure which shows the principle of PVH method. The principle figure of the calculation which calculates | requires one surface with Visual Hull. The figure which shows an example of the processing flow which calculates the whole Visual Hull. The figure which shows the principle of the feature point addition type PVH method.

Explanation of symbols

201, 801, 901 ... input image acquisition processing unit 2021, 8021, 9021 ... contour information extraction unit 2022, 8022, 9022 ... surface feature extraction unit 203, 803, 903 ... polygon generation unit 8031 ... PVH processing unit 8032 ... feature point addition Processing unit 8033 ... Polygon patch generation processing unit 9031 ... Feature point addition type PVH processing unit 9032 ... Polygon patch generation unit C ... Target object D _A ... Two-dimensional silhouette D ₁ , D ₂ obtained on the imaging surface of the camera ... Silhouette F ... Face
F _1, F ₂ ... viewing surface IA ... 3 dimensional plane OA ... overlap region constituting a volume _{P A, P B, P 1} , P 2 ... camera area the projection center PF ... Face projected onto the imaging plane of the camera SP, SP ₁ , SP ₂ ... imaging surface of camera T1 ... table showing experimental environment T2 ... table showing experimental results TI ₁ ... foreground TF, TV ... texture feature points V _A , V _B ... visual volume VH ... visual volume Intersection e11 to e13, e22, e23 ... polygon approximate contour e21a ... part in silhouette image p11 to p14 ... input image pg11, pg12 ... image showing examples of polygons ps11 to ps14 ... silhouette image pt11 ... showing examples of texture feature points Image vh11 ... Image showing an example of Visual Hull vt11 ... Image showing an example of texture feature points projected on Visual Hull

Claims (12)

An input image acquisition processing unit configured to acquire an image captured by a plurality of imaging devices installed at multiple viewpoints from an input image acquisition unit and input the acquired image as an input image;
A three-dimensional shape restoration device for restoring a three-dimensional shape of the target object,
Contour information for extracting a portion corresponding to the target object from the input image input by the input image acquisition processing unit, calculating a contour of the extracted portion, and extracting a polygon approximate contour obtained by approximating the contour to a polygon An extractor;
A surface feature extraction unit that extracts a texture feature point that is a two-dimensional point representing the texture feature of the target object from the input image;
A three-dimensional polyhedron is restored using the polygonal approximate contour extracted by the contour information extraction unit, and a feature point that is a three-dimensional point along the texture is added based on the texture feature point extracted by the surface feature extraction unit A polygon generation unit that generates a polygon based on the restored three-dimensional polyhedron and a feature point that is a three-dimensional point along the added texture;
A three-dimensional shape restoration apparatus comprising: The three-dimensional shape restoration apparatus according to claim 1,
The polygon generation unit
Using the polygonal approximate contour extracted by the contour information extraction unit, based on the PVH method, while restoring the shape of a three-dimensional polyhedron consisting of three-dimensional points indicating the shape of the target object,
Reconstructing projection means for projecting the texture feature points extracted by the surface feature extracting unit onto the surface of the reconstructed three-dimensional polyhedron;
Adding feature points that are three-dimensional points along the texture projected by the reconstruction projection means;
A polygon generating unit along the texture that connects the vertex of the restored three-dimensional polyhedron and the feature point that is the projected three-dimensional point along the texture, and generates a polygon by dividing the three-dimensional polyhedron;
A three-dimensional shape restoration apparatus comprising: The three-dimensional shape restoration apparatus according to claim 2,
The restoration projection means comprises:
In the process of projecting the visual volume in the PVH method, among the texture feature points extracted by the surface feature extraction unit, a texture feature point existing in a plane configured by projecting the visual volume onto an image is calculated, Means for back-projecting the calculated texture feature points to calculate feature points that are three-dimensional points along the texture;
A three-dimensional shape restoration apparatus comprising: The three-dimensional shape restoration apparatus according to claim 1,
The polygon generation unit
PVH processing means for restoring the shape of a three-dimensional polyhedron consisting of three-dimensional points indicating the shape of the target object based on the PVH method using the polygonal approximate contour extracted by the contour information extraction unit;
A feature point addition processing means for projecting the texture feature point extracted by the surface feature extraction unit onto the surface of the restored three-dimensional polyhedron, and adding a feature point that is a projected three-dimensional point along the texture; ,
Means for connecting a vertex of the restored three-dimensional polyhedron and a feature point that is a projected three-dimensional point along the texture, dividing the three-dimensional polyhedron, and generating a polygon;
A three-dimensional shape restoration apparatus comprising: In the three-dimensional shape restoration apparatus according to claim 4,
The feature point addition processing means is
By projecting the texture feature point extracted by the surface feature extraction unit onto the surface constituting the intersection of the visual volumes calculated by the PVH method, the intersection with the intersection of the visual volumes is calculated, and the intersection Means to regard as a feature point that is a three-dimensional point along the texture,
A three-dimensional shape restoration apparatus comprising: An input image acquisition processing unit configured to acquire an image captured by a plurality of imaging devices installed at multiple viewpoints from an input image acquisition unit and input the acquired image as an input image;
A three-dimensional shape restoration method used in an apparatus for restoring the three-dimensional shape of the target object,
Contour information for extracting a portion corresponding to the target object from the input image input by the input image acquisition processing unit, calculating a contour of the extracted portion, and extracting a polygon approximate contour obtained by approximating the contour to a polygon An extraction step;
A surface feature extraction step of extracting a texture feature point that is a two-dimensional point representing the texture feature of the target object from the input image;
A three-dimensional polyhedron is restored using the polygonal approximate contour extracted in the contour information extraction step, and a feature point that is a three-dimensional point along the texture is added based on the texture feature point extracted in the surface feature extraction step A polygon generating step for generating a polygon based on the restored three-dimensional polyhedron and a feature point that is a three-dimensional point along the added texture;
A three-dimensional shape restoration method characterized by comprising: The three-dimensional shape restoration method according to claim 6,
The polygon generation step includes
Using the polygonal approximate contour extracted in the contour information extraction step, based on the PVH method, while restoring the shape of a three-dimensional polyhedron consisting of three-dimensional points indicating the shape of the target object,
A reconstructing projection step of projecting the texture feature points extracted in the surface feature extracting step onto the surface of the reconstructed three-dimensional polyhedron;
Adding feature points that are three-dimensional points along the texture projected by the restoration projection step;
Generating a polygon along the texture by connecting a vertex of the restored three-dimensional polyhedron and a feature point that is a projected three-dimensional point along the texture, and generating a polygon by dividing the three-dimensional polyhedron;
A three-dimensional shape restoration method characterized by comprising: The three-dimensional shape restoration method according to claim 7,
The restoration projection step comprises:
In the process of projecting the visual volume in the PVH method, out of the texture feature points extracted in the surface feature extraction step, calculate the texture feature points present in the plane configured by projecting the visual volume onto the image, Backprojecting the calculated texture feature points to calculate feature points that are three-dimensional points along the texture;
A three-dimensional shape restoration method characterized by comprising: The three-dimensional shape restoration method according to claim 6,
The polygon generation step includes
PVH processing step for restoring the shape of a three-dimensional polyhedron composed of three-dimensional points indicating the shape of the target object based on the PVH method using the polygonal approximate contour extracted in the contour information extraction step;
A feature point addition processing step of projecting the texture feature point extracted in the surface feature extraction step onto the surface of the restored three-dimensional polyhedron, and adding a feature point that is a projected three-dimensional point along the texture; ,
Connecting the restored vertex of the three-dimensional polyhedron and the feature point that is the projected three-dimensional point along the texture, dividing the three-dimensional polyhedron, and generating a polygon;
A three-dimensional shape restoration method comprising: The three-dimensional shape restoration method according to claim 9,
The feature point addition processing step includes:
By projecting the texture feature point extracted in the surface feature extraction step onto the surface constituting the intersection of the visual volumes calculated by the PVH method, the intersection with the intersection of the visual volumes is calculated, and the intersection A feature point that is a three-dimensional point along the texture,
A three-dimensional shape restoration method characterized by comprising:   A three-dimensional shape restoration program that causes a computer to function as each unit and means in the three-dimensional shape restoration device according to any one of claims 1 to 5.   12. A recording medium on which the three-dimensional shape restoration program according to claim 11 is recorded.