JP2016024728A - Information processing device, method for controlling information processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce time and effort of inputting a three-dimensional shape without providing a constraint in a measurement target object by suppressing decrease in a processing speed.SOLUTION: An information processing device includes an acquisition part for acquiring a picked-up image obtained by imaging a first object and a second object, a first estimation part for estimating the shape of the first object on the basis of the picked-up image, a second estimation part for estimating the shape of a contact surface where the first object comes into contact with the second object on the basis of the shape of the first object, a generation part for generating an image of the shape of the second object on the basis of the shape of the contact surface, and a composition part for compositing the picked-up image and the image of the shape of the second object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program, and more particularly to a technique for inputting a three-dimensional shape of a real object.

  Various methods for sequentially inputting a three-dimensional shape of a real object in real time have been proposed. In recent years, a method of inputting a three-dimensional shape by mapping a depth value on a space while measuring the position and orientation of the depth sensor by a depth sensor capable of depth measurement has been proposed.

  Non-Patent Document 1 proposes a method for capturing a shape such as a desk or a wall arranged in a real environment in real time based on a measurement value measured by a depth sensor and image information obtained by a color camera. In addition to using a depth sensor, a method of measuring a three-dimensional shape from video (a plurality of images) of a real object photographed by a camera has been proposed.

  In Non-Patent Document 2, an image feature point of an object reflected in an image captured by a camera is associated with an image feature point of an image in a past frame, and further, the position and orientation of the camera in each image are estimated, thereby A method for generating a three-dimensional shape has been proposed. Furthermore, a method of inputting a three-dimensional shape of a real object by pointing a plurality of points of the real object with a sensor capable of measuring the position and orientation in the real space such as a magnetic sensor has been proposed.

  Patent Document 1 proposes a method of inputting a three-dimensional shape by pointing a three-dimensional position of a real object using a pen-type stylus with a built-in magnetic sensor. In Patent Document 1, it is possible for an operator who inputs a three-dimensional shape to directly input a three-dimensional shape by an intuitive operation by pointing and inputting the object to which the shape is to be input. .

  In the above three-dimensional shape input method, the camera position and orientation can be estimated in the process of measuring the three-dimensional shape, and the three-dimensional shape data can be sequentially output. The shape can be presented to the operator by combining polygons and vertex information. That is, it is possible to sequentially check how much accuracy the measured shape has, and it is possible to assist in determining whether to remeasure.

  In addition, by superimposing 3D shape data on the real object image, in the augmented reality or mixed reality, the collision expression between the real object and the CG model and the hiding of the CG model in the real object (Occlusion) expression is possible, leading to an improvement in reality.

JP 2004-62758 A

Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon, KinectFusion: Real-Time Dense Surface Mapping and Tracking, in IEEE ISMAR, IEEE, October 2011 Pan, Q., Reitmayr, G., Drummond, T .: ProFORMA: Probabilistic Feature-based On-line Rapid Model Acquisition. In Proc. Of the 20th British Machine Vision Conference (2009) Kenichi Hayashi, Hirokazu Kato, Shogo Nishida, Occlusion Detection of Real Objects using Contour Based Stereo Matching, in 15th International Conference on Artificial Reality and Telexistence 2005, Proc. Of ICAT 2005 pp.180-186, University of Canterbury, Christchurch, New Zealand , Dec. 2005

  However, in the technique of Non-Patent Document 1, since a three-dimensional shape model obtained by a depth sensor becomes higher definition than other methods, the number of polygons used for occlusion expression is enormous, and the drawing update processing speed decreases. There is a case.

  In the technique of Non-Patent Document 2, a pattern as an image feature point is required on the surface of an input real object, and a wall or a table with few patterns is not suitable as a target for inputting a three-dimensional shape. That is, there are restrictions on the real objects that can be input.

  Furthermore, in the technique of Patent Document 1, for a simple shape, the operator can input a three-dimensional shape of a real object simply by sequentially inputting several pieces of position information. However, in order to designate a plane position, three or more points are required, and it takes time and effort to input a three-dimensional shape of a real object composed of a plurality of planes. Furthermore, when inputting a three-dimensional shape of a real object including a curved surface, the number of vertices to be input further increases, which increases the labor of the operator.

  The present invention has been made in view of the above-described problems, and provides an information processing apparatus that suppresses a reduction in processing speed and reduces the labor of inputting a three-dimensional shape without restricting a measurement target object. For the purpose.

An information processing apparatus according to the present invention that achieves the above object has the following configuration. That is,
Acquisition means for acquiring a captured image obtained by imaging the first object and the second object;
First estimating means for estimating a shape of the first object based on the captured image;
Second estimating means for estimating a shape of a contact surface where the first object contacts the second object based on the shape of the first object;
Generating means for generating an image of the shape of the second object based on the shape of the contact surface;
Synthesis means for synthesizing the captured image and the image of the shape of the second object.

  According to the present invention, in a method of sequentially inputting a three-dimensional shape of a real object in real time, it is possible to suppress a decrease in processing speed such as updating, and to input a three-dimensional shape without restricting a measurement target object. Time and effort can be reduced.

It is a block diagram which shows the function structure of the information processing apparatus in one Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the information processing apparatus in one Embodiment of this invention. It is a flowchart which shows the process sequence of information processing apparatus in one Embodiment of this invention. It is a flowchart which shows the procedure of the estimation process of the contact surface of information processing apparatus in one Embodiment of this invention. It is a flowchart which shows the procedure of the recording process of the three-dimensional shape of information processing apparatus in one Embodiment of this invention. It is a schematic diagram in the case of extracting a hand region with a background difference from a stereo image in an embodiment of the present invention. It is a schematic diagram explaining the calculation method when calculating | requiring the three-dimensional position on the outline of a hand in one Embodiment of this invention. It is a schematic diagram which shows the example of the description of the process which inputs the three-dimensional shape of the measurement object in one Embodiment of this invention, and the composite image displayed.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this embodiment, as a method for inputting the three-dimensional shape of a real object, the three-dimensional shape of the contour line of the operator's hand is estimated by a stereo camera, and the real object with which the hand is in contact based on the three-dimensional shape A method for measuring the three-dimensional shape and presenting it to the operator will be described.

<1. Configuration of information processing apparatus>
First, a functional configuration of an information processing apparatus 200 according to an embodiment of the present invention will be described with reference to FIG. As illustrated in FIG. 1, the information processing apparatus 200 includes an image acquisition unit 1000, an image storage unit 1010, a shape estimation unit 1020, a contact surface shape estimation unit 1030, a shape storage unit 1040, and an image generation unit 1050. And an image composition unit 1060.

  The image acquisition unit 1000 acquires a stereo image captured by the imaging device 100 and the imaging device 110. Since the images of the imaging device 100 and the imaging device 110 are used as processed images for stereo measurement, it is assumed that the positional relationship between the imaging device 100 and the imaging device 110 is fixed. It is assumed that the operator's hand 150 and the measurement target object 160 are reflected in the captured image.

  However, the present invention is not limited to acquiring a stereo image, and a monocular image may be used as long as it is a method capable of acquiring the shape of an object in contact with a measurement target object. For example, a monocular image may be used if the method is a method for estimating a shape by matching a teacher image in which the three-dimensional shape of a hand is known and accumulated in the past with the shape of a hand / finger extracted from the image. Is possible.

  The image storage unit 1010 acquires and temporarily stores the video acquired by the image acquisition unit 1000. Image data is transmitted from the image acquisition unit 1000 to the image storage unit 1010 in 1/30 seconds, for example. However, it is necessary to input the image used in the processing in each processing unit from the shape estimation unit 1020 to the image generation unit 1050 to the image synthesis unit 1060. This is because the image synthesizing unit 1060 synthesizes these images in a state in which the image of the three-dimensional shape generated by the image generating unit 1050 and the image of the image storage unit 1010 are synchronized. Therefore, the image storage unit 1010 acquires a new image from the image acquisition unit 1000 when the processing of the image composition unit 1060 is completed in order to realize image synchronization.

  The shape estimation unit 1020 acquires a stereo image stored in the image storage unit 1010, and calculates the three-dimensional position at the sampling point on the contour line representing the shape of the hand 150, that is, the three-dimensional shape of the hand contour line. . A method for calculating the three-dimensional shape of the contour line of the hand 150 will be described later.

  The contact surface shape estimation unit 1030 acquires the three-dimensional shape of the contour line of the hand 150 output from the shape estimation unit 1020 and the input from the operator's keyboard 207, and the measurement target object 160 and the hand 150 are obtained. Estimate the contact surface in contact. A method for estimating the contact surface will be described later.

  The shape storage unit 1040 joins the contact surface output from the contact surface shape estimation unit 1030 and the three-dimensional shape of the measurement target object 160 created from the contact surface in the past, and updates the three-dimensional measurement target object. Generate and store the shape.

  The image generation unit 1050 acquires the three-dimensional shape of the measurement target object 160 stored in the shape storage unit 1040, and based on the acquired information, the three-dimensional shape image at the viewpoint position and orientation of the imaging device 100 and the imaging device 110. Is generated.

  The image composition unit 1060 is configured to generate images of the three-dimensional shapes of the measurement target object 160 generated by the image generation unit 1050 with respect to the images of the imaging device 100 and the imaging device 110 stored in the image storage unit 1010. Is synthesized. The generated composite image is output to the display 208 and visually presented to the operator.

  FIG. 2 is a block diagram illustrating an example of a hardware configuration of the information processing apparatus 200 according to the present embodiment. Reference numeral 201 denotes a CPU, 202 denotes an image capturing device, 203 denotes a storage medium, 204 denotes a ROM, 205 denotes a RAM, 206 denotes a mouse, 207 denotes a keyboard, and 208 denotes a display.

  The hardware configuration shown in FIG. 2 is the same as that of a normal personal computer, for example. The image capturing device 202 is connected to the image capturing apparatus 100 and the image capturing apparatus 110. The image capturing device 202 captures images captured by the imaging devices 100 and 110 into a computer, and corresponds to the image acquisition unit 1000. The image capturing device 202 is, for example, a video capture board. Any device that captures images captured by the imaging device 100 and the imaging device 110 into the computer may be used. The CPU 201 reads and executes a program stored in the storage medium 203, ROM 204 or RAM 205, or an external storage device (not shown). Accordingly, the image storage unit 1010, the shape estimation unit 1020, the contact surface shape estimation unit 1030, the image generation unit 1050, and the image composition unit 1060 function. Each processing unit stores information in the storage medium 203 or reads information from the storage medium 203. In this case, the program code itself read from the storage medium 203 realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  In addition, the operating system (OS) running on the computer may perform part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments may be realized by the processing. Needless to say, it is included.

  Further, the program code read from the storage medium 203 is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU 201 included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the present embodiment is realized by the processing. Needless to say.

<2. Processing flow of information processing apparatus>
Next, with reference to the flowchart of FIG. 3, the procedure of the process which the information processing apparatus 200 which concerns on this embodiment implements is demonstrated.

  In step S3010, the image acquisition unit 1000 acquires images (stereo images) from the imaging device 100 and the imaging device 110. In step S3020, the image storage unit 1010 temporarily records the image acquired by the image acquisition unit 1000 in the memory.

  In step S3030, the shape estimation unit 1020 extracts the region of the hand 150 reflected in the stereo image, obtains the contour line, and acquires the three-dimensional shape of the contour line. In order to acquire the three-dimensional shape of the contour line, for example, the technique presented in Non-Patent Document 3 may be used. Hereinafter, the details of the process of acquiring the three-dimensional shape of the contour line will be described.

  (1) For each of the stereo images 501 and 502 in which the hand 150 shown in FIG. 6 is captured, the regions 150A and 150B of the hand 150 are separated by the background difference.

  (2) Trace the contour lines 150A and 150B of the hand 150 extracted by the background difference, and perform labeling.

  (3) In each of the left and right stereo images 501, 502, the center of gravity position of the region surrounded by the outline is calculated, and the center of gravity position is close to the epipolar line on the other image corresponding to the center of gravity position of one image. Determine the contour line.

  (4) A sampling point is determined on the contour line of one image, and an intersection point between the contour line and the epipolar line associated with the other image is determined and associated.

  (5) Use the principle of triangulation from the camera parameters of the imaging device 100 and the imaging device 110 calibrated in advance and the image coordinate values 610 and 620 (see FIG. 7) of the sampling points associated with the images 501 and 502. Then, the three-dimensional position 630 of the sampling point is obtained. The camera parameters are parameters including a focal length, a principal point position, a lens distortion coefficient, a relative position and orientation of the left and right cameras, and the like.

  (6) The processing of (5) is executed for all sampling points, the three-dimensional positions of the sampling points on the contour line are obtained, and output as the three-dimensional shape 635 of the contour line.

  For example, when the estimation of the contact surface shape is instructed with the keyboard 207 when the hand 150 is brought into contact with the measurement target object 160 shown in FIG. 8A, the three-dimensional shape 635 of the contour line in FIG. A three-dimensional point group as shown in FIG.

  However, since the noise shape other than the hand 150 may be output in the extraction process based on the background difference in (1), the vertex output as the three-dimensional shape 635 is the contour line labeled in (2). Suppose that only the three-dimensional vertex of the object with the longest contour is output.

  However, the present invention is not limited to using the method of Non-Patent Document 3 by the shape estimation unit 1020, and can be applied to any method that can obtain the three-dimensional shape of a real object that contacts the measurement target object 160. Is possible. For example, a flexible cloth may be brought into contact with the measurement target object 160, and the three-dimensional shape may be obtained using the texture of the cloth surface.

  In step S3040, the contact surface shape estimation unit 1030 estimates the 3D shape of the surface where the hand 150 is in contact with the measurement target object 160 from the 3D shape of the contour estimated by the shape estimation unit 1020. The calculation of the three-dimensional shape of the contact surface will be described later with reference to the flowchart of FIG.

  In step S3050, the shape storage unit 1040 updates the three-dimensional shape information stored in the shape storage unit 1040 based on the three-dimensional shape of the contact surface acquired from the contact surface shape estimation unit 1030. In step S3060, the image generation unit 1050 acquires the three-dimensional shape of the measurement target object stored in the shape storage unit 1040 and renders it with a wire frame.

  However, the present invention is not limited to the image generation unit 1050 rendering the three-dimensional shape of the measurement target object with a wire frame, and the degree of coincidence when the operator combines the three-dimensional shape with the live-action image. Any display method that can be confirmed is applicable. For example, in addition to the wire frame display, the inside may be filled with a translucent color. In order to check the state of the occlusion expression of the CG model when the CG model and the real object are combined and displayed, the three-dimensional shape may be rendered with a polygon having a transparency of 100%. That is, since a transparent polygon is drawn on the image of the real object, the CG model arranged behind the three-dimensional shape of the real object is not hidden and displayed by the transparent polygon, and occlusion expression is possible.

  In step S3070, the image composition unit 1060 generates a composite image obtained by combining the stereo image stored in the image storage unit 1010 and the three-dimensional shape image of the real object generated by the image generation unit 1050.

  In step S3080, the image composition unit 1060 functions as a presentation control unit that outputs the composite image to the display 208 and presents it. Thus, each process of the flowchart of FIG. 3 is completed.

<3. Details of the contact surface estimation process>
Next, details of the contact surface estimation processing in step S3040 will be described with reference to the flowchart of FIG. In step S4000, the contact surface shape estimation unit 1030 receives input from the operator via the keyboard 207. The operator recognizes that the hand 150 is in contact with the measurement target object 160 and inputs the keyboard, and instructs the information processing apparatus 200 to record a three-dimensional shape. When the 3D shape acquired in the past frame is stored in the shape storage unit 1040, the acquisition position is obtained while observing the composite image of the measurement target object 160 displayed on the display 208 and the 3D shape. It is possible to issue an instruction to record while adjusting. If the operator gives an instruction to record the contact surface via the keyboard 207, the process proceeds to step S4010. If the contact surface is not recorded in the current frame (acquired stereo image), the process ends.

  Note that the present invention is not limited to the operator inputting the timing for recording the shape of the contact surface using a keyboard, but can be applied to any interface that can instruct the recording timing of the operator. is there. For example, the contact surface shape estimation unit 1030 may be instructed by recognizing a mouse click or a gesture of the other hand of the operator shown in the captured image.

  In step S4010, the contact surface shape estimation unit 1030 acquires the three-dimensional shape of the contour line estimated by the shape estimation unit 1020. In step S4020, the contact surface shape estimation unit 1030 estimates a proximity plane that is close to the three-dimensional shape of the contour line acquired in step S4010. For the plane close to the three-dimensional shape of the contour line, a plurality of three-dimensional vertex positions constituting the three-dimensional shape and the plane parameters that minimize the plane may be obtained by a least square approximation expression. Here, the plane parameter refers to a normal vector of a, b, c of the plane equation ax + by + cz = 1.

  However, the present invention is not limited to using the method of least square approximation as a method of estimating adjacent planes, and can be applied to any method that can calculate a plane close to the three-dimensional shape of the contour line. is there.

  In step S4030, the contact surface shape estimation unit 1030 obtains a cuboid circumscribing the three-dimensional shape of the contour of the hand 150, derives a line of intersection with the proximity plane estimated in step S4020, and determines the plane region of the hand 150. decide. First, a cuboid circumscribing the three-dimensional shape of the contour line is a cuboid using the maximum and minimum values of the X coordinate value, the Y coordinate value, and the Z coordinate value that can be taken by a plurality of three-dimensional vertex positions constituting the three-dimensional shape. What is necessary is just to obtain | require the vertex position of. Further, the intersection line between the rectangular parallelepiped and the proximity plane may be calculated by calculating the formula of six planes constituting the rectangular parallelepiped, obtaining the intersection line between the six planes and the proximity plane, and extracting the line segment on the surface of the rectangular parallelepiped. Further, the intersections of the obtained plurality of line segments are temporarily recorded as the apexes of the adjacent planar area.

  However, the present invention is not limited to obtaining a circumscribed rectangular parallelepiped and obtaining a planar region from the intersection line of the rectangular parallelepiped and the adjacent plane in order to set the planar region, but the three-dimensional shape of the contour line of the hand 150 Any method can be applied as long as it can calculate a plane region close to.

  In step S4040, the contact surface shape estimation unit 1030 generates a Bezier patch using the three-dimensional vertex position of the contour line as a control point for the proximity plane region determined in step S4030, and defines a curved surface. The Bezier patch is one method of expressing a curved surface in which a Bezier curve is expressed by two parameters (u, v). The curved surface defined by the Bezier patch is a curved surface close to a plurality of three-dimensional vertex positions of the contour line of the hand 150 (hereinafter referred to as a proximity curved surface).

  The present invention is not limited to obtaining a curved surface using a Bezier patch, but can be applied to any method that can generate a close curved surface of the contour of the hand 150. For example, the curved surface may be defined by a B-spline curved surface.

  In step S4050, the contact surface shape estimation unit 1030 translates the proximity curved surface generated near the contour line of the hand 150 in the surface direction assumed to be in contact. Thereby, the error which arises in the surface which contacts the outline measured by a stereo image under the influence of the thickness of the hand 150 can be reduced. The surface direction assumed to be in contact has a small relative angle to the visual axis direction of the imaging device 100 and the imaging device 110, for example, among the normals of the two surfaces having the largest area among the rectangular parallelepiped obtained in step S4040. Select the normal as the direction vector of the contact surface. That is, it is assumed that the hand 150 is spread out and brought into contact with the measurement target object 160 and is offset to the palm side that is not reflected in the stereo image. For example, in consideration of the thickness of the hand 150, the offset movement amount is 1 cm, and the contact surface and the proximity plane are brought closer by translating the proximity curved surface along the direction vector of the contact surface described above. However, the present invention is not limited to setting the offset movement amount to 1 cm, and can be applied as long as the offset amount reduces the difference between the object to be contacted and the contact surface. Further, not only the parallel movement but also the posture change by rotation may be added.

  In step S4060, the contact surface shape estimation unit 1030 performs polygonization by inputting a constant to the (u, v) parameter of the proximity curved surface expressed by a Bezier patch and calculating a three-dimensional position on the proximity plane. . For the parameters of (u, v) to be input, for example, u and v may each be specified between 0 and 1 at 0.2 intervals and specified in 5 equal parts in each direction. As described above, when the proximity plane is made into a polygon, by controlling so as not to obtain high-definition information, it is possible to prevent a decrease in processing speed as in the case of using the method of Non-Patent Document 1. When the polygon vertices of the proximity curved surface can be generated, the proximity curved surface is output as “contact surface shape information” to the shape storage unit 1040, and the processing is terminated. The present invention is not limited to designating u and v between 1 and 0.2 at intervals of 0.2. The interval may be adjusted according to the processing performance of the device to be executed. Thus, the processes in the flowchart of FIG. 4 are completed.

<4. Details of 3D shape recording process>
Further, details of the three-dimensional shape recording process in step S3050 will be described with reference to the flowchart of FIG. In step S5000, the shape storage unit 1040 determines whether or not the contact surface shape information output from the contact surface shape estimation unit 1030 is input to the shape storage unit 1040. If input, the process proceeds to step S5010. If not entered, the process is terminated.

  In step S5010, the shape storage unit 1040 reads the contact surface shape information input from the contact surface shape estimation unit 1030. In step S5020, the shape storage unit 1040 calculates the distance between the three-dimensional vertex position of the three-dimensional shape of the measurement target object 160 calculated in the past frame and the three-dimensional vertex position of the contact surface. For example, the top five corresponding points are temporarily stored from the calculated vertex distance list.

  In step S5030, the shape storage unit 1040 connects the corresponding upper five points obtained in step S5020 with straight lines. The present invention is not limited to connecting the corresponding points of the top five points with straight lines, and may be controlled by the operator. In step S5040, the shape storage unit 1040 records the connected shape as an updated three-dimensional shape. Thus, the processes in the flowchart of FIG. 5 are completed.

  Through the above processing, the three-dimensional shape of the measurement target object 160 becomes wider by adding the input information of the contact surface. For example, when the operator touches the measurement target object 160 as shown in FIG. 8A with the hand 150 as shown in FIG. 8B and the operator gives an input instruction via the keyboard 207, 3 A dimensional shape 700 is generated. Further, as shown in FIG. 8C, in another frame after the elapse of time, the operator causes the hand 150 to contact the measurement target object 160 in another place, and the operator gives an input instruction via the keyboard 207. As a result, the three-dimensional shape 710 is joined to the existing three-dimensional shape 700 and the contact surfaces are joined, and stored as a new three-dimensional shape.

  The three-dimensional shape stored in this manner is presented as a composite image on the display 208 through the processing of the image generation unit 1050 and the image synthesis unit 1060. Since the three-dimensional shape of the current measurement target object 160 is combined with the actual image of the measurement target object 160 and displayed, the operator can determine which region should be filled closer to the measurement target object 160. .

  In addition, since the operator can measure the measurement target object 160 by bringing his / her hand 150 into contact with the measurement target object 160, the shape can be input even if there is no pattern on the surface of the measurement target object 160. Furthermore, it is not necessary to input a huge number of vertices for a complicated shape, and it is possible to input a rough shape simply by performing an input instruction while sliding the hand 150 on the measurement target object 160. Less effort.

(Modification 1: A flat surface, not a curved surface, is used as a contact surface)
In the above-described embodiment, the contact surface shape estimation unit 1030 generates a proximity curved surface by estimating the curved surface in step S4040 and outputs it as a three-dimensional shape of the contact surface. However, the present invention is not limited to generating a curved surface as the contact surface, and a flat surface may be used.

  In the case where a plane is used, the processing of step S4040 and step S4060 in the flowchart of FIG. 4 showing the processing procedure of the contact surface shape estimation unit 1030 is different. Other configurations and processes may be the same as those in the above-described embodiment.

  Specifically, the processing of the contact surface shape estimation unit 1030 according to the first modification excludes the processing of step S4040 and step S4060 in the flowchart shown in FIG. 4 and uses the proximity plane region acquired in step S4030 as it is as the contact surface. It may be output as the shape.

(Modification 2: The contact surface is not joined as one shape, but is recorded as an independent contact surface)
In the above-described embodiment, in step S5030, the shape storage unit 1040 searches for a proximity point between the input contact surface and the existing three-dimensional shape, and performs a process of joining.

  However, the present invention is not limited to adding the three-dimensional shape of the additionally input contact surface as one continuous three-dimensional shape while joining. A three-dimensional shape may be composed of a plurality of planes or curved surfaces.

  In order to record the shape of a plurality of contact surfaces as a three-dimensional shape, it is input by excluding the processing of step S5020 and step S5030 in the flowchart of FIG. 5 showing the processing procedure of the shape storage unit 1040 of the above-described embodiment. The three-dimensional shape of the contact surface may be added as a three-dimensional shape without joining. Other configurations and processes are the same as those in the above-described embodiment.

(Variation 3: Record the contact surface using both hands)
In the above-described embodiment, the method in which the operator calculates the contact surface by imaging the one hand 150 by the imaging device 100 and the imaging device 110 is illustrated. However, the present invention is not limited to estimating the contact surface by observing only one hand, and the contact surface may be estimated using information on both hands.

  When it is desired to input a real object having a wide plane as the measurement target object 160, the contact plane estimation process performed by the contact plane shape estimation unit 1030 in step S4020 is performed by estimating the contact plane using information on both hands. The accuracy may be improved. In addition, since two contact surfaces can be estimated at the same time, the trouble of inputting by the operator is reduced.

  Specifically, in the process of the above-described embodiment, the process of calculating the three-dimensional shape 635 of the hand outline of the shape estimation unit 1020 may be configured to output the outlines of a plurality of objects. For example, in the above-described embodiment, the contour line with the longest line is selected and output in the process of step S3030, but the contour line with the second longest line may be output. That is, both hands are extracted using the premise that the two hands reflected in the image are reflected larger than the outline of the other noise object. The configuration and processing other than step S3030 are the same as those in the above-described embodiment.

  As described above, according to the present invention, in the method of sequentially inputting the three-dimensional shape of a real object in real time, it is possible to suppress a decrease in processing speed such as update without limiting the measurement target object, It is possible to reduce the trouble of inputting a three-dimensional shape.

  In addition, according to the present invention, since the composite image of the live-action image and the three-dimensional shape image can be input to the operator while presenting the operator with the display, the operator can input the current three-dimensional shape. And the degree of coincidence between the measurement target object and the part of the measurement object can be confirmed in real time. That is, since the operator can determine which part should be preferentially input, the input of the three-dimensional shape of the measurement target object can be completed with a relatively small number of man-hours.

  Furthermore, according to the present invention, since it is possible to input a three-dimensional shape of a target real object with a relatively small number of man-hours, mixed reality (hereinafter abbreviated as MR) in a situation where it takes time to prepare for installation. Even in applications, realism can be improved. In other words, since the pre-preparation of the three-dimensional shape used for the collision expression between the real object and the CG model in the MR application and the hidden (occlusion) expression of the CG model on the real object is shortened, the application scene can be expanded. Become.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  200: Information processing device 1000: Image acquisition unit 1010: Image storage unit 1020: Shape estimation unit 1030: Contact surface shape estimation unit 1040: Shape storage unit 1050: Image generation unit 1060: Image synthesis unit

Claims (13)

Acquisition means for acquiring a captured image obtained by imaging the first object and the second object;
First estimating means for estimating a shape of the first object based on the captured image;
Second estimating means for estimating a shape of a contact surface where the first object contacts the second object based on the shape of the first object;
Generating means for generating an image of the shape of the second object based on the shape of the contact surface;
An information processing apparatus comprising: a combining unit that combines the captured image and an image of the shape of the second object. It further comprises an input means for receiving an operator's instruction,
The information processing apparatus according to claim 1, wherein the second estimation unit estimates the shape of the contact surface based on an instruction from the operator received by the input unit.   The information processing apparatus according to claim 1, wherein the first estimation unit estimates a three-dimensional shape of a contour line of the first object.   The second estimation means derives a surface close to the three-dimensional shape of the contour line of the first object and estimates the shape of the contact surface by translating the surface. Item 4. The information processing apparatus according to any one of Items 1 to 3.   The generation unit generates a three-dimensional image of the second object based on the shape of one or a plurality of contact surfaces estimated by the second estimation unit. 5. The information processing apparatus according to any one of 4.   The information according to claim 5, wherein the generation unit generates a three-dimensional image of the second object by joining a plurality of contact surfaces estimated by the second estimation unit. Processing equipment.   The information processing apparatus according to claim 1, wherein the first object is an operator's hand.   The information processing apparatus according to claim 1, wherein the contact surface is a curved surface.   The information processing apparatus according to claim 1, wherein the contact surface is a flat surface.   The information processing apparatus according to claim 1, wherein the captured image is a stereo image.   11. The display control unit according to claim 1, further comprising a presentation control unit that causes a display to present the captured image synthesized by the synthesis unit and an image of the shape of the second object. Information processing device. A method for controlling an information processing apparatus,
An obtaining unit obtaining a captured image obtained by imaging the first object and the second object;
First estimating means estimating the shape of the first object based on the captured image;
A second estimating unit estimating a shape of a contact surface on which the first object contacts the second object based on a shape of the first object;
A step of generating an image of the shape of the second object based on the shape of the contact surface;
A method for controlling the information processing apparatus, comprising: a step of synthesizing the captured image and an image of the shape of the second object.   A program for causing a computer to execute each step of the control method for the information processing apparatus according to claim 12.