JP2018142273A - Information processing apparatus, method for controlling information processing apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform both image composition and contact determination with high accuracy.SOLUTION: An information processing apparatus includes a model generation unit configured to generate a first model constituted by surfaces connecting contour points of a real object extracted from an image obtained by photographing the real object, a model acquisition unit configured to acquire a second model representing a surface shape of the real object, a contact determination unit configured to determine whether the real object and a virtual object are in contact with each other on the basis of the second model and a CG model of the virtual object, and an image composition unit configured to synthesize the virtual object with the image on the basis of the first model and the CG model of the virtual object.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program, and more particularly, to a technique for realizing an interaction between a real object and a virtual object.

  There is a mixed reality (MR) technique that fuses a real space and a virtual space so that an experiencer can interact with a virtual object. In MR technology, interaction is achieved by synthesizing and presenting computer graphics (hereinafter referred to as “CG”) representing a virtual object to a real landscape, or expressing contact between a real object and a virtual object. Is realized.

  Patent Document 1 discloses a technique for generating a polygon by connecting the contour of a real object extracted from a captured image, and synthesizing the virtual object with the captured image based on the front-rear relationship between the polygon and the virtual object. . Patent Document 2 discloses a technique for estimating a finger shape from a captured image and determining contact with a virtual object.

Japanese Patent No. 5574852 JP 2009-3813 A

  However, since the polygon generated by the technique of Patent Document 1 is obtained by connecting the contour of a real object and does not reflect the thickness, if the contact determination is performed using this polygon, the determination result is shifted by the thickness. There is a risk. In addition, since the finger shape estimated by the technique of Patent Document 2 does not necessarily follow the contour line, if image synthesis is performed using this finger shape, the image may be synthesized by protruding the contour. . As described above, the conventional technique has a problem that it is difficult to perform both image synthesis and contact determination with high accuracy.

  The present invention has been made in view of the above problems, and an object thereof is to provide a technique for performing both image synthesis and contact determination with high accuracy.

An information processing apparatus according to the present invention that achieves the above object is as follows.
Model generating means for generating a first model composed of a plane connecting contour points of the real object extracted from an image of the real object;
Model acquisition means for acquiring a second model representing the surface shape of the real object;
Contact determination means for determining contact between the real object and the virtual object based on the second model and a CG model of the virtual object;
Image synthesizing means for synthesizing the virtual object in the image based on the first model and the CG model of the virtual object;
It is characterized by providing.

  According to the present invention, both image synthesis and contact determination can be performed with high accuracy.

2 is a diagram illustrating a hardware configuration of the information processing apparatus according to the first embodiment. FIG. 1 is a diagram illustrating a functional configuration of an information processing apparatus according to a first embodiment. 3 is a flowchart illustrating a procedure of processing performed by the information processing apparatus according to the first embodiment. It is a figure which shows an example of the contour point extraction method of the finger which concerns on Embodiment 1. FIG. It is a figure which shows typically the process of the contact determination part which concerns on Embodiment 1. FIG. It is a figure which shows typically the process of the image synthetic | combination part which concerns on Embodiment 1. FIG. It is a figure which shows typically the outline model and surface model which concern on Embodiment 1. FIG. It is a figure which shows typically the contact determination process using the surface model which concerns on table Embodiment 1. FIG. It is a figure which shows the function structure of the information processing apparatus which concerns on Embodiment 3. FIG. 10 is a flowchart illustrating a procedure of processing performed by the information processing apparatus according to the third embodiment. It is a figure which shows typically the projection area | region of the outline model and surface model which concern on Embodiment 3. FIG. It is a figure which shows the example of the judgment by the combination judgment part which concerns on Embodiment 3. FIG. 10 is a flowchart illustrating a procedure of processing performed by the information processing apparatus according to the second embodiment. 10 is a flowchart illustrating a processing procedure of a contact determination unit according to the second embodiment.

  Hereinafter, embodiments will be described with reference to the drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(Embodiment 1)
In the present embodiment, the contact determination process uses (selects) the surface model of the real object, and the image composition process uses (selects) the contour model of the real object, so that high-precision contact in consideration of the thickness of the real object is used. An example will be described in which both the determination process and the clean synthesis process along the contour of the real object are realized.

<Hardware configuration>
FIG. 1 is a diagram illustrating a hardware configuration of the information processing apparatus according to the present embodiment. The information processing apparatus 100 includes a CPU 101, a RAM 102, a ROM 103, an HDD 104, an interface 105, and a system bus 106. The information processing apparatus 100 is connected to an HMD (Head Mounted Display) 107 and a sensor 108 via an interface 105.

  The CPU 101 executes a program stored in the ROM 103 using the RAM 102 as a work memory, and comprehensively controls each component to be described later via the system bus 106. Thereby, various processes described later are executed.

  The HDD 104 has a role as a secondary storage device. The CPU 101 can read data from the HDD 104 and write data to the HDD 104. The secondary storage device may be a storage device such as an optical disk drive in addition to the HDD.

  The interface 105 exchanges data with external devices such as the HMD 107 and the sensor 108. The HMD 107 includes photographing units 109 and 110 and display units 111 and 112, and displays captured images of the photographing units 109 and 110 and a composite image described later on the display units 111 and 112. Note that the present invention is not limited to the case where the HMD 107 is connected to the information processing apparatus 100, and can be applied to any configuration that can acquire a captured image of a real object.

  The sensor 108 is a sensor that measures the shape of a specific real object. In the following description, it is assumed that the sensor 108 is a sensor that measures the shape of a finger. However, the present invention is not limited to this, and can be applied to measuring an arbitrary real object. As the sensor 108, for example, a depth sensor, an IR (Infrared Ray) sensor, a data glove for hand tracking, and the like can be used.

  In addition, although the component of the information processing apparatus 100 exists besides the above, since it is not the main point of this invention, description is abbreviate | omitted.

<Functional configuration>
FIG. 2 is a diagram illustrating a functional configuration of the information processing apparatus according to the present embodiment. The information processing apparatus 100 includes an image acquisition unit 201, a contour point extraction unit 202, a three-dimensional position calculation unit 203, a contour model generation unit 204, a surface model acquisition unit 205, a contact determination unit 206, and an image composition unit 207. .

  The image acquisition unit 201 acquires captured images from the imaging units 109 and 110, respectively. The contour point extraction unit 202 extracts the contour points of the fingers from each of the captured images acquired by the image acquisition unit 201. The three-dimensional position calculation unit 203 calculates the three-dimensional position of the contour point extracted by the contour point extraction unit 202.

  The contour model generation unit 204 generates a contour model by connecting the contour points extracted by the contour point extraction unit 202 based on the three-dimensional position of the contour points calculated by the three-dimensional position calculation unit 203. The surface model acquisition unit 205 acquires a surface model representing the surface shape of the finger from the sensor 108.

  The contact determination unit 206 determines contact between the finger and the virtual object based on the overlap between the surface model acquired by the surface model acquisition unit 205 and the virtual object. Details of the virtual object will be described later. The image composition unit 207 synthesizes the virtual object in the captured image acquired by the image acquisition unit 201 based on the front-rear relationship between the contour model and the CG model of the virtual object.

<Processing>
FIG. 3 is a flowchart illustrating a procedure of processing performed by the information processing apparatus according to the present embodiment. In step S301, the image acquisition unit 201 acquires captured images from the imaging units 109 and 110, respectively. Hereinafter, the captured images acquired from the imaging units 109 and 110 are collectively referred to as “stereo images”.

  In step S302, the contour point extraction unit 202 extracts the contour points of the fingers from each of the stereo images acquired in step S301. Here, FIG. 4 is a diagram illustrating an example of a finger contour point extraction method. The contour point extraction unit 202 first extracts finger regions 403 and 404 from the captured images 401 and 402 based on the hand color registered in advance. Then, intersection points between horizontal lines (or epipolar lines) drawn at regular intervals in the captured images 401 and 402 and the finger regions 403 and 404 are extracted as contour points. Note that contour point extraction in the present invention is not limited to this method, and contour points of fingers can be extracted by any method.

  In step S303, the three-dimensional position calculation unit 203 calculates the three-dimensional position of the contour point extracted in step S302. Here, any method can be used to calculate the three-dimensional position. For example, as described in Patent Document 1, the contour points extracted from the captured images 401 and 402 are associated with each other, and the three-dimensional position of the contour points is calculated by applying a triangulation technique. Can do.

  In step S304, the contour model generation unit 204 generates a finger contour model by connecting the contour points extracted in step S302. At that time, the contour model generation unit 204 generates a contour model as a three-dimensional polygon by using the three-dimensional position calculated in step S303. Details of the contour model will be described later.

  In step S <b> 305, the surface model acquisition unit 205 acquires a surface model representing the surface shape of the finger from the sensor 108. Details of the finger surface model will be described later.

  In step S306, the contact determination unit 206 determines the contact between the finger and the virtual object based on the overlap between the surface model acquired in step S305 and the CG model of the virtual object. Here, FIG. 5 is a diagram schematically illustrating processing of the contact determination unit 206 according to the present embodiment. FIG. 5A shows a state where the finger 501 and the virtual object 502 are not in contact, and FIG. 5B shows a state where the finger 501 and the virtual object 502 are in contact. The contact determination unit 206 acquires a CG model representing the shape of the virtual object 502 from the RAM 102, the ROM 103, the HDD 104, and the like, and checks whether or not there is an overlap with the surface model representing the shape of the finger 501 so that the finger 501 and the virtual object 502 are overlapped. Judging contact with.

  In step S <b> 307, the image composition unit 207 synthesizes the virtual object 502 on the captured images 401 and 402 based on the front and back relationship between the contour model generated in step S <b> 304 and the CG model of the virtual object 502. Here, FIG. 6 is a diagram schematically illustrating processing of the image composition unit 207 according to the present embodiment. 6A shows a state in which the finger 501 is in front of the virtual object 502, and FIG. 6B shows a state in which the finger 501 is behind the virtual object 502. 6A, a part of the virtual object 502 is hidden by the finger 501 and a part of the finger 501 is hidden by the virtual object 502 in FIG. The image composition unit 207 generates a composite image by performing a rendering process based on the front-rear relationship between the finger 501 and the virtual object 502.

  Specifically, the distances from the photographing units 109 and 110 to the finger 501 and the virtual object 502 are respectively calculated, and the closer one of the finger 501 and the virtual object 502 is drawn for each pixel. Further, the image composition unit 207 blinks or moves the virtual object 502 on the synthesized image based on the determination result by the contact determination unit 206, thereby displaying the display mode of the virtual object 502 on the synthesized image. May be changed dynamically. For example, it may be blinked or moved when touching. Thereby, the interaction between the finger 501 and the virtual object 502 can be realized.

  Hereinafter, the features of the contour model and the surface model will be described with reference to FIG. FIG. 7A schematically shows the contour model 701, and FIG. 7B schematically shows the surface model 702. As shown in FIG. 7A, since the contour model 701 is a model that connects the contour points extracted by the contour point extraction unit 202, the shape thereof follows the contour line of the finger 501 on the captured images 401 and 402. It becomes a thing. Therefore, when the contour model 701 is used to represent the shape of the finger 501 in the processing of the image composition unit 207, it is possible to perform a beautiful composition along the contour line of the finger 501 (the protrusion of the contour can be reduced). ). However, since the contour model 701 does not have a vertex in a portion other than the contour (the inside of the finger 501 such as the palm or the back of the hand), the thickness of the finger 501 cannot be reflected. Therefore, if the shape of the finger 501 is represented by the contour model 701 in the processing of the contact determination unit 206, the determination result is shifted by the thickness of the finger 501.

  On the other hand, as shown in FIG. 7B, the surface model 702 is a model representing the surface shape of the finger generated by fleshing the bone connecting the finger joints by the thickness of the finger. This model reflects the thickness of 501. Therefore, when the shape of the finger 501 is represented by the surface model 702 in the processing of the contact determination unit 206, it is possible to perform highly accurate contact determination in consideration of the thickness of the finger 501. However, unlike the contour model 701 in which contour points are connected, the surface model 702 does not necessarily have a shape along the contour line of the finger 501 on the captured images 401 and 402. This is because, if the joint position and finger thickness are different from the actual ones, the contours of the fingers 501 on the captured images 401 and 402 are deviated accordingly. For this reason, when the shape of the finger 501 is represented by the surface model 702 in the processing of the image composition unit 207, it is not possible to cleanly synthesize along the contour line of the finger 501 and the virtual object 502 (the contours increase in number). ).

  On the other hand, the main point of the present invention is to appropriately use a plurality of models such as the contour model 701 and the surface model 702 in the processing of the contact determination unit 206 and the image composition unit 207. In particular, in the present embodiment, the surface model 702 is used for the processing of the contact determination unit 206, and the contour model 701 is used for the processing of the image composition unit 207. Thereby, both a highly accurate contact determination process considering the thickness of the finger 501 and a beautiful composition process along the contour line of the finger 501 can be realized at the same time.

  The surface model of the finger 501 is projected on the captured images 401 and 402, and the result is displayed semi-transparently so that the user can recognize where the surface model is on the captured images 401 and 402. Good.

  As described above, in this embodiment, the surface model of the real object is used in the contact determination process, and the contour model of the real object is used in the image composition process. Thereby, it is possible to realize both a highly accurate contact determination process in consideration of the thickness of the real object and a beautiful synthesis process along the contour line of the real object.

(Embodiment 2)
In the first embodiment, the example in which the surface model 702 is used for the processing of the contact determination unit 206 and the contour model 701 is used for the processing of the image composition unit 207 has been described. In contrast, in this embodiment, an example will be described in which contact determination processing with less discomfort is realized by using both the contour model 701 and the surface model 702 for the processing of the contact determination unit 206. That is, the present embodiment is different from the first embodiment mainly in that both the contour model 701 and the surface model 702 are used for the processing of the contact determination unit 206. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals as those in FIGS.

<Processing>
FIG. 13 is a flowchart illustrating a procedure of processing performed by the information processing apparatus according to the present embodiment. Note that the processes in steps S301 to S305 and S307 in FIG. 13 are the same as the processes described with reference to FIG.

  In step S <b> 1301, the contact determination unit 206 determines contact between the finger 501 and the virtual object 502 based on the overlap between the contour model 701 and the surface model 702 and the virtual object 502. Details will be described later.

  Here, FIG. 8 is a diagram schematically showing a contact determination process using the surface model 702. Below, with reference to FIG. 8, the problem of the contact determination process using the surface model 702 is demonstrated. As shown in FIG. 8, when the surface model 702 is projected onto the captured images 401 and 402, the shape may not follow the contour line of the finger 501. Therefore, even when the finger 501 does not overlap the virtual object 502 on the captured images 401 and 402, it may be determined that the finger 501 and the virtual object 502 are in contact with each other. This is a state where there is a mismatch between the visual information of the user who is performing the MR experience and the contact determination, which gives the user a sense of incongruity. In the present embodiment, in order to solve such a problem, the CG model representing the shape of the virtual object 502 and the surface model 702 overlap, and the finger 501 overlaps the virtual object 502 on the captured images 401 and 402. It is determined that the two are in contact. By making such a determination, it is possible to prevent the user from being determined to be in contact with the user's visual sense (on the captured images 401 and 402).

  Next, FIG. 14 is a flowchart illustrating a processing procedure of the contact determination unit according to the present embodiment. In step S1401, the contact determination unit 206 determines the overlap between the surface model 702 and the virtual object 502. If it is determined in step S1401 that they do not overlap, in step S1402, the contact determination unit 206 determines that the finger 501 and the virtual object 502 are not in contact, and ends the process. On the other hand, if it is determined in step S1401 that they overlap, the process proceeds to step S1403.

  In step S1403, the contact determination unit 206 projects the contour model 701 and the virtual object 502 onto the captured images 401 and 402. In step S1404, the contact determination unit 206 determines whether the projection result of the contour model 701 and the projection result of the virtual object 502 overlap. If it is determined in step S1404 that they do not overlap, the process proceeds to step S1402, and the contact determination unit 206 determines that the finger 501 and the virtual object 502 are not in contact with each other, and ends the process. On the other hand, if it is determined in step S1404 that they overlap, the process proceeds to step S1405, and the contact determination unit 206 determines that the finger 501 and the virtual object 502 are in contact, and ends the process.

  Note that the finger areas 403 and 404 extracted in step S302 may be used instead of the projection area of the contour model 701 in step S1404. Since other processes are the same as those in the first embodiment, the description thereof is omitted.

  As described above, in this embodiment, the CG model representing the shape of the virtual object 502 and the surface model 702 overlap, and the finger 501 overlaps the virtual object 502 on the captured images 401 and 402. In addition, it is determined that both are in contact. Accordingly, it is possible to prevent the finger 501 and the virtual object 502 from being determined to be in contact with each other when the finger 501 and the virtual object 502 do not overlap on the captured images 401 and 402.

(Embodiment 3)
In the first embodiment and the second embodiment, an example in which a predetermined combination of the contour model 701 and the surface model 702 is used for the processing of the contact determination unit 206 and the image composition unit 207 has been described. On the other hand, in the present embodiment, an example will be described in which a combination of the contour model 701 and the surface model 702 used for processing of the contact determination unit 206 and the image composition unit 207 is determined, and processing is performed based on the determination result. The present embodiment is different from Embodiments 1 and 2 mainly in that the combination of the contour model 701 and the surface model 702 used for the processing of the contact determination unit 206 and the image composition unit 207 is determined. Therefore, in the description of the present embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals as those in FIGS.

<Functional configuration>
FIG. 9 is a diagram illustrating a functional configuration of the information processing apparatus according to the present embodiment. The information processing apparatus 100 further includes a combination determination unit 901 in addition to the configuration described with reference to FIG. The combination determination unit 901 determines a combination of the contour model 701 and the surface model 702 used for processing by the contact determination unit 206 and the image composition unit 207. Details of the determination process will be described later.

<Processing>
FIG. 10 is a flowchart illustrating a procedure of processing performed by the information processing apparatus according to the present embodiment. Note that the processing in steps S301 to S305 is the same as the processing described with reference to FIG.

  In step S <b> 1001, the combination determination unit 901 determines a combination of the contour model 701 and the surface model 702 that are used for processing by the contact determination unit 206 and the image composition unit 207. Details of the processing of the combination determination unit 901 will be described later.

  In step S1002, the contact determination unit 206 determines contact between the finger 501 and the virtual object 502 using the combination model determined in step S1001. Since the contact determination method is the same as that of the first embodiment, the description thereof is omitted.

  In step S1003, the image composition unit 207 synthesizes the virtual object 502 on the captured images 401 and 402 using the combination model determined in step S1001. Since the image composition method is the same as that of the first embodiment, the description thereof is omitted.

  Here, FIG. 11 is a diagram schematically showing the projection areas of the contour model 701 and the surface model 702. Below, with reference to FIG. 11, the overlapping degree of the projection area calculated by the process of the combination determination part 901 is demonstrated.

  The combination determination unit 901 projects both the contour model 701 and the surface model 702 on the captured images 401 and 402, and combines them according to the degree of overlap between the projection region 1101 of the contour model 701 and the projection region 1102 of the surface model 702. Judging. As an index indicating the degree of overlap, for example, an F value represented by the following equation can be used.

  Here, TP is a logical sum area of “contour model projection area 1101” and “surface model projection area 1102”, and FN is “contour model projection area 1101” and “logical model negative of surface model projection area 1102”. This is the number of pixels in the logical sum area. Further, FP is a logical sum area of “logical contour projection area 1101” and “surface model projection area 1102”, and TN is “logical negation of contour model projection area 1101” and “surface model projection area 1102”. This is the number of pixels in the logical sum area of “logical negation of”. Note that the present invention is not limited to the case where the F value is used for the overlapping degree, and any other index can be used as the overlapping degree.

  Next, FIG. 12 is a diagram illustrating an example of determination by the combination determination unit 901. Hereinafter, with reference to FIG. 12, details of the processing of the combination determination unit 901 will be described. In the following example, the combination is determined based on whether the degree of overlap between the projection area 1101 of the contour model and the projection area 1102 of the surface model is greater than or less than a predetermined threshold.

  In Example 1 of FIG. 12, when the degree of overlap is equal to or greater than a threshold (high), the surface model 702 is used for contact determination, and the contour model 701 is used for image composition, and when the degree of overlap is less than the threshold (low), The model to be used is switched to the contour model 701. That is, the contour model is used in both processes. In Example 2, when the degree of overlap is less than the threshold, the model used for image composition is switched to the surface model 702. That is, the surface model is used for both processes.

  With these processes, when the degree of overlap is high, both contact determination and image composition are performed with high accuracy, and when the degree of overlap is low, the sense of discomfort caused by the difference between the result of contact determination and the result of image composition is prevented. be able to.

  In addition, this invention is not limited to the example of FIG. 12, Arbitrary combinations can be determined. Further, as in the second embodiment, the combination may be determined so that both the contour model 701 and the surface model 702 are used for one process. Further, the present invention is not limited to the case where the overlapping degree is used to determine the combination of the contour model 701 and the surface model 702. For example, the combination of models may be determined according to mode settings such as a high accuracy mode and a speed priority mode. In this case, for example, different models are used for contact determination and image composition in the high accuracy mode, and the same model is used for contact determination and image composition in the speed priority mode.

  Further, the user may directly specify the combination of the surface model 702 and the contour model 701. Further, a warning may be given to the user when the degree of overlap is lower than a preset threshold value. Although the warning method is arbitrary, for example, the warning may be output by outputting a warning in text on the display units 111 and 112 or by sounding from a speaker (not shown).

  As described above, in the present embodiment, the combination of the contour model 701 and the surface model 702 used for processing of the contact determination unit 206 and the image composition unit 207 is determined, and processing is performed based on the determination result. Thus, processing can be performed with a suitable combination of models, and contact determination and image synthesis can be performed with higher accuracy.

[Modification]
In each embodiment described above, the case where the surface model 702 acquired from the sensor 108 is used has been described as an example. However, the present invention is not limited to this, and the surface model 702 may be generated using information (depth image, IR image, etc.) acquired from the sensor 108 or the captured images 401 and 402, and the surface model 702 may be used. .

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  100: Information processing device 201: Image acquisition unit 202: Contour point extraction unit 203: Three-dimensional position calculation unit 204: Contour model generation unit 205: Surface model acquisition unit 206: Contact determination unit 207: Image Combining unit, 901: combination determining unit

Claims (13)

Model generating means for generating a first model composed of a plane connecting contour points of the real object extracted from an image of the real object;
Model acquisition means for acquiring a second model representing the surface shape of the real object;
Contact determination means for determining contact between the real object and the virtual object based on the second model and a CG model of the virtual object;
Image synthesizing means for synthesizing the virtual object in the image based on the first model and the CG model of the virtual object;
An information processing apparatus comprising:   The information processing apparatus according to claim 1, wherein the contact determination unit determines the contact based on an overlap between the second model and a CG model of a virtual object.   3. The information according to claim 1, wherein the image synthesizing unit synthesizes the virtual object in the image based on a front-to-back relationship between the first model and the CG model of the virtual object. Processing equipment.   In the contact determination means, the second model and the CG model of the virtual object overlap each other, and the projection area of the first model on the image and the image of the CG model of the virtual object on the image 4. The information processing apparatus according to claim 1, wherein when the projection area overlaps, the information processing apparatus determines that the real object and the virtual object are in contact with each other.   The contact determination means projects the projection area of the first model onto the image and the image of the CG model of the virtual object onto the image even if the second model and the CG model of the virtual object overlap. 5. The information processing apparatus according to claim 1, wherein when the area does not overlap, the information processing apparatus determines that the real object and the virtual object are not in contact with each other. A combination of the first model and the second model is determined based on the degree of overlap between the projection area of the first model onto the image and the projection area of the second model onto the image. It further comprises a combination judgment means,
The combination determination unit uses the second model in the processing of the contact determination unit and uses the first model in the processing of the image synthesis unit when the degree of overlap is equal to or greater than a threshold value. The information processing apparatus according to claim 1, wherein the information processing apparatus determines whether or not the information processing apparatus is in an appropriate state. Model generating means for generating a first model composed of a plane connecting contour points of the real object extracted from an image of the real object;
Model acquisition means for acquiring a second model representing the surface shape of the real object;
One model selected from the first model or the second model based on the degree of overlap between the projection area of the first model onto the image and the projection area of the second model onto the image Contact determination means for determining contact between the real object and the virtual object based on a CG model of the virtual object;
One model selected from the first model or the second model based on the degree of overlap between the projection area of the first model onto the image and the projection area of the second model onto the image Image synthesizing means for synthesizing the virtual object in the image based on the CG model of the virtual object;
An information processing apparatus comprising:   The contact determination means selects the first model when the degree of overlap is less than a threshold, and the image composition means selects the first model when the degree of overlap is less than a threshold. The information processing apparatus according to claim 7, wherein the information processing apparatus is selected.   The contact determination means selects the second model when the degree of overlap is less than a threshold value, and the image composition means selects the second model when the degree of overlap is less than a threshold value. The information processing apparatus according to claim 7, wherein the information processing apparatus is selected.   The information processing apparatus according to claim 7, further comprising a warning unit that outputs a warning when the degree of overlap is less than a threshold value.   The said image synthetic | combination means dynamically changes the display mode of the said virtual object on the synthetic | combination image based on the determination result of the said contact determination means. The information processing apparatus described in 1. A method for controlling an information processing apparatus,
A model generation step of generating a first model composed of surfaces connecting contour points of the real object extracted from an image obtained by photographing the real object;
A model acquisition step of acquiring a second model representing the surface shape of the real object;
A contact determination step of determining contact between the real object and the virtual object based on the second model and a CG model of the virtual object;
An image synthesis step of synthesizing the virtual object in the image based on the first model and the CG model of the virtual object;
A method for controlling an information processing apparatus, comprising:   The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 11.

Images