JP2018077732A - Image processing apparatus, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of generating alternative information on an object in a state where a user holds the object at a part of his/her body, even when it is failed to acquire the position and orientation of the body part or a model superposed on the body part or the position and orientation of the object.SOLUTION: The technique includes generating a first shape model of a body part of a user based on a measurement result of the body part, acquiring the position and orientation of an object to be gripped by the body part, and generating a second shape model of the object. The technique includes generating an image of a virtual space including the first shape model and the second shape model having the acquired position and orientation. When the result of the processing for acquiring the position and orientation of the object satisfies a predetermined condition, the technique further includes estimating the position and orientation of the object based on the position and orientation of the body part and the position and orientation of the object acquired when the body part has previously gripped the object, and includes generating the second shape model based on the estimated result.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a mixed reality presentation technique.

  In recent years, there has been a demand for shortening the evaluation period using prototypes and reducing costs in the design and manufacturing fields. A mixed reality (MR) system has been introduced for virtual evaluation of ease of assembly and maintainability using design (shape / design) data created by a CAD (Computer Aided Design) system. For example, when evaluating ease of assembly, a virtual object (hereinafter referred to as an operation CG model) is grasped and moved by hand, and contact with another virtual object (hereinafter referred to as a non-operation CG model) is performed in the virtual space. A simulation is assumed. At this time, in order to simulate the contact between the hand and the operation CG model, it is necessary to model the hand and superimpose it on the hand. Similarly, to simulate the contact between the operation CG model and the non-operation CG model, It is necessary to obtain the position and orientation of the CG model.

  In order to model the hand, for example, Leap Motion manufactured by Leap Motion may be used. Leap Motion can measure the position and orientation including the fingers. In Leap Motion, a hand region can be detected from a built-in stereo camera, and a three-dimensional polygon model imitating the shape of a hand (hereinafter referred to as a hand model) can be output in real time. In addition to Leap Motion, the position and orientation of hands and fingers can be estimated from a depth sensor such as Kinect of Microsoft (registered trademark) (Non-Patent Document 1). In the technique described in Non-Patent Document 1, a three-dimensional polygon in which hand and finger postures are estimated by repeatedly calculating from an initial position based on a depth image of a hand shape obtained by a depth sensor and optimizing costs. A model is being generated. Hereinafter, Leap Motion and a depth sensor that can output a hand model are collectively referred to as a sensor.

  In order to obtain the position and orientation of an object, a known geometric pattern (hereinafter referred to as a marker) is pasted on the object, the marker image is acquired by the camera, the geometric change of the marker is calculated, and the relative position between the marker and the camera The posture may be calculated. Here, the camera is assumed to be a stereo camera mounted on a video see-through head mounted display (hereinafter abbreviated as HMD) which is a display device for presenting mixed reality. By synchronizing the position and orientation of the virtual camera when drawing the CG model with the position and orientation of the stereo camera installed in the HMD, it can be displayed as if the hand model and CG model exist in the real space. it can.

JP 2008-40913 A

Qian, C., Sun, X., Wei, Y., Tang, X., Sun, J .: Realtime and Robust Hand Tracking from Depth. In CVPR 2014. Pp. 1106-1113. Colum¬bus, USA (2014 ) D. Holz, S. Ullrich, M. Wolter, and T. Kuhlen .: Multi-contact grasp interaction for virtual environments.Journal of Virtual Reality and Broadcasting, 5 (7), 2008. Moehring, M., Froehlich, B .: Pseudo-Physical Interaction with a Virtual Car Interior in Immersive Environments. In Proceedings of IPT / EGVG-Workshop. 2005. Aalborg

  However, when the object is held by the hand, the marker attached to the object cannot be seen because the hand is in the way, and the position and orientation of the object may not be calculated correctly. Similarly, there are cases where the object is in the way and the hand cannot be seen, and the position / posture shape of the hand cannot be calculated correctly.

  Patent Document 1 proposes a method for correcting the position and orientation calculated from the marker assuming that the shape of the object is known when detecting the position and orientation of the object to which the marker is attached. It must be assumed that it is already known. Further, Patent Document 1 is a method for correcting a position and orientation calculated using a marker, and does not describe a coping method when the position and orientation cannot be calculated using a marker.

  The present invention has been made in view of such problems, and in a state where the user is holding an object at his / her site, the model superimposed on the site, the position and orientation of the site, the position and orientation of the object, Provided is a technology capable of generating alternative information even if either of them cannot be acquired.

  According to one aspect of the present invention, a first generation unit that generates a first shape model of a part based on a measurement result for a part of a user, and acquisition of acquiring a position and orientation of an object to be grasped by the part Virtual space including means, second generation means for generating a second shape model of the object, the first shape model, and the second shape model having the position and orientation acquired by the acquisition means A second generation unit configured to generate an image of the object, wherein the second generation unit includes a past process when the result of the process of acquiring the position and orientation of the object by the acquisition unit satisfies a predetermined condition. The position and orientation of the object are estimated based on the position and orientation of the part when the part grips the object and the position and orientation of the object, and the second shape is based on the estimated result. Specially for generating models To.

  According to the configuration of the present invention, in a state where the user is holding an object at his / her part, any one of the model superimposed on the part, the position / posture of the part, and the position / posture of the object cannot be acquired. However, alternative information can be generated.

The block diagram which shows the function structural example of a mixed reality space presentation system. The figure explaining a hand model and operation CG model. 5 is a flowchart of processing performed by the image processing apparatus 190. 5 is a flowchart of processing performed by the image processing apparatus 190. The block diagram which shows the hardware structural example of a computer apparatus.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
In the present embodiment, an example of an image processing apparatus having the following configuration will be described. The image processing apparatus generates a first shape model of the part based on a measurement result for the part of the user (first generation), acquires a position and orientation of the target object to be grasped by the part, A second shape model of the object is generated (second generation). Then, the image processing apparatus generates (third generation) an image of a virtual space including the first shape model and the second shape model having the acquired position and orientation. Here, in the second generation, when the result of the process of acquiring the position and orientation of the object satisfies a predetermined condition, the position and orientation of the part when the part has gripped the object in the past and The position and orientation of the object are estimated based on the position and orientation of the object. In the second generation, a second shape model is further generated based on the estimated result.

  In the mixed reality space presentation system according to the present embodiment, as shown in FIG. 2, the position and orientation of the user's hand 201 wearing a head-mounted display device such as an HMD on his / her head is used. A hand model 211 that is a three-dimensional virtual object imitating a shape is generated and arranged. Furthermore, the mixed reality space presentation system arranges an operation CG model 212 that is a three-dimensional virtual object imitating the shape of the driver 202 with the position and orientation of the driver 202 as a real object. At this time, since the operation CG model 212 is arranged according to the position and orientation of the driver 202, the marker 203 is attached to the driver 202 in order to make the mixed reality space presentation system recognize the position and orientation. In FIG. 2, the marker 203 is shown as a two-dimensional barcode, but what can be used as the marker 203 is not limited to this, and any index may be used. Further, the marker 203 may be provided at any location of the driver 202. The mixed reality space presentation system recognizes the marker 203 to determine the position and orientation of the driver 202, and arranges the operation CG model 212 with the obtained position and orientation. There may be a case of concealing. When such a case occurs, the mixed reality space presentation system cannot correctly recognize the marker 203, and as a result, cannot correctly recognize the position and orientation of the driver 202. As will be described later, there are other cases where the position and orientation of the driver 202 cannot be recognized correctly. If the position and orientation of the driver 202 cannot be recognized correctly, the operation CG model 212 cannot be correctly placed with the position and orientation of the driver 202.

  In the present embodiment, when a case where the position and orientation of the driver 202 cannot be correctly recognized occurs, the position and orientation relationship between the hand 201 and the driver 202 when the hand 201 has gripped the driver 202 in the past is taken into consideration. The current position and orientation of the driver 202 are obtained.

  First, a functional configuration example of the mixed reality space presentation system according to the present embodiment will be described with reference to the block diagram of FIG. The configuration illustrated in FIG. 1 is an example, and any configuration may be adopted as long as the configuration can handle the above case. As shown in FIG. 1, the mixed reality space presentation system according to the present embodiment includes a sensor 101, an HMD 151 that is an example of a head-mounted display device, an image processing device 190, and a display device 142.

  First, the sensor 101 will be described. The sensor 101 is provided to measure the position and orientation and shape of a user's hand (including a finger) wearing the HMD 151 on his / her head. As the sensor 101, for example, Kinect described in Non-Patent Document 1 may be used. The sensor 101 may be attached to the HMD 151 or may be attached to a predetermined position in the real space without being attached to the HMD 151.

  Next, the HMD 151 will be described. The display unit 111 is attached to the HMD 151 so as to be positioned in front of the eyes (right eye and left eye) of the user wearing the HMD 151 on his / her head, and displays an image output from the image processing apparatus 190. To do. The imaging unit 141 is attached to the HMD 151 so as to capture the user's line-of-sight direction from the position near the eyes (right eye and left eye) of the user wearing the HMD 151 on his / her head. Capture a moving image. The image of each frame (physical space captured image) captured by the imaging unit 141 is sequentially output to the image processing device 190.

  Next, the image processing apparatus 190 will be described. Based on the measurement result of the user's hand by the sensor 101, the hand model generation unit 102 generates a shape model that is a three-dimensional virtual object imitating the shape of the hand, that is, a hand model (hand model 211 in FIG. 2). . For example, Kinect SDK manufactured by Microsoft (registered trademark) may be used to generate the hand model by the hand model generation unit 102. That is, by extracting a three-dimensional region of the hand from the sensor 101 and matching with reference data, a hand model reflecting the shape of the hand changing in real time is output.

  The generation success determination unit 103 determines whether the hand model generation unit 102 has successfully generated the hand model. Various criteria for determining whether or not the hand model has been successfully generated can be considered.

  For example, if the difference between the hand position / posture in the current frame and the hand position / posture in a past frame (for example, one frame past the current frame) is greater than or equal to a threshold value, it is determined that the generation of the hand model has failed. To do. The difference may be a position component only difference, a posture component only difference, or both a position and orientation difference. Further, instead of the hand position / posture, the position / posture of each finger or joint may be used.

  Further, as another criterion for judgment, there is another real object on the straight line from the sensor 101 to the hand, so that sufficient measurement information (information sufficient to generate a hand model) cannot be obtained from the sensor 101. In such a case, it may be determined that the generation of the hand model has failed.

  In any case, unless the generation success determination unit 103 determines that the generation of the hand model has failed, the generation of the hand model is handled as successful.

  The position / orientation recognition unit 112 recognizes a marker (marker 203 in FIG. 2) that appears in the captured image output from the imaging unit 141, and determines the position / orientation (relative to the imaging unit 141) of the marker (see FIG. 2). It is obtained (recognized) as the position and orientation of the driver 202). If the position and orientation of the driver can be acquired, the acquisition method is not limited to a method using an image captured by the imaging unit 141. For example, a magnetic sensor or an optical sensor is attached to the driver, and the sensor The position and orientation of the driver may be obtained from the measurement result obtained by the above.

  The recognition success determination unit 113 determines whether the position / posture recognition unit 112 has successfully recognized the driver's position / posture. Various criteria can be considered as criteria for determining whether or not the driver's position and orientation have been successfully recognized.

  For example, when the marker cannot be detected from the captured image output from the imaging unit 141, it is determined that the recognition of the position and orientation of the driver has failed. Also, if the difference between the driver position and orientation in the current frame and the driver position and orientation in the past frame (for example, one frame past the current frame) is equal to or greater than a threshold value, the driver position and orientation recognition fails. Judge that The difference may be a position component only difference, a posture component only difference, or both a position and orientation difference.

  The grip determination unit 121 determines whether or not the user's hand is holding the driver. A method for determining whether or not the user's hand is holding the driver can be implemented by a well-known technique. For example, the method described in Non-Patent Document 2 or Non-Patent Document 3 is used. It may be determined whether or not the driver is holding the driver. For example, if the position and orientation relationship between the user's hand and the driver is a prescribed position and orientation relationship (position and orientation relationship between the hand and the driver when the user's hand is holding the driver), the user's hand Judge that it is gripping.

  The information storage unit 122 functions as a memory for registering various types of information used by the estimation unit 123 described later to estimate the position and orientation of the driver in the current frame.

  The estimation unit 123 estimates the position and orientation of the driver using information registered in the information storage unit 122 when the hand model is successfully generated but the recognition of the position and orientation of the driver fails. Details of the operation of the estimation unit 123 will be described later.

  First, the virtual space generation unit 132 is arranged with the hand model arranged with the position and orientation of the hand according to the measurement result by the sensor 101, with the position and orientation of the driver recognized by the position and orientation recognition unit 112 or estimated by the estimation unit 123. A virtual space including the operation CG model is constructed. Then, the virtual space generation unit 132 generates an image obtained by viewing the constructed virtual space from a viewpoint having the position and orientation of the imaging unit 141 as a virtual space image. The position and orientation of the imaging unit 141 may be obtained using, for example, a natural feature in an image captured by the imaging unit 141, or a magnetic sensor or an optical sensor is attached to the imaging unit 141 according to a measurement result by the sensor. You may ask.

  The image generation unit 133 generates a combined image obtained by combining the captured image output from the imaging unit 141 and the virtual space image generated by the virtual space generation unit 132. The image output unit 134 outputs the composite image generated by the image generation unit 133 to the HMD 151 (display unit 111) and the display device 142. Note that the output destination of the composite image by the image output unit 134 is not limited to the HMD 151 and the display device 142.

  Next, processing performed by the image processing apparatus 190 to generate and output a composite image for one frame will be described with reference to FIG. 3 showing a flowchart of the processing. That is, the image processing apparatus 190 performs processing according to the flowchart of FIG. 3 for the captured image of each frame output from the imaging unit 141.

  In step S <b> 301, the hand model generation unit 102 generates a hand model based on the measurement result of the user's hand by the sensor 101. In step S <b> 302, the generation success determination unit 103 determines whether the hand model generation by the hand model generation unit 102 is successful. As a result of this determination, if it is determined that the generation of the hand model is successful, the process proceeds to step S303, and if it is determined that the generation of the hand model has failed, the process proceeds to step S311.

  In step S303, the generation success determination unit 103 associates the position and orientation of the hand indicated by the measurement result of the sensor 101 (the position and orientation of the hand model) with the frame information (frame number, imaging date and time) specifying the current frame. Is registered in the information storage unit 122.

  In step S311, the position / orientation recognition unit 112 recognizes a marker in the captured image output from the imaging unit 141, and determines the position / orientation of the marker based on the recognition result based on the recognition result (operation CG). (Recognize) as model position and orientation).

  In step S312, the recognition success determination unit 113 determines whether or not the position / posture recognition unit 112 has successfully recognized the driver's position / posture. As a result of this determination, if it is determined that the driver's position and orientation have been successfully recognized, the process proceeds to step S313. On the other hand, if it is determined that the recognition of the driver's position and orientation has failed, the process proceeds to step S321.

  In step S313, the recognition success determination unit 113 associates the position and orientation of the driver recognized in step S311 (position and orientation of the operation CG model) with the frame information (frame number, imaging date and time) specifying the current frame. Register in the information storage unit 122.

  At this time, if the success condition “determination that the hand model generation is successful and that the driver's position / posture recognition is successful” is satisfied, the process goes through step S321. Proceed to step S322. On the other hand, if the success condition is not satisfied, the process proceeds to step S323 via step S321.

  In step S322, the grip determination unit 121 determines whether or not the user's hand is gripping the driver, and uses a value indicating the determination result (determination value) as frame information (frame number, imaging). The date is registered in the information storage unit 122 in association with the date and time. The determination value has, for example, a value “1” when the user's hand is holding the driver, and a value “0” when the user's hand is not holding the driver.

  When the process proceeds to step S323, if the hand model generation unit 102 has successfully generated the hand model, the process proceeds to step S324. If the hand model generation has failed, the process proceeds to step S330. Proceed to

  In step S324, the estimation unit 123 first refers to the information registered in the information storage unit 122, and registers the frame registered in association with the determination value indicating that “the user's hand is holding the driver”. Of the information, frame information representing the latest frame is specified. Then, the estimation unit 123 searches for the position and orientation of the hand and the position and orientation of the driver registered in the information storage unit 122 in association with the identified frame information. If this search is successful, the process proceeds to step S325, and if this search fails, the process proceeds to step S330. That is, when the search is successful, the estimation unit 123 has acquired the position and orientation of the hand and the position and orientation of the driver when the hand has recently held the driver.

  In step S325, the estimation unit 123 calculates the relative position / posture Δ of the driver with respect to the hand, using the hand position / posture and the driver's position / posture acquired by the search. Then, the estimation unit 123 calculates a position and orientation in which the relative position and orientation relative to the position and orientation of the hand model in the current frame become the obtained relative position and orientation Δ as the position and orientation of the operation CG model in the current frame.

  In step S330, the virtual space generation unit 132 generates, as a virtual space image, an image obtained by viewing the virtual space in which the hand model and the operation CG model are arranged as described above from the viewpoint having the position and orientation of the imaging unit 141. Note that the hand model and the operation CG model cannot be arranged when the generation of the hand model fails or when the recognition of the position and orientation of the operation CG model fails. As the virtual space image in such a case, for example, the virtual space image in the immediately preceding frame may be used as the virtual space image of the current frame.

  In step S340, the image generation unit 133 generates a composite image obtained by combining the captured image output from the imaging unit 141 and the virtual space image generated by the virtual space generation unit 132. The image output unit 134 outputs the composite image generated by the image generation unit 133 to the HMD 151 (display unit 111) and the display device 142.

<Modification 1>
In the first embodiment, the hand position and orientation are registered in the information storage unit 122 every time the hand model is successfully generated, and the driver position and orientation are registered in the information storage unit 122 every time the driver position and orientation are successfully recognized. I was registered. However, what is actually used among the positions and orientations registered in the information storage unit 122 is the position and orientation of the hand when the hand has recently gripped the driver and the position and orientation of the driver. However, the position and orientation of the hand and the position and orientation of the driver may be registered only when it is determined that the user's hand is holding the driver. Further, the position and orientation to be registered may be only in the latest frame.

  In addition to this, various conditions for registering the position and orientation of the hand and the position and orientation of the driver are conceivable. For example, registration may be performed when the amount of change in the relative position between the hand and the driver is greater than or equal to a threshold, or when the amount of change in the shape of the hand model is greater than or equal to the threshold. The shape change of the hand model can be obtained from the position and orientation changes of the fingers and joints of the hand model.

[Second Embodiment]
The first embodiment deals with a case where the hand model is successfully generated, but the driver's position and orientation recognition fails. This embodiment deals with a case where the hand model generation has failed, although the driver's position and orientation have been recognized. In each of the following embodiments including this embodiment, differences from the first embodiment will be described mainly, and unless otherwise noted, the same as the first embodiment.

  In the present embodiment, the image processing apparatus 190 performs processing according to the flowchart of FIG. 4 instead of performing processing according to the flowchart of FIG. 3. In FIG. 4, the same processing steps as those shown in FIG. 3 are denoted by the same step numbers, and description thereof will be omitted.

  In step S400, the generation success determination unit 103 associates the hand model generated in step S301 with the position and orientation of the hand model (hand) and the frame information (frame number, imaging date and time) specifying the current frame. Is registered in the information storage unit 122.

  If the success condition “determined that the hand model has been successfully generated and the driver's position / posture has been successfully recognized” is not satisfied, the process proceeds to step S401 via step S321. move on.

  When the process proceeds to step S401, if the position / orientation recognition unit 112 has successfully recognized the position / orientation of the driver, the process proceeds to step S324. If the recognition of the driver's position / orientation has failed, The process proceeds to step S330. If the search in step S324 is successful, the process proceeds to step S403. If this search fails, the process proceeds to step S330.

  In step S403, the estimation unit 123 calculates a relative position / posture Δ of the hand with respect to the driver using the hand position / posture and the driver's position / posture acquired by the search. Then, the estimation unit 123 calculates, as the position and orientation of the hand model in the current frame, the position and orientation in which the relative position and orientation with respect to the position and orientation of the operation CG model in the current frame become the obtained relative position and orientation Δ. If the hand model itself cannot be generated, the hand model registered in the information storage unit 122 is read in association with the frame information specified in step S324, and the read hand model is used as the hand model in the current frame. May be used. In step S400, instead of the hand model, the bone of the hand model may be registered from the viewpoint of memory efficiency in the information storage unit 122, or a two-dimensional image viewed from the viewpoint of the hand model may be registered. good.

[Third Embodiment]
In the first embodiment, when generation of a hand model fails or when recognition of a driver's position and orientation fails, the hand model of the current frame, its position and orientation, and the position and orientation of the operation CG model of the current frame are obtained. I asked for it. However, even if it is determined that the generation of the hand model and the recognition of the position and orientation of the driver have been successful, the reliability of the generated hand model, the accuracy of the position and orientation, and the accuracy of the position and orientation of the operation CG model are considered. May be low. However, instead of controlling the information registration in the information storage unit 122 according to whether the hand model is successfully generated / failed, or the driver's position / posture is recognized / failed, the reliability is considered. Then, it may be controlled.

  For example, if the data amount of the hand obtained from the sensor 101 is less than the prescribed data amount (data amount originally obtained from the sensor 101), the reliability is set to 0, and the process proceeds from step S302 to step S311. On the other hand, if the data amount of the hand obtained from the sensor 101 is equal to or greater than the prescribed data amount, the reliability is set to 1, and the process proceeds from step S302 to step S303.

  For example, it may be determined that the greater the number of markers in the image captured by the imaging unit 141, the higher the reliability of recognition of the position and orientation of the driver. If the reliability is greater than or equal to the threshold, the process proceeds from step S312 to step S313. If the reliability is less than the threshold, the process proceeds from step S312 to step S321.

[Fourth Embodiment]
Although all the functional units included in the image processing apparatus 190 illustrated in FIG. 1 may be configured by hardware, the information storage unit 122 is implemented by a memory, and the other functional units are configured by software (computer It may be implemented by a program). In such a case, a computer apparatus having the information storage unit 122 as a memory and capable of executing the computer program can be applied to the image processing apparatus 190 described above. A hardware configuration example of a computer apparatus applicable to the image processing apparatus 190 will be described with reference to the block diagram of FIG.

  CPU 510 executes processing using computer programs and data stored in ROM 520 and RAM 530. As a result, the CPU 510 controls the operation of the entire computer apparatus and executes or controls each process described above as being performed by the image processing apparatus 190.

  The ROM 520 stores setting data, a boot program, and the like of the computer apparatus that do not require rewriting. The RAM 530 has an area for storing data received from the outside via the I / F (interface) 540, computer programs loaded from the external storage device 560, and data. Further, the RAM 530 has a work area used when the CPU 510 executes various processes. As described above, the RAM 530 can provide various areas as appropriate.

  The I / F 540 is for connecting the sensor 101, the HMD 151, and the display device 142 described above. The hand measurement result output from the sensor 101 and the captured image of each frame output from the HMD 151 are input to the RAM 530 and the external storage device 560 via the I / F 540. Further, information such as a composite image generated by the computer apparatus is output to the HMD 151 (display unit 111) and the display apparatus 142 via the I / F 540.

  The external storage device 560 is a large-capacity information storage device represented by a hard disk drive device. The external storage device 560 stores an OS (Operating System) and computer programs and data for causing the CPU 510 to execute or control each of the above-described processes performed by the image processing apparatus 190. This computer program includes a computer program for causing the CPU 510 to realize the functions of the functional units excluding the information storage unit 122 among the functional units shown as the image processing apparatus 190 in FIG. The data stored in the external storage device 560 includes hand model and operation CG model data and information described as known information in the above description.

  Computer programs and data stored in the external storage device 560 are appropriately loaded into the RAM 530 under the control of the CPU 510 and are processed by the CPU 510. The CPU 510, ROM 520, RAM 530, I / F 540, and external storage device 560 are all connected to the bus 500.

[Fifth Embodiment]
In each of the above embodiments, the HMD 151 has been described as using a video see-through method, but an optical see-through method may be employed. In that case, the image captured by the imaging unit 141 is used for imaging a marker and calculating the position and orientation of the HMD 151, and is not used for display. In addition, when the image processing device 190 generates a virtual space image, the image processing device 190 outputs the virtual space image to the display device 142 and the HMD 151. Moreover, you may use a smart phone and a tablet terminal device with a camera instead of HMD151.

  In each of the embodiments and modifications described above, the user's part that holds the object is described as a hand, and the target object that is to be held is a driver. However, the user's part that holds an object and the target that is to be held are described. Each thing is not limited to a hand or a driver.

  Further, when there is a contact between the operation CG model and a non-operation CG model that is a virtual object other than the hand model and the operation CG model, the display unit 111 and the display device 142 may be notified of the contact. good. For example, a location (polygon or the like) where the operation CG model is in contact with the non-operation CG model may be explicitly displayed. Also, some or all of the embodiments and modifications described above may be combined as appropriate.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  102: Hand model generation unit 103: Generation success determination unit 112: Position and orientation recognition unit 113: Recognition success determination unit 121: Grasping determination unit 123: Estimation unit 132: Virtual space generation unit 133: Image generation unit

Claims (12)

First generation means for generating a first shape model of the part based on a measurement result for the part of the user;
Acquisition means for acquiring the position and orientation of an object to be grasped by the part;
Second generating means for generating a second shape model of the object;
A third generation unit configured to generate an image of a virtual space including the first shape model and the second shape model having the position and orientation acquired by the acquisition unit;
The second generation means includes
When the result of the process of acquiring the position and orientation of the object by the acquisition unit satisfies a predetermined condition, the position and orientation of the part and the position of the object when the part has gripped the object in the past An image processing apparatus that estimates a position and orientation of the object based on an orientation, and generates the second shape model based on the estimated result.   When the acquisition of the position and orientation of the object by the acquisition unit fails as the predetermined condition, the second generation unit determines the position of the part when the part has gripped the object in the past. The position and orientation of the object are estimated based on the orientation and the position and orientation of the object, and the second shape model is generated based on the estimated result. Image processing device.   When the second generation means cannot detect an index provided on the target object for recognizing the position and orientation of the target object as the predetermined condition, the part is determined to be the target in the past. Estimating the position and orientation of the object based on the position and orientation of the part when the object is gripped and the position and orientation of the object, and generating the second shape model based on the estimated result The image processing apparatus according to claim 1, wherein:   If the position and / or orientation recognized by the object has changed from the position and / or orientation recognized in the past by a threshold value or more as the predetermined condition, the second generation means determines that the part has Estimating the position and orientation of the object based on the position and orientation of the part and the position and orientation of the object when the object is gripped, and generating the second shape model based on the estimated result The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   When the acquisition of the position and orientation of the object by the acquisition unit fails as the predetermined condition, the second generation unit determines the position of the part when the part has gripped the object in the past. A relative position / posture relationship between a posture and a position / posture of the object is obtained, and the target is based on the obtained relative position / posture relationship and the current position / posture of the first shape model. The image processing apparatus according to claim 1, wherein a position and orientation of an object is estimated, and the second shape model is generated based on the estimated result. First generation means for generating a first shape model of the part based on the measurement result of the position and orientation of the part of the user;
Second generation means for generating a second shape model of an object to be grasped by the part;
Third generation means for generating an image of a virtual space including the first shape model and the second shape model,
The first generation means includes:
When the result of the process of acquiring the position and orientation of the part satisfies a predetermined condition, based on the position and orientation of the part and the position and orientation of the object when the part has gripped the object in the past An image processing apparatus that estimates a position and orientation of the part and generates the first shape model based on the estimated position and orientation.   In the case where acquisition of the position and orientation of the part fails as the predetermined condition, the first generation unit determines the position and orientation of the part when the part grips the target in the past and the target The image processing apparatus according to claim 6, wherein the position and orientation of the part is estimated based on the position and orientation of the image, and the first shape model is generated based on the estimated position and orientation.   The first generation means, as the predetermined condition, when the generation of the first shape model fails, the first shape model when the part has gripped the object in the past, The image processing apparatus according to claim 6, wherein the first shape model is the current shape model. Furthermore, a means for acquiring a captured image of the real space is provided,
The image processing apparatus according to claim 1, wherein the third generation unit generates a composite image of the image in the virtual space and the captured image. An image processing method performed by an image processing apparatus,
A first generation step in which a first generation unit of the image processing apparatus generates a first shape model of the part based on a measurement result for the part of the user;
An acquisition step in which the acquisition unit of the image processing apparatus acquires the position and orientation of an object to be grasped by the part;
A second generation step in which a second generation means of the image processing apparatus generates a second shape model of the object;
A third generation step in which a third generation unit of the image processing device generates an image of a virtual space including the first shape model and the second shape model having the position and orientation acquired in the acquisition step. And
In the second generation step,
If the result of the process of acquiring the position and orientation of the object in the acquisition step satisfies a predetermined condition, the position and orientation of the part and the position of the object when the part has gripped the object in the past An image processing method characterized by estimating a position and orientation of the object based on an orientation, and generating the second shape model based on the estimated result. An image processing method performed by an image processing apparatus,
A first generation step in which a first generation unit of the image processing apparatus generates a first shape model of the part based on a measurement result of a position and orientation of the part of the user;
A second generation step in which a second generation unit of the image processing apparatus generates a second shape model of an object to be grasped by the part;
A third generation step of generating a virtual space image including the first shape model and the second shape model, wherein the third generation means of the image processing apparatus comprises:
In the first generation step,
When the result of the process of acquiring the position and orientation of the part satisfies a predetermined condition, based on the position and orientation of the part and the position and orientation of the object when the part has gripped the object in the past An image processing method, wherein the position and orientation of the part are estimated, and the first shape model is generated based on the estimated position and orientation.   The computer program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 9.

Images