JP2009025918A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for further facilitating user operation to a real object superposed with a virtual object. <P>SOLUTION: A virtual world generation part 106 generates a virtual space image for the left eye and a virtual space image for the right eye when viewing the virtual object having a position/attitude of the object that is an operation target by a hand of a user wearing an HMD (Head-Mounted Display) 131 on the head from positions/attitudes of a left camera 132 and a right camera 133. An image composition part 103 superposes the respective images on images by the left camera 132 and the right camera 133. The virtual world generation part 106 controls transparency of the virtual space image according to a distance between a position of the hand and the position of the object. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for providing mixed reality.

  2. Description of the Related Art In recent years, devices (mixed reality devices) that apply a mixed reality (MR) technology that naturally couples the real world and the virtual world naturally are not proposed. These mixed reality devices present mixed reality to the user of the device as follows. That is, a virtual world image drawn by computer graphics (CG) is synthesized with a real world image taken by an imaging device such as a camera. Then, the synthesized image by the synthesis is displayed on a display device such as a head-mounted display (HMD).

  In order to enhance the mixed reality, these mixed reality devices acquire the viewpoint position and orientation of the user of this device in real time, follow the change of the real world image, generate the virtual world image, It is necessary to display in real time on a display device such as an HMD for the user. In the mixed reality device, the viewpoint position / orientation of the user measured by the sensor device is set as a virtual viewpoint position / orientation in the virtual world, and based on this setting, an image of the virtual world is drawn by CG, and the real world Composite with the image.

  In addition, since the HMD presents mixed reality, the visual field of the user of the mixed reality device includes display by the HMD, and the display includes a region for drawing CG.

  With these technologies, the user of the mixed reality apparatus can observe an image as if a virtual object exists in the real world.

If such a technique is used, for example, a desired virtual object is superimposed on a real object, and the user grasps and operates the real object, as if operating the object indicated by the virtual object. Can be given to the user (Patent Document 1).
JP 2006-146588 A

  However, when a user tries to grasp a real object on which a virtual object is superimposed, there is a problem that it is difficult to understand the distance between his / her hand and the real object. In addition, there is a possibility that a part of the user's own body is concealed by the virtual object.

  As described above, it is more difficult for the user to grab the real object on which the virtual object is superimposed than to act on the real object on which the virtual object is not superimposed.

  The present invention has been made in view of the above problems, and an object thereof is to provide a technique for facilitating a user operation on a real object on which a virtual object is superimposed.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, a first acquisition unit that acquires a position of a body part of a user wearing a head-mounted display device having an imaging device and a display device on the head;
Second acquisition means for acquiring the position and orientation of the imaging device;
Third acquisition means for acquiring a captured image by the imaging device;
Fourth acquisition means for acquiring a position and orientation of an object to be operated by the part;
Generating means for generating, as a virtual space image, an image when the virtual object having the position and orientation acquired by the fourth acquisition means is viewed from the position and orientation acquired by the second acquisition means;
An output unit that superimposes the virtual space image generated by the generation unit on the captured image acquired by the third acquisition unit and outputs the image to the display device;
The generating means includes
The transparency of the virtual space image is controlled according to the distance between the position acquired by the first acquisition unit and the position acquired by the fourth acquisition unit.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, a first acquisition step of acquiring a position of a body part of a user wearing a head-mounted display device having an imaging device and a display device on the head;
A second acquisition step of acquiring the position and orientation of the imaging device;
A third acquisition step of acquiring a captured image by the imaging device;
A fourth acquisition step of acquiring the position and orientation of the object to be operated by the part;
A generation step of generating an image when the virtual object having the position and orientation acquired in the fourth acquisition step is viewed from the position and orientation acquired in the second acquisition step as a virtual space image;
An output step of superimposing the virtual space image generated in the generation step on the captured image acquired in the third acquisition step and outputting it to the display device;
In the generating step,
The transparency of the virtual space image is controlled according to the distance between the position acquired in the first acquisition step and the position acquired in the fourth acquisition step.

  According to the configuration of the present invention, a user operation on a real object on which a virtual object is superimposed can be made easier.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  The outline of each embodiment described below is as follows.

  That is, the position of the body part of the user wearing the head-mounted display device having the imaging device and the display device on the head is acquired (first acquisition). Further, the position and orientation of the imaging device are acquired (second acquisition). Also, a captured image by the imaging device is acquired (third acquisition). Further, the position and orientation of the object to be operated by the part are acquired (fourth acquisition). Then, an image when the virtual object having the position and orientation acquired by the fourth acquisition is viewed from the position and orientation acquired by the second acquisition is generated as a virtual space image. Then, the generated virtual space image is superimposed on the captured image acquired by the third acquisition and output to the display device.

  At that time, the transparency of the virtual space image is controlled according to the distance between the position acquired by the first acquisition and the position acquired by the fourth acquisition.

  In the following embodiments, the “part” is referred to as the “hand” of the observer, and the “object” is referred to as “the hammer” in order to specifically describe the outline. Of course, even if the “part” and “object” are other things, the gist is the same.

[First Embodiment]
FIG. 1 is a block diagram showing a functional configuration of a system according to the present embodiment. As shown in FIG. 1, the system according to the present embodiment includes a system control unit 101 and a video see-through HMD 131 (hereinafter simply referred to as HMD 131).

  First, the HMD 131 will be described. The HMD 131 is an example of a head-mounted display device, and as is well known, is mounted on the user's head.

  The left camera 132 is a device that captures a moving image of the real space that can be seen from the left eye (viewpoint) of the user wearing the HMD 131 on the head. The captured image of each frame (the real space image for the left eye) is a left real image. It is sent to the output unit 134. The left reality image output unit 134 sends the left eye reality space image received from the left camera 132 to the system control unit 101.

  The right camera 133 is a device that captures a moving image of the real space that can be seen from the right eye (viewpoint) of the user wearing the HMD 131 on the head. The captured image of each frame (the real space image for the right eye) is a right real image. It is sent to the output unit 135. The right reality image output unit 135 sends the right eye reality space image received from the right camera 133 to the system control unit 101.

  The mixed reality image input unit 136 and the image display unit 137 will be described later.

  Next, the system control unit 101 will be described. The system control unit 101 is a device that performs processing for generating an image to be presented to a user (observer) wearing the video see-through HMD 131 on the head.

  The real image input unit 102 receives the left-eye real space image and the right-eye real space image transmitted from the left real image output unit 134 and the right real image output unit 135, respectively. The received images are sent to the image composition unit 103, the camera position / posture acquisition unit 105, the hand position / posture acquisition unit 107, and the actual position / posture acquisition unit 109, respectively.

  When acquiring the right-eye real space image, the camera position and orientation acquisition unit 105 calculates the position and orientation of the right camera 133 using the image. Similarly, when the real space for the left eye is acquired, the position and orientation of the left camera 132 are calculated using the image.

  Here, a technique for obtaining the position and orientation of the camera using an image captured by the camera is a conventional technique. That is, when a real space image including a marker or a natural feature whose arrangement position is known in the real space is acquired, the coordinate position of the marker or the natural feature in the real space image is detected. Then, by using the detected coordinate position and the corresponding arrangement position, and then using a calculation technique widely used in the field of triangulation, the position and orientation of the camera that captured the real space image can be obtained. it can. Of course, there are various techniques for calculating the position and orientation of the camera that captured the image using the image, and any of these may be used in the present embodiment.

  There are other methods for calculating the position and orientation of the camera. For example, there is a method using a sensor system. For example, consider a case where a magnetic sensor system is connected to the camera position and orientation acquisition unit 105. In this case, a transmitter that generates a magnetic field is arranged at a predetermined position in the real space, and a receiver that detects a change in the magnetic field according to its position and orientation in the magnetic field is attached to the camera. A signal indicating the result detected by the receiver is input to a device called a sensor control device, and the sensor control device obtains the position and orientation of the receiver in the sensor coordinate system based on the signal. Then, the sensor control apparatus sends the obtained position and orientation to the camera position and orientation acquisition unit 105. Thereby, the camera position and orientation acquisition unit 105 can acquire the position and orientation of the receiver in the sensor coordinate system as the position and orientation of the camera. Here, the sensor coordinate system is a coordinate system in which the position of the transmitter is the origin, and three axes orthogonal to each other at the origin are the x axis, the y axis, and the z axis.

  The camera position and orientation acquisition unit 105 may convert the acquired position and orientation into any coordinate system.

  In addition, since the technique for acquiring the position and orientation of the camera described above is well known, further description is omitted.

  The hand position / posture acquisition unit 107 acquires the position of the user's hand using the right-eye real space image, the left-eye real space, and the position / posture acquired by the camera position / posture acquisition unit 105. For example, a marker is pasted on the user's hand, and the position of the hand is obtained using a captured image of the hand including the marker. In addition, since the technique concerned is also well-known, the detailed description about this is abbreviate | omitted. Further, the position of the hand may be acquired using a sensor, and the acquisition method in that case is the same as the method for acquiring the position and orientation of the camera described above.

  The real position / orientation acquisition unit 109 uses the real space image for the right eye, the real space for the left eye, and the position / orientation acquired by the camera position / orientation acquisition unit 105 as a real object (in this case, a hammer) ) Position. For example, a marker is affixed to a hammer and the position of this hammer is obtained using a captured image of the hammer including the marker. In addition, since the technique concerned is also well-known, the detailed description about this is abbreviate | omitted. Further, the position of the hammer may be acquired using a sensor, and the acquisition method in that case is the same as the method for acquiring the position and orientation of the camera described above.

  First, the virtual world generation unit 106 arranges a virtual object to be superimposed on the hammer on the position and orientation obtained by the actual position and orientation acquisition unit 109. If there are other virtual objects to be placed in the virtual space, they are also placed. In this way, a virtual space is constructed.

  Then, a viewpoint for observing the virtual space is arranged at the position and orientation of the left camera 132 obtained by the camera position and orientation acquisition unit 105, and an image of the virtual space seen from the viewpoint is generated as a left-eye virtual space image. Since a technique for generating an image of a virtual space that can be seen from a viewpoint having a predetermined position and orientation is well known, a description thereof will be omitted. On the other hand, a viewpoint for observing the virtual space is arranged at the position and orientation of the right camera 133 obtained by the camera position and orientation acquisition unit 105, and an image of the virtual space seen from the viewpoint is generated as the right-eye virtual space image.

  The virtual world generation unit 106 sends each of the generated left-eye virtual space image and right-eye virtual space image to the image composition unit 103.

  The image composition unit 103 generates a right-eye composite image in which the right-eye virtual space image is superimposed on the right-eye real space image received from the real image input unit 102. Similarly, the image composition unit 103 generates a left-eye composite image in which the left-eye virtual space image is superimposed on the left-eye real space image received from the real image input unit 102. Then, each of the generated composite image for the right eye and composite image for the left eye is sent to the mixed reality image output unit 104.

  The mixed reality image output unit 104 sends the received right-eye composite image and left-eye composite image to the mixed reality image input unit 136 of the HMD 131.

  The mixed reality image input unit 136 sends each of the received right-eye composite image and left-eye composite image to the image display unit 137.

  The image display unit 137 includes a display unit for the right eye and a display unit for the left eye. The display unit for the right eye and the display unit for the left eye are respectively the right eye and the left eye of the user wearing the HMD 131 on the head. It is attached to the HMD 131 so as to be located in front of the eyes. Therefore, the composite image for the right eye received from the mixed reality image input unit 136 is sent to the display unit for the right eye, and the composite image for the left eye is sent to the display unit for the left eye.

  As a result, the left eye composite image is presented in front of the left eye of the user wearing the HMD 131 on the head, and the right eye composite image is presented in front of the right eye.

  The above is the basic operation of the system according to the present embodiment. As described above, the system according to the present embodiment has a virtual object to be superimposed on the hammer as the position of the user's hand approaches the hammer. Performs processing to increase the transparency of the image.

  FIG. 2 is a diagram illustrating an example of a hammer as a real object. FIG. 3 is a diagram illustrating an example of a virtual object to be superimposed on a hammer. Therefore, in the present embodiment, the image of the virtual object shown in FIG. 3 is superimposed on the hammer as the real object shown in FIG. In this embodiment, a large virtual object as shown in FIG. 4 is arranged in the virtual space. FIG. 4 is a diagram illustrating an example of a large virtual object arranged in the virtual space.

  In the following description, it is assumed that the user holds a hammer as a real object in his hand and measures the interaction with the virtual object shown in FIG. Of course, if the user looks at his / her hand through the HMD 131, the image of the virtual object shown in FIG. 3 can be seen on the hammer shown in FIG.

  FIG. 5 is a diagram illustrating a state where a third party other than the user observes a state in which the user 501 wearing the HMD 131 goes to pick up the hammer 500.

  FIG. 6 is a diagram illustrating a state where a third person who can observe the same space observes the state where the user wearing the HMD 131 goes to pick up the hammer. That is, the user 501 can see the virtual object 601 superimposed on the metal hammer and the large virtual object 602. FIG. 7 is a diagram illustrating a state of the space that can be seen from the viewpoint of the user 501.

  Here, even if the user 501 tries to grasp the hammer 500 with his hand, even if he extends his hand to the position of the virtual object 601, it is difficult to understand the perspective, so even if he feels that his hand has reached the virtual object 601, As shown in FIG. 8, the hand 801 may still be located in front of the hammer 500.

  In the present embodiment, in order to solve the problem, the transparency of the virtual object 601 to be superimposed on the hammer 500 is increased as the distance between the hammer 500 and the hand 801 becomes smaller. Further, as the distance between the hammer 500 and the hand 801 becomes smaller, the virtual space image drawn around the hammer 500 is masked so that only the real space can be seen around the hammer 500. This makes it easy for the user 501 to grasp the actual position of his / her hand 801 and the position of the hammer 500 and to grasp the distance between the positions.

  FIG. 9 is a diagram illustrating an image to be presented to the user 501 when the hand 801 is brought close to the position of the hammer 500.

  In addition, a state in which the hand 801 of the user 501 holds the hammer 500 (for example, a state in which the position of the hand 801 and the position of the hammer 500 are equal to or less than a predetermined threshold) is a predetermined time (for example, 2 If more than (seconds) have elapsed, the following processing is performed. That is, the transparency of the image of the virtual object 601 superimposed on the hammer 500 is restored (transparency is made opaque), and the mask process is also canceled. Thereby, after the user 501 grasps the hammer 500 with his hand 801, the user 501 can interact with the large virtual object 602 using the hammer 500.

  FIG. 10 is a flowchart of a series of processing (image processing) performed by the system control unit 101 for sending a composite image of a real space image and a virtual space image to the HMD 131.

  In step S1001, the real image input unit 102 receives the left-eye real space image and the right-eye real space image transmitted from the left real image output unit 134 and the right real image output unit 135, respectively. The received images are sent to the image composition unit 103, the camera position / posture acquisition unit 105, the hand position / posture acquisition unit 107, and the actual position / posture acquisition unit 109, respectively.

  In step S <b> 1002, when the camera position and orientation acquisition unit 105 acquires the right-eye real space image, the camera position and orientation acquisition unit 105 calculates the position and orientation of the right camera 133 using the image. Similarly, when the real space for the left eye is acquired, the position and orientation of the left camera 132 are calculated using the image.

  In step S1003, the hand position / posture acquisition unit 107 acquires the position of the user's hand as described above. Further, the actual position / posture acquisition unit 109 acquires the position of the hammer as described above.

  In step S1004, the virtual world generation unit 106 obtains a distance D between the hand position acquired in step S1003 and the position of the hammer. If the distance D is less than or equal to the threshold θ1, the process proceeds to step S1009 via step S1005, and if greater than the threshold θ1, the process proceeds to step S1006 via step S1005.

  In step S <b> 1006, the virtual world generation unit 106 first arranges a virtual object to be superimposed on the hammer on the position and orientation obtained by the real position and orientation acquisition unit 109. If there are other virtual objects to be placed in the virtual space, they are also placed. In this way, a virtual space is constructed.

  Then, a viewpoint for observing the virtual space is arranged at the position and orientation of the left camera 132 obtained by the camera position and orientation acquisition unit 105, and an image of the virtual space seen from the viewpoint is generated as a left-eye virtual space image. On the other hand, a viewpoint for observing the virtual space is arranged at the position and orientation of the right camera 133 obtained by the camera position and orientation acquisition unit 105, and an image of the virtual space seen from the viewpoint is generated as the right-eye virtual space image. The virtual world generation unit 106 sends each of the generated left-eye virtual space image and right-eye virtual space image to the image composition unit 103.

  In step S <b> 1007, the image composition unit 103 generates a right-eye composite image in which the right-eye virtual space image is superimposed on the right-eye real space image received from the real image input unit 102. Similarly, the image composition unit 103 generates a left-eye composite image in which the left-eye virtual space image is superimposed on the left-eye real space image received from the real image input unit 102. Then, each of the generated composite image for the right eye and composite image for the left eye is sent to the mixed reality image output unit 104.

  In step S <b> 1008, the mixed reality image output unit 104 sends the received right-eye composite image and left-eye composite image to the mixed reality image input unit 136 included in the HMD 131.

  On the other hand, in step S1009, the virtual world generation unit 106 increases the transparency of the “virtual object to be superimposed on the hammer” to be generated. Actually, the transparency is increased by increasing the alpha value of each pixel constituting the virtual object. The degree to which the transparency is increased with respect to the distance D is not particularly limited, but the relationship between the distance D and the transparency is such that the transparency increases as the distance D decreases and decreases as the distance D increases. I just need it.

  Next, if the distance D is equal to or smaller than the threshold θ2 (<θ1), the process proceeds to step S1011 via step S1010, and if greater than the threshold θ2, the process proceeds to step S1006 via step S1010. At this time, in the case where a “virtual object to be superimposed on the hammer” is generated in step S1006, the transparency is set in step S1009.

  In step S1011, the mask area creating unit 108 uses the real space image for the image compositing unit 103 instead of compositing the surroundings of the “virtual object to be superimposed on the hammer” in the compositing process performed in step S1007. To instruct. That is, as shown in FIG. 9, nothing is superimposed on the real space image around the drawing area of the “virtual object to be superimposed on the hammer” on the virtual space image.

  Next, when the condition that the distance D is equal to or less than the threshold value θ3 (<θ2) and the distance D is equal to or less than the threshold value θ3 (<θ2) continuously elapses for a predetermined time is satisfied. Advances the process to step S1013 via step S1012. On the other hand, if the condition is not satisfied, the process proceeds to step S1006 via step S1012.

  Here, the condition is basically for determining whether or not the user is holding a hammer as a real object with his / her hand. Therefore, the threshold θ3 used here is preferably a value close to the distance between the hand and the real hammer as a real object when the user holds the hammer as a real object.

  When the process proceeds from step S1012 to step S1006, the virtual space image is generated as described above in step S1006, and the composition process is performed as described above in step S1007. However, in step S1007, as described above, when superimposing the virtual space image on the real space image, the image composition unit 103 uses the real space image as it is around the “virtual object to be superimposed on the hammer”. Thereby, for example, as shown in FIG. 9, it is possible to obtain a composite image in which images of other virtual objects are not superimposed around the “gold bar as a real object”.

  On the other hand, in step S1013, since the user has a hammer as a real object in his hand, a virtual object may be superimposed on the hammer as a real object. Restore the transparency of the virtual object image. Further, the virtual world generation unit 106 causes the mask area creation unit 108 to cancel issuing a mask instruction.

  In this embodiment, as described first, “hand” is used as the user's part, but other than “hand”, for example, “foot” may be used.

[Second Embodiment]
In the first embodiment, the description has been made assuming that all the units constituting the system control unit 101 illustrated in FIG. 1 are implicitly configured by hardware. However, some of them may be configured with software. In this case, the system control unit 101 may be configured by a computer such as a PC (personal computer) that can execute the software.

  FIG. 11 is a block diagram illustrating a hardware configuration example of a computer applicable to the system control unit 101.

  A CPU 1101 controls the entire computer using programs and data stored in the RAM 1102 and the ROM 1103, and executes the above-described processes performed by the system control unit 101 to which the computer is applied.

  Reference numeral 1102 denotes a RAM having an area for temporarily storing programs and data loaded from the external storage device 1106, image data received from the HMD 131 via the I / F (interface) 1107, and the like. Further, the RAM 1102 also has a work area used when the CPU 1101 executes various processes. That is, the RAM 1102 can provide various areas as appropriate.

  A ROM 1103 stores computer setting data, a boot program, and the like.

  An operation unit 1104 includes a keyboard, a mouse, and the like, and various instructions can be input to the CPU 1101 by an operation of a computer operator.

  A display unit 1105 includes a CRT, a liquid crystal screen, and the like, and can display a processing result by the CPU 1101 using an image, text, or the like.

  An external storage device 1106 is a large-capacity information storage device represented by a hard disk drive device. The external storage device 1106 stores an OS (operating system) and programs and data for causing the CPU 1101 to execute the above-described processes described as being performed by the system control unit 101.

  This program is for causing the CPU 1101 to execute the functions of the camera position and orientation acquisition unit 105, the hand position and orientation acquisition unit 107, the actual position and orientation acquisition unit 109, the mask area creation unit 108, the virtual world generation unit 106, and the image composition unit 103. The program is also included.

  In addition, the data includes data of each virtual object constituting the virtual space.

  Programs and data stored in the external storage device 1106 are appropriately loaded into the RAM 1102 under the control of the CPU 1101. The CPU 1101 executes processing using the loaded program and data, so that the computer executes each processing described above as being performed by the system control unit 101.

  Reference numeral 1107 denotes an I / F, to which the HMD 131 is connected. As a result, the computer can receive image data sent from the left reality image output unit 134 and the right reality image output unit 135 of the HMD 131 via the I / F 1107. Further, the computer can transmit the composite image for the right eye and the composite image for the left eye to the HMD 131 via the I / F 1107. In other words, the I / F 1107 functions as the real image input unit 102 and the mixed reality image output unit 104 shown in FIG.

  A bus 1108 connects the above-described units.

[Other Embodiments]
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the function structure of the system which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the hammer as a real object. It is a figure which shows an example of the virtual object superimposed on a hammer. It is a figure which shows an example of the large sized virtual object arrange | positioned in virtual space. It is a figure which shows a mode when the third party other than the said user observes a mode that the user 501 with which HMD131 was mounted | worn is going to pick up the hammer 500. It is a figure which shows a mode when the third person who can observe the same space observes a mode that the user with which HMD131 was mounted | worn on the head goes to take a hammer. It is a figure which shows the mode of the space seen from a user's 501 viewpoint. It is a figure which shows the positional relationship of the hand 801 and the hammer 500. It is a figure which shows the image shown with respect to the user 501 when the hand 801 is brought close to the position of the hammer 500. 10 is a flowchart of a series of processing performed by the system control unit 101 for sending a composite image of a real space image and a virtual space image to the HMD 131. FIG. 2 is a block diagram illustrating an exemplary hardware configuration of a computer applicable to the system control unit 101.

Claims (8)

First acquisition means for acquiring a position of a body part of a user wearing a head-mounted display device having an imaging device and a display device on the head;
Second acquisition means for acquiring the position and orientation of the imaging device;
Third acquisition means for acquiring a captured image by the imaging device;
Fourth acquisition means for acquiring a position and orientation of an object to be operated by the part;
Generating means for generating, as a virtual space image, an image when the virtual object having the position and orientation acquired by the fourth acquisition means is viewed from the position and orientation acquired by the second acquisition means;
An output unit that superimposes the virtual space image generated by the generation unit on the captured image acquired by the third acquisition unit and outputs the image to the display device;
The generating means includes
An image processing apparatus, wherein the transparency of the virtual space image is controlled according to a distance between the position acquired by the first acquisition unit and the position acquired by the fourth acquisition unit.   The image processing apparatus according to claim 1, wherein the generation unit increases the transparency of the virtual space image as the distance decreases, and decreases the transparency of the virtual space image as the distance increases.   2. The generation unit according to claim 1, wherein when the time during which the distance is equal to or less than a predetermined threshold is equal to or longer than a predetermined time, the transparency of the virtual space image is made opaque. An image processing apparatus according to 1.   The image processing apparatus according to claim 1, wherein the generation unit further does not generate a virtual space image of another virtual object around the virtual space image generated by the generation unit.   The image processing apparatus according to claim 1, wherein the part is a hand or a foot. A first acquisition step of acquiring a position of a body part of a user wearing a head-mounted display device having an imaging device and a display device on the head;
A second acquisition step of acquiring the position and orientation of the imaging device;
A third acquisition step of acquiring a captured image by the imaging device;
A fourth acquisition step of acquiring the position and orientation of the object to be operated by the part;
A generation step of generating an image when the virtual object having the position and orientation acquired in the fourth acquisition step is viewed from the position and orientation acquired in the second acquisition step as a virtual space image;
An output step of superimposing the virtual space image generated in the generation step on the captured image acquired in the third acquisition step and outputting it to the display device;
In the generating step,
The transparency of the virtual space image is controlled according to the distance between the position acquired in the first acquisition step and the position acquired in the fourth acquisition step.   A program for causing a computer to execute the image processing method according to claim 6.   A computer-readable storage medium storing the program according to claim 7.

Images