JP2018063567A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, processing method and program that are capable of presenting a mixed reality without an uncomfortable feeling while reducing a contact possibility with a real object.SOLUTION: An image processing device includes: first acquiring means for acquiring a first image in which a real space is imaged; second acquiring means for acquiring a moving speed of an object imaged in the first image; determining means for determining, on the basis of the moving speed acquired by the second acquiring means, a transparency of a second image created on the basis of a positional attitude of an imaging device that has imaged the first image; and display control means for displaying the second image on a display unit in accordance with the transparency determined by the determining means.SELECTED DRAWING: None

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

In recent years, research on mixed reality (MR) technology for the purpose of seamless connection between real space and virtual space has been actively conducted. An image display device that presents mixed reality in MR technology is realized by a video see-through method or an optical see-through method.
The video see-through method is a method for displaying a composite image in which a virtual space image generated according to the position and orientation of an imaging device is superimposed and drawn on a real space image captured by an imaging device such as a video camera. . Here, the image in the virtual space includes a virtual object drawn by CG (computer graphics), character information, and the like. The optical see-through method is a method of displaying an image of a virtual space generated according to the position and orientation of the observer's viewpoint on an optical see-through display mounted on the observer's head.

  In either method, an image of the virtual space (virtual space image) is displayed while obstructing the observer's field of view, so the real object may be hidden behind the virtual space image and hit the real object There was a fear. Patent Document 1 discloses an image processing apparatus that controls the transparency of a virtual space image according to the distance between the position of an observer's hand and the position of a real object. In the technique described in Patent Document 1, for example, the transparency of the virtual space image is increased as the distance is smaller.

JP 2009-25918 A

However, in the above conventional method, since the virtual space image is made transparent according to the distance between the hand and the real object, the virtual space image is obtained when the hand is brought close to the real object at a high speed from a place away from the real object. There is a risk that the timing at which the screen becomes transparent will cause the hand to collide with a real object. In addition, even if the hand is moving slowly near the real object and the hand touches the real object, even if there is little impact, the virtual space image becomes transparent, so the observer can see the virtual space image. Will be disturbed.
Therefore, an object of the present invention is to make it possible to present a mixed reality without a sense of incongruity while reducing the possibility of contact with a real object.

  In order to solve the above-described problem, an aspect of the image processing apparatus according to the present invention includes a first acquisition unit that acquires a first image in which a real space is imaged, and the first image. A second acquisition unit configured to acquire a moving speed of the object; and a second generation unit generated based on the position and orientation of the imaging apparatus that captured the first image based on the moving speed acquired by the second acquisition unit. Determining means for determining the transparency of the second image; and display control means for displaying the second image on the display unit in accordance with the transparency determined by the determining means.

  According to the present invention, it is possible to present a mixed reality without a sense of incongruity while reducing the possibility of contact with a real object.

It is a figure which shows the structural example of the mixed reality system in embodiment of this invention. It is a figure showing the example of a marker and the example of a display of CG model. It is a hardware block diagram of an image processing apparatus. It is a flowchart which shows the flow of a process of an image processing apparatus. It is a flowchart which shows an object estimation process. It is a flowchart which shows CG rendering processing. It is an example of a display when all CG is made transparent. It is a flowchart which shows the CG rendering process of 2nd embodiment. It is a display example when only the CG around the hand is made transparent.

DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed depending on the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.
(First embodiment)
FIG. 1A is a diagram illustrating a configuration example of a mixed reality system (MR system) 10 according to an embodiment of the present invention. The MR system 10 in the present embodiment is a system for presenting an observer with a mixed reality space (MR space) in which a real space and a virtual space are fused.
The mixed reality system 10 includes imaging devices 21L and 21R, a display device 22, and an image processing device 30. In the present embodiment, the imaging devices 21L and 21R and the display device 22 are provided in a head-mounted display (HMD) that is a head-mounted display device that is mounted on the head of an observer who experiences MR space. explain. Here, the HMD is a video see-through HMD that can present a synthesized image obtained by synthesizing a real space image and a virtual space image such as computer graphics (CG) to an observer.

  The imaging devices 21L and 21R are cameras fixed to each other so that when the observer wears the HMD on the head, the real space in the gaze direction can be imaged from the viewpoint position of the observer. The camera constitutes a stereo camera. The imaging devices 21L and 21R image an object in the real space (real object) that can be seen from the viewpoint position of the observer. The imaging devices 21L and 21R image, for example, a part of the observer's body (for example, the hand 40) and the marker 50 for measuring the position and orientation of the imaging devices 21L and 21R. Then, the imaging devices 21L and 21R output the captured images to the image processing device 30, respectively. It is assumed that camera internal parameters such as the focal length and lens distortion coefficient of the stereo camera are obtained in advance by a predetermined method and are known.

  The display device 22 includes a display that displays the composite image output from the image processing device 30. The display is a CRT or a liquid crystal screen. Here, the display may be arranged corresponding to the left and right eyes of the observer. In this case, a composite image for the left eye is presented on the display corresponding to the left eye of the observer, and a composite image for the right eye is presented on the display corresponding to the right eye of the observer. The display device 22 may include an optical system for guiding an image on the display to the eyeball. In addition, the arrangement positions of the imaging devices 21L and 21R and the display device 22 may be arbitrarily set as long as the imaging device 21L and 21R can capture a visual field image of the observer and the display device 22 can present the image to the observer. Can be set.

The image processing device 30 includes an image acquisition unit 31, an image storage unit 32, an object estimation unit 33, a model shape storage unit 34, a position / orientation estimation unit 35, an image generation unit 36, an image synthesis unit 37, A display control unit 38.
The image acquisition unit 31 acquires a stereo image captured by the imaging devices 21L and 21R and outputs the stereo image to the image storage unit 32. This stereo image is used as a processing image for stereo measurement. The image storage unit 32 temporarily stores the image received from the image acquisition unit 31. For example, an image is transmitted from the image acquisition unit 31 every 1/30 seconds.

The object estimation unit 33 acquires a stereo image stored in the image storage unit 32, and estimates the three-dimensional shape and position of the hand 40 from the acquired stereo image. Further, the position information of the hand 40 is stored with time, and the moving speed and moving direction of the hand 40 are estimated based on the information. A method for estimating the three-dimensional shape and position of the hand 40 and a method for estimating the moving speed and direction of the hand 40 will be described later. The object estimation unit 33 outputs the three-dimensional shape and position of the hand 40 to the model shape storage unit 34, and outputs the moving speed and moving direction of the hand 40 to the image generation unit 36.
As shown in FIG. 1B, the model shape storage unit 34 stores the data of the three-dimensional model (CG model) of the virtual object 60 displayed on the display device 22 and the three-dimensional shape of the hand 40 received from the object estimation unit 33. Keep the data.

The position / orientation estimation unit 35 estimates the position / orientation of the imaging devices 21L and 21R. In the present embodiment, the position / orientation estimation unit 35 estimates the position / orientation of the imaging devices 21L and 21R based on the projection image of the rectangular marker 50 reflected in the image. For example, the position / orientation estimation unit 35 binarizes the image, extracts square vertices by straight line fitting, minimizes the projection error on the image by repetitive calculation of the hill-climbing method, and determines the position / orientation of the imaging devices 21L and 21R. Can be estimated.
The position / orientation estimation method by the position / orientation estimation unit 35 is not limited to the above method, and the position / orientation of the imaging devices 21L and 21R is estimated by a method using a motion capture device or a magnetic sensor. Also good.

The image generation unit 36 inputs CG model data stored in the model shape storage unit 34 and information regarding the three-dimensional shape and position of the hand 40. Further, the image generation unit 36 inputs information regarding the position and orientation of the imaging devices 21L and 21R estimated by the position and orientation estimation unit 35 and the movement speed and movement direction of the hand 40 estimated by the object estimation unit 33. Then, the image generation unit 36 generates an image (CG) of the virtual object 60 based on the input information.
Specifically, the image generation unit 36 determines the position and orientation of the virtual object 60 based on the CG model data and the positions and orientations of the imaging devices 21L and 21R. Then, the image generation unit 36 compares the context of the drawing pixel of the virtual object 60 with the hand 40 and determines whether to draw the virtual object 60. That is, when it is determined that the hand 40 is in front of the virtual object 60, the virtual object 60 is not drawn on the pixel, and the hand that is a real image in the composite image generated by the image composition unit 37 described later. The image of the virtual object 60 is processed so that 40 can be seen. In addition, the image generation unit 36 determines the transparency of the image of the virtual object 60 according to the movement speed and movement direction of the hand 40 estimated by the object estimation unit 33. Details will be described later.

The image composition unit 37 synthesizes the image of the virtual object 60 generated by the image generation unit 36 with the respective images of the imaging devices 21L and 21R stored in the image storage unit 32. The generated composite image is output to the display control unit 38. The display control unit 38 performs display control for displaying the composite image output from the image composition unit 37 on the display of the display device 22. Thereby, the observer can observe the composite image in which the front-rear relationship between the virtual object 60 and the hand 40 is correct on the display, and can feel as if the virtual object 60 actually exists at the place. it can.
In the present embodiment, the image storage unit 32 inputs the real image used in the processing of the object estimation unit 33, the position / orientation estimation unit 35, and the image generation unit 36 to the image composition unit 37. This is because the image synthesizing unit 37 synthesizes the image in a state where the image of the three-dimensional shape generated by the image generating unit 36 and the image of the image storage unit 32 are synchronized. In order to handle synchronized images in the image composition unit 37, it is preferable to complete all the processes of the object estimation unit 33, the position / orientation estimation unit 35, and the image generation unit 36 within 1/30 second.

Next, an example of the marker 50 for measuring the position and orientation of the imaging devices 21L and 21R and a display example of CG will be described. FIG. 2A shows an example of the marker 50. As shown in FIG. 2A, a marker 50 may be attached to a box-shaped object 51 with wheels. Moreover, you may display the image of the virtual object 60 as shown, for example in FIG.2 (b) on the display apparatus (display) 22. FIG. In FIG. 2B, the virtual object 60 is an office cabinet, and shows an example in which the virtual object 60 is displayed superimposed on the object 51 of FIG. At this time, if the observer's hand 40 is present in front of the object 51 in FIG. 2A, the virtual object 60 is not displayed in the area where the hand 40 exists, as shown in FIG. 2B. To. Thereby, the front-rear relationship between the virtual object 60 and the hand 40 can be presented.
Note that the position at which the virtual object 60 is disposed is not limited to the position of the object 51 to which the marker 50 is attached, as long as the position is in accordance with the position and orientation of the imaging devices 21L and 21R.

FIG. 3 is a hardware configuration diagram of the image processing apparatus 30.
The image processing apparatus 30 includes a CPU 301, a RAM 302, a ROM 303, a keyboard 304, a mouse 305, an external storage device 306, a storage medium drive 307, an interface (I / F) 308, and a system bus 309. Prepare.
The CPU 301 is a processor that comprehensively controls the operation of the image processing apparatus 30, and controls each component (302 to 308) via the system bus 309. The RAM 302 has an area for temporarily storing programs and data loaded from the external storage device 306 and the storage medium drive 307. Further, the RAM 302 is an area for temporarily storing data (stereo images in the real space in the present embodiment) received from the external device (in the present embodiment, the imaging devices 21L and 21R) via the I / F 308. Have The RAM 302 also has a work area used when the CPU 301 executes each process. That is, the RAM 302 can provide various areas as appropriate. For example, the RAM 302 can function as the image storage unit 32 and the model shape storage unit 34 in FIG.

The ROM 303 stores computer setting data, a boot program, and the like. A keyboard 304 and a mouse 305 are examples of an operation input device, and various instructions can be input to the CPU 301 by being operated by a computer user.
The external storage device 306 is a large-capacity information storage device represented by a hard disk drive (HDD) device. The external storage device 306 stores an OS (operating system) and programs and data for causing the CPU 301 to execute the above-described processes described as being performed by the image processing apparatus 30. Such programs can include programs corresponding to the image acquisition unit 31, the object estimation unit 33, the position and orientation estimation unit 35, the image generation unit 36, and the image synthesis unit 37, respectively. The data can include CG model data and the like.

Programs and data stored in the external storage device 306 are appropriately loaded into the RAM 302 under the control of the CPU 301. The CPU 301 can implement the functions of the respective units of the image processing apparatus 30 shown in FIG. 1 by executing processing using the loaded program and data. Note that the external storage device 306 can also function as the image storage unit 32 and the model shape storage unit 35 of FIG.
The storage medium drive 307 can read a program and data recorded on a storage medium such as a CD-ROM and a DVD-ROM, and can write a program and data on the storage medium. In the above, a part or all of the programs and data described as being stored in the external storage device 306 may be recorded on this storage medium. Programs and data read from the storage medium by the storage medium drive 307 are output to the external storage device 306 and the RAM 302.

The I / F 308 includes an analog video port for connecting the imaging devices 21L and 21R, a digital input / output port such as IEEE1394, an Ethernet (registered trademark) port, or the like. Data received via the I / F 308 is input to the RAM 302 and the external storage device 306.
As described above, the function of each unit of the image processing apparatus 30 illustrated in FIG. 1 can be realized by the CPU 301 executing a program. However, at least some of the units of the image processing apparatus 30 shown in FIG. 1 may operate as dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU 301.

  Hereinafter, the procedure of processing executed in the image processing apparatus 30 will be described with reference to FIG. The process shown in FIG. 4 is started in response to an instruction input by the user, for example. However, the start timing of the process in FIG. 3 is not limited to the above timing. The image processing apparatus 30 can implement the processing illustrated in FIG. 4 by the CPU 301 reading and executing a necessary program. However, as described above, at least a part of each element of the image processing apparatus 30 shown in FIG. 1 may operate as dedicated hardware so that the processing in FIG. 4 is realized. In this case, the dedicated hardware operates based on the control of the CPU 301. Hereinafter, the alphabet S means a step in the flowchart.

First, in S1, the image acquisition unit 31 acquires a stereo image from the imaging devices 21L and 21R. Next, in S <b> 2, the image storage unit 32 temporarily stores the stereo image acquired from the image acquisition unit 31.
In S <b> 3, the object estimation unit 33 extracts the region of the hand 40 from the stereo image stored in the image storage unit 32 and estimates the three-dimensional shape and position of the hand 40. Further, the position information of the hand 40 is stored with time, and the moving speed and moving direction of the hand 40 are estimated based on the information. Details of the object estimation process in S3 will be described later.

In S4, the position / orientation estimation unit 35 estimates the position / orientation of at least one of the imaging devices 21L and 21R. The estimated position and orientation are used for CG rendering processing in the image generation unit 36. In S5, the image generation unit 36 generates (renders) an image of the virtual object 60 viewed from the position and orientation of the imaging devices 21L and 21R. Details of the CG rendering process in S5 will be described later.
In S6, the image composition unit 37 superimposes the CG generated in S5 on the live-action image stored in S2, and generates a composite image. In S <b> 7, the display control unit 38 performs display control for displaying the composite image generated in S <b> 6 on the display of the display device 22.

(Object estimation processing)
FIG. 5 is a flowchart showing the flow of the object estimation process executed by the object estimation unit 33 in S3 of FIG.
First, in S <b> 31, the object estimation unit 33 estimates the three-dimensional shape of the hand 40 from the stereo image stored in the image storage unit 32. This process can be performed by a known process. For example, the foreground region of the hand 40 is extracted by extracting the difference between the background image acquired in advance and the current live-action image. Further, the areas of the two hands 40 extracted from the stereo image are associated with each other. Thereafter, based on the known arrangement information of the imaging devices 21L and 21R, the region of the hand 40 is measured in stereo, and the three-dimensional shape of the hand 40 is estimated. The object estimation unit 33 outputs the estimated three-dimensional shape to the model shape storage unit 34.

In S32, the object estimation unit 33 estimates the position of the hand 40. The position of the hand 40 is expressed by, for example, the coordinates of the center of gravity of the region of the hand 40 from the three-dimensional shape of the hand 40 estimated in S31. The position of the hand 40 is not limited to the center of gravity of the hand 40, and may be the coordinates of the tip of a specific finger. Further, the method of estimating the position of the hand 40 is not limited to the method of estimating based on the three-dimensional shape of the hand 40, and may be a method using a motion capture device or a magnetic sensor. The object estimation unit 33 outputs the estimated position to the model shape storage unit 34.
In S33, the object estimation unit 33 stores the position of the hand 40 estimated in S32 together with the time at that time. The time can be derived from the time code of the stereo image used when estimating the three-dimensional shape of the hand 40. Or you may acquire by a timer measurement means. By storing the position of the hand 40 together with the time, a history of the position of the hand 40 in a certain past can be stored.

  In S34, the object estimation unit 33 estimates the moving speed and moving direction of the hand 40 at that time. The moving speed and moving direction can be estimated from the position of the hand 40 estimated in S32 and the history of the position of the hand 40 at the past time stored in S33. For example, if the hand 40 exists at the coordinates (X1, Y1, Z1) at the time T1 and exists at the coordinates (X2, Y2, Z2) at the time T2, the distance between the two time points is calculated and divided by the time difference. Thus, the moving speed can be calculated. Further, the moving direction can be calculated by obtaining vectors at two time points. Here, acceleration may be calculated instead of the moving speed. The object estimation unit 33 outputs the estimated movement speed and movement direction to the image generation unit 37, and ends the process of FIG.

(CG rendering process)
FIG. 6 is a flowchart showing the flow of the CG rendering process executed by the image generation unit 36 in S5 of FIG.
First, in S51, the image generation unit 36 determines the transparency of the image (CG) of the virtual object 60. The transparency is determined according to the moving speed of the hand 40 determined in S34 of FIG.
Specifically, when the moving speed of the hand 40 is less than a predetermined speed, the CG is made opaque and displayed as it is, and when it is equal to or higher than the predetermined speed, the CG is made transparent. Thereby, when the observer quickly moves the hand 40, the CG becomes transparent. If the real object is hidden behind the virtual object 60, the real object can be displayed and presented to the observer. . On the other hand, when the observer does not move the hand 40 or when the movement of the hand 40 is slow, the CG is not transparent, and the observer can observe the virtual object 60.

FIG. 7 shows a display example when the observer quickly moves the hand 40 when displaying the image of the virtual object 60 shown in FIG. 2B using the object 51 and the marker 50 shown in FIG. Is shown. As shown in FIG. 7, when the hand 40 is quickly moved at a moving speed equal to or higher than a predetermined speed, the CG becomes transparent, and the object 51 with the marker 50 on the back side of the virtual object 60 can be seen through. It will be.
When the moving speed of the hand 40 is equal to or higher than a predetermined speed, the transparency may be increased in proportion to the moving speed so that the hand 40 is completely transparent at a certain moving speed or higher. In this case, when the observer does not move the hand 40 or when the movement of the hand 40 is slow, the CG does not become transparent, but when the hand 40 moves at a predetermined speed or more, the observer moves the hand 40 quickly. The transparency of CG increases. When the hand 40 moves at a certain moving speed or more, the CG becomes completely transparent. Note that the moving speed of the hand 40 and the transparency may be linked by a method other than the above. Moreover, when the object estimation part 33 estimates acceleration instead of speed in S34 of FIG. 4, you may link acceleration and transparency. For example, the transparency may be determined so as to make the CG transparent when the estimated acceleration is equal to or greater than a predetermined acceleration.

  In S <b> 52, the image generation unit 36 captures the three-dimensional shape data of the hand 40 and the three-dimensional model data of the virtual object 60 stored in the model shape storage unit 34 and the imaging device estimated by the position / orientation estimation unit 35. The positions and orientations of 21L and 21R are acquired. Then, the image generation unit 36 generates an image of the virtual object 60 viewed from the position and orientation of the imaging devices 21L and 21R. However, when generating an image, the distance from the imaging devices 21L and 21R between the hand 40 and the virtual object 60 is determined for each drawing pixel, and the virtual object 60 is not drawn for the pixel in which the hand 40 is in front, Make it transparent. That is, for a pixel in which the hand 40 is in front, the photographed image is presented to the observer so that the hand 40 is present in front. In S52, the image generation unit 36 generates an image of the virtual object 60 according to the transparency determined in S51.

As described above, the image processing device 30 in the present embodiment displays the image of the virtual object 60 based on the position and orientation of the imaging devices 21L and 21R that capture the real space. At this time, the image processing device 30 acquires the moving speed of the object imaged in the images of the imaging devices 21L and 21R, and determines the transparency of the image of the virtual object 60 based on the acquired moving speed. The image of the virtual object 60 is displayed according to the transparency. Here, the object for obtaining the moving speed can be the hand 40 of the observer who observes the image of the virtual object 60.
The image processing apparatus 30 can determine the transparency so that the image of the virtual object 60 is transparent when the moving speed of the hand 40 is equal to or higher than a predetermined speed. At this time, the image processing apparatus 30 may increase the transparency as the moving speed increases. As described above, when the image of the virtual object 60 is made transparent, if a real object exists behind the virtual object 60, the real object is displayed, and the observer can confirm the real object. Therefore, the possibility that the observer's hand 40 contacts the real object can be reduced.

  Further, since the image of the virtual object 60 becomes transparent when the moving speed of the hand 40 is equal to or higher than a predetermined speed, even if the hand 40 is in a place away from the real object, the hand 40 is at a high speed. When moving, the image of the virtual object 60 can be made transparent. Therefore, the observer can easily confirm the position of the real object and can avoid the collision of the hand 40 with the real object with a margin. On the other hand, when the moving speed of the hand 40 is less than a predetermined speed, the image of the virtual object 60 does not become transparent. Therefore, if the hand 40 is moving slowly near the real object and there is little impact even if the hand 40 contacts the real object, the image of the virtual object 60 can be displayed without being transparent. Therefore, it does not hinder the observer from observing the virtual object 60.

The display device on which the image processing device 30 displays the image of the virtual object 60 is a head-mounted display device (HMD) worn by the observer on the head, and the HMD is a video see-through HMD. Can do. The image processing apparatus 30 can superimpose and display the image of the virtual object 60 on the image obtained by capturing the real space from the viewpoint position of the observer.
If the observer wears the head-mounted display device and moves while displaying the CG on the display, there is a possibility that the real object hidden behind the CG may come into contact without noticing. In particular, when the observer's hand moves quickly, there is a high possibility that the hand will come into contact with a real object hidden in the CG, and the impact upon contact will be great. On the other hand, in this embodiment, the transparency of CG displayed on the head-mounted display device worn by the observer can be controlled according to the movement speed of the observer's hand. It is possible to present a mixed reality without any sense of incongruity while appropriately suppressing sex.

(Modification)
In the above embodiment, the case where the three-dimensional shape and position of the hand 40 are estimated and the moving speed and moving direction are estimated has been described. However, the object to be estimated for the moving speed and moving direction is the observer's hand. It is not limited to 40, but may be a part of another body such as a foot or a head. The object is not limited to a part of the observer's body, and may be an object operated by the observer. Further, the object may be a robot arm that is remotely operated by an observer, and the imaging devices 21L and 21R are not limited to cameras that capture the real space from the viewpoint position of the observer.

Furthermore, in the above-described embodiment, the case where a video see-through type HMD is used as the display device has been described. However, an optical see-through type HMD can also be realized. In the present embodiment described above, the image composition unit 37 composites the images of the imaging devices 21L and 21R so as to overwrite the image of the virtual object 60 generated by the image generation unit 36. However, instead of combining the image of the virtual object 60 with the images of the imaging devices 21L and 21R, the image of the virtual object 60 may simply be displayed on the display of the display device 22.
In addition, the display device is not limited to the HMD, and a handheld display (HHD) may be used. HHD is a handheld display. That is, it may be a display in which an observer takes a picture and observes an image by looking into it like binoculars. Furthermore, the display device may be a display terminal such as a tablet or a smartphone.

In S51 of FIG. 6, the image generation unit 36 is based on the distance between the position of the hand 40 and the position of the real object 51 on which the virtual object 60 is superimposed (position where the virtual object 60 is arranged). The transparency may be determined. For example, when the moving speed of the hand 40 is equal to or higher than a predetermined speed and the distance from the real object 51 is equal to or lower than the predetermined distance, the CG is made transparent. That is, even when the moving speed of the hand 40 is equal to or higher than a predetermined speed, the CG is not made transparent if the distance from the real object 51 exceeds the predetermined distance.
In this case, when the real object 51 including the marker 50 as shown in FIG. 2A is imaged by the imaging devices 21L and 21R, the position of the real object 51 is determined in the same manner as the object estimation process in S3 of FIG. presume. Next, the distance between the hand 40 and the real object 51 is calculated by calculating the difference between the position of the hand 40 and the position of the real object 51. Then, the transparency of the CG is determined according to the calculated distance. Even if the hand 40 is moved quickly when the hand 40 is away from the real object 51, there is no possibility of touching the real object 51. Therefore, in such a case, the CG is not made transparent, and a mixed reality without a sense of incongruity can be presented.

  Furthermore, in S51 of FIG. 6, the image generation unit 36 may determine the transparency based on the moving direction of the hand 40. When the observer moves the hand 40 in a direction away from the real object 51, the real object 51 is not contacted even if the hand 40 is moved quickly. Therefore, in such a case, the CG may be made transparent only when the moving direction of the hand 40 is a direction approaching the real object 51 so that the CG is not made transparent.

(Second embodiment)
In the first embodiment described above, the case where the entire image of the virtual object 60 is made transparent according to the moving speed of the object (hand 40) has been described. In the second embodiment, a case where a part of the image of the virtual object 60 is made transparent will be described.
The configuration of the MR system 10 including the image processing apparatus 30 in the present embodiment is the same as the configuration shown in FIG. 1 described above. Also, the operation of the image processing apparatus 30 in the present embodiment is the same as the operation shown in FIG. 4 described above. However, the CG rendering process in S5 of FIG. 4 is different from the first embodiment described above. Therefore, the following description will focus on the different parts of the process.

FIG. 8 is a flowchart showing the flow of the CG rendering process executed by the image generation unit 36 in this embodiment in S5 of FIG.
First, in S <b> 151, the image generation unit 36 determines a region (hereinafter referred to as “transparent region”) that makes the CG transparent according to the position of the hand 40. The transparent region can be a region corresponding to a predetermined range including the hand 40, for example, a region inside a certain distance from the hand 40. This transparent region can be determined from the three-dimensional shape and position of the hand 40 estimated in the object estimation process and the predetermined distance set in advance.

  FIG. 9A shows a state in which only the virtual object 60 around the hand 40 is made transparent. In FIG. 9A, the transparent area 71 is an area of a certain distance from the hand 40, and only the virtual object 60 included in the transparent area 71 is transparent, and the real object 51 behind it is A part of the image is displayed. In this case, when the observer moves the hand 40, the real object 51 around the hand 40 is displayed according to the position of the hand 40. Therefore, the possibility that the hand 40 contacts the real object 51 is increased. Can be reduced. Moreover, since CG other than the surroundings of the hand 40 does not become transparent, it is not hindered that an observer sees CG.

In the present embodiment, a case has been described in which a region within a certain distance from the hand 40 is a transparent region, but the transparent region in the moving direction of the hand 40 may be set wider than other directions. FIG. 9B shows a display example when the hand 40 is moving in the fingertip direction. In this case, the transparent region 72 has a wider range in the fingertip direction of the hand 40 than in other directions.
Thereby, the transparent range of the virtual object 60 arranged in the moving direction of the hand 40 is widened, and the possibility that the hand 40 comes into contact with the real object 51 hidden behind the virtual object 60 can be further reduced. it can. The transparent area in this case can be determined from the three-dimensional shape and position of the hand 40 estimated in the object estimation process, the moving direction of the hand 40, and a predetermined distance set in advance. Note that the range in which the transparent region is expanded may be constant regardless of the movement speed, or may be increased as the movement speed increases.

Returning to FIG. 8, in S <b> 152, the image generation unit 36 determines the transparency of the image (CG) of the virtual object 60 in consideration of the overlap between the transparent region determined in S <b> 151 and the virtual object 60. Specifically, it is as follows.
First, the virtual object 60 that does not overlap the transparent area is not made transparent.
The transparency of the image of the virtual object 60 that overlaps the transparent region is determined according to the moving speed of the hand 40 determined in S34. The transparency determination method in this case may be the same as the transparency determination method in S51. Alternatively, the transparency of the image of the virtual object 60 that overlaps the transparent area may be weighted in the transparent area. That is, a region close to the hand 40 in the transparent region may have higher transparency, and a region far from the hand 40 may have low transparency. Thereby, the virtual object 60 near the hand 40 that has a higher possibility of contact with the real object can be made transparent, and the possibility of contact with the real object can be appropriately reduced.

In S153, the image generation unit 36 generates an image of the virtual object 60 according to the transparency determined in S152. Detailed processing is the same as that in S52, and a description thereof will be omitted.
Similar to the first embodiment described above, also in this embodiment, the transparency may be determined in S152 according to the distance between the real object and the hand. Since this process can be performed in the same manner as in the first embodiment, a description thereof will be omitted.
As described above, the image processing apparatus 30 according to the present embodiment makes a region corresponding to a predetermined range including the hand 40 in the image of the virtual object 60 transparent. Since a part of the image of the virtual object 60, not the entire image of the virtual object 60, is a transparent region, the CG is not unnecessarily transparent. In addition, since the transparent region in the moving direction of the hand 40 is set wider than in other directions, the possibility of contact with a real object can be more reliably reduced.

  As described above, in each of the embodiments described above, the image processing apparatus 30 controls the transparency of the CG according to the moving speed of an object such as a hand. Therefore, when a hand or the like is moved quickly, the CG becomes transparent, and it is possible to display a real object that is normally hidden behind the CG and thus cannot be seen. can do. In addition, when the hand or the like does not move or is moved slowly, the CG does not become transparent, so that it is possible to prevent the observer from seeing the CG.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a recording medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  DESCRIPTION OF SYMBOLS 10 ... Mixed reality system, 21L, 21R ... Imaging device, 22 ... Display device, 30 ... Image processing device, 31 ... Image acquisition part, 32 ... Image storage part, 33 ... Object estimation part, 34 ... Model shape storage part, 35 ... Position and orientation estimation unit, 36 ... Image generation unit, 37 ... Image composition unit, 38 ... Display control unit

Claims (15)

First acquisition means for acquiring a first image obtained by imaging a real space;
Second acquisition means for acquiring the moving speed of the object imaged in the first image;
Determining means for determining the transparency of the second image generated based on the position and orientation of the imaging device that captured the first image based on the moving speed acquired by the second acquiring means;
An image processing apparatus comprising: a display control unit configured to display the second image on a display unit according to the transparency determined by the determination unit. The determining means includes
The image processing apparatus according to claim 1, wherein when the moving speed acquired by the second acquiring unit is equal to or higher than a predetermined speed, the transparency is determined so that the second image is transparent. . The determining means includes
The image processing apparatus according to claim 1, wherein the transparency is increased as the moving speed acquired by the second acquisition unit increases. The second image includes an image of a virtual object;
The determining means includes
The image processing apparatus according to claim 1, wherein the transparency is determined based on a distance between the position of the object and a position where the virtual object is arranged. The determining means includes
Based on the distance between the position of the first object whose movement speed is acquired by the second acquisition means and the position of the second object different from the first object captured in the first image. The image processing apparatus according to claim 1, wherein the transparency is determined.   The image processing apparatus according to claim 4, wherein the determining unit determines the transparency so that the second image is transparent when the distance is equal to or less than a predetermined distance. The second acquisition means further acquires a moving direction of the object,
The determining means includes
The image processing apparatus according to claim 1, wherein the transparency is determined based on a moving speed and a moving direction acquired by the second acquiring unit. The determining means includes
8. The image processing apparatus according to claim 1, wherein transparency of an area corresponding to a predetermined range including the object is determined in the second image. 9. The second acquisition means further acquires a moving direction of the object,
The determining means includes
The image processing apparatus according to claim 8, wherein the range in the movement direction acquired by the second acquisition unit is set wider than other directions.   The image processing apparatus according to claim 1, wherein the object is a part of an observer's body. The second acquisition means further acquires acceleration of the object,
The determining means includes
The image processing apparatus according to claim 1, wherein the transparency is determined based on an acceleration acquired by the second acquisition unit. The display control means includes
The image processing apparatus according to claim 1, wherein the second image is combined with the first image and displayed on the display unit. The display unit is a head-mounted display device worn by an observer on the head,
The image processing apparatus according to claim 1, wherein the first image is an image obtained by capturing a real space from the viewpoint position of the observer. Obtaining a first image in which real space is imaged;
Obtaining a moving speed of an object imaged in the first image;
Determining the transparency of the second image generated based on the position and orientation of the imaging device that captured the first image based on the moving speed;
And displaying the second image on a display unit in accordance with the transparency.   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 13.

Images