JP2019029868A - Image processing system and processing method, program thereof - Google Patents

Abstract

To provide a user-friendly mechanism, by controlling shake correction of a composite image by a state of a testee.SOLUTION: An image processing system (PC100) includes generation means for generating a composite image obtained by synthesizing an image of a virtual object displayed according to a position and an attitude related to a view point of a testee, first output means for outputting the composite image, generated by the generation means, to a first display device (HMD101) used by the testee, and second output means for outputting the composite image, subjected to shake correction by using the composite image, to a second display device (160). The second display device outputs the composite image, generated by the generation means, to the second display device, when the conditions related to the shake correction are satisfied, based on the position of the testee.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique related to blur correction of a composite image.

In recent years, mixed reality (hereinafter referred to as MR) technology has become widespread. By using a mixed reality technology, a mixed reality image in which a CG model is arranged on a real image is provided to a user wearing a head-mounted display (hereinafter referred to as an HMD), and a mixed reality world in which reality and virtual are combined ( (Mixed reality space) can be experienced. In generating a mixed reality image, a method of specifying the position of the HMD and the position of the CG model using a sensor or a two-dimensional marker is used.

Patent Document 2 discloses a technique of combining an image including an experienced person photographed by an objective camera and a CG image projected when the experienced person is experiencing and displaying the synthesized image on a display.

As described above, not only displaying an image to the experience person but also displaying the image viewed by the experience person and the experience state of the experience person is generally performed.

JP 2003-308514 A Japanese Patent Laying-Open No. 2015-170232

As described above, in MR, an image viewed by an experienced person is displayed on an external display (the same image as the image displayed on the experienced person's HMD is displayed), and a person who has not experienced confirms from outside. Sometimes.

In this case, since the experience person is moving, the actual image (moving image) photographed by the HMD is greatly blurred (many blurring occurs). Experienced users will see MR images generated using real images on the HMD according to their movements, so even if there is a blurry display, there is no sense of incongruity, but those who have not experienced their own When a video (blurred video) that is not related to movement is viewed on an external display, the user feels uncomfortable (easy to get sick).

In particular, a large display is often used as the external display, and blurring may be felt more greatly, and there is a problem that the image displayed on the HMD is not suitable for those who have not experienced it.

On the other hand, even if it is a person who has not experienced, if it is necessary to see the actual working state of the experienced person (in the case of verification work etc.), it is necessary to see the video displayed on the experienced person's HMD as it is. There is also a problem in controlling blurring.

Accordingly, an object of the present invention is to provide an easy-to-use mechanism by controlling the blur correction of a composite image according to the state of an experienced person.

An image processing apparatus for achieving an object of the present invention is generated by a generating unit that generates a composite image by combining virtual object images displayed according to a position and orientation related to a viewpoint of an experiencer, and the generating unit. First output means for outputting the composite image to the first display device used by the experience person, and second output means for outputting the composite image corrected for blur using the composite image to the second display device, The second output means outputs the composite image generated by the generation means to the second display device when the condition relating to blur correction is satisfied based on the position of the experience person.

  According to the present invention, it is possible to provide an easy-to-use mechanism by controlling the blur correction of a composite image according to the state of an experienced person.

The figure which shows the outline | summary of a system. FIG. 3 is a block diagram showing an example of the hardware configuration of each of the HMD 101 and the PC 100. The block diagram which shows the function structural example of PC100. The block diagram which shows the module structural example of PC100. The flowchart of the process which HMD101, PC100, and the external display 160 perform. The flowchart of the process in step S510. The figure which shows the structural example of the information hold | maintained at the external memory. The figure which shows the example of a display at the time of the blurring correction of the external display 160. The figure which shows the example of a display at the time of cancellation | release of blur correction of the external display 160. FIG.

Embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

In the system according to the present embodiment, a mixed reality space that is a composite space of a real space and a virtual space is used in a head-mounted display device such as an HMD or a display device in which a head-mounted display device is provided separately. It is a mixed reality presentation system. First, an outline of a system according to the present embodiment will be described with reference to FIG.

The HMD 101 is attached to the user's head, and an image of the mixed reality space output from the PC 100 as the image processing apparatus is displayed on the display screen of the HMD 101. The user experiences the mixed reality space by observing the mixed reality space image displayed on the display screen of the HMD 101.

Further, in the present embodiment, a description will be given assuming that there are a plurality of users (experience person A, experience person B) who are experiencing.

The optical marker 103 that can be detected (measured) by the optical sensor 104 is attached to the HMD 101, and the measurement result of the optical marker 103 by the optical sensor 104 is sent to the PC 100. In FIG. 1, in order to simplify the illustration, the number of the optical sensors 104 is set to 1, but a plurality of optical sensors 104 are actually arranged. The PC 100 obtains the position and orientation of the optical marker 103 based on the measurement result of the optical marker 103 by the optical sensor 104. Since a technique for obtaining the position and orientation of the optical marker 103 based on the measurement result of the optical marker 103 by the optical sensor 104 is well known, a detailed description thereof will be omitted. For example, when the camera provided in the HMD 101 is used as the user's viewpoint (user viewpoint), the PC 100 uses the “relative position and orientation relationship between the user viewpoint and the optical marker 103” obtained in advance. Then, the position and orientation of the optical marker 103 are converted into the position and orientation of the user viewpoint.

The method for acquiring the position and orientation of the user viewpoint is not limited to a specific method. For example, the position and orientation measurement sensor used to obtain the position and orientation of the user viewpoint is not limited to the optical sensor 104, but the position and orientation of the user viewpoint using other types of position and orientation measurement sensors such as a magnetic sensor and an ultrasonic sensor. You may make it acquire. Alternatively, the two-dimensional marker 102 arranged in the real space may be imaged by a camera provided in the HMD 101, and the position and orientation of the user viewpoint may be obtained using a captured image obtained by the imaging. Further, the position and orientation of the user viewpoint may be acquired by using the position and orientation measurement sensor and the two-dimensional marker together.

Then, the PC 100 generates a virtual object such as the CG model 130 and arranges it in the virtual space, and generates an image of the virtual object viewed from the user viewpoint. The PC 100 generates a composite image obtained by synthesizing the generated virtual object image and the real space image captured by the camera of the HMD 101 as an image of the mixed reality space, and the generated mixed reality space image. Is sent to the HMD 101. Thereby, the user can observe the image of the mixed reality space according to the position and orientation of his / her viewpoint in front of the eyes.

In the real space, an external display 160 for displaying an image when the user is experiencing the mixed reality space is arranged. The external display 160 is used for confirming the video of the experienced person by a user who has not experienced.

In the present embodiment, it is assumed that the external display 160 is connected to each PC 100 of the HMD 101. The video displayed on each HMD 101 is displayed on each external display 160. The image of the HMD 101 of the user A is displayed on the external display 160 for the user A. Also, the video of the HMD 101 of the experience person B is displayed on the external display 160 for the experience person B.

The display control on the external display 160 is an example of the feature in the present embodiment.

Next, a hardware configuration example of each of the HMD 101 and the PC 100 will be described with reference to the block diagram of FIG. The configuration illustrated in FIG. 2 is an example of a hardware configuration applicable to the HMD 101 and the PC 100, and is not limited to the configuration illustrated in FIG.

First, the HMD 101 will be described. The right-eye / left-eye video camera 221 includes a right-eye video camera for capturing a real-space image provided to the user's right eye, a left-eye video camera for capturing a real-space image provided to the user's left eye, Images taken by the respective video cameras (real space images) are sent to the PC 100.

The right-eye / left-eye display 222 includes a right-eye display for providing video (including images and characters) to the user's right eye, and a left-eye display for providing video (including images and characters) to the user's left eye. Have. The right-eye video sent from the PC 100 is displayed on the right-eye display, and the left-eye video is displayed on the left-eye display.

Next, the PC 100 will be described. The general-purpose bus 212 functions as an interface for receiving a real space image transmitted from the right-eye / left-eye video camera 221 and includes, for example, an external input terminal (for example, an IEEE 1394 terminal).

The CPU 201 executes various processes using computer programs and data stored in the ROM 202 and RAM 203. As a result, the CPU 201 controls the operation of the entire PC 100 and executes or controls each process described later as what the PC 100 performs.

The ROM 202 stores rewrite-free computer programs and data, for example, BIOS (Basic Input / Output System) which is a control program of the CPU 201.

The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 and the external memory 211. The RAM 203 has an area for storing an image of the real space received from the right-eye / left-eye video camera 221 via the general-purpose bus 212. The RAM 203 has an area for storing measurement results received from the optical sensor 104 via the communication I / F controller 208. The RAM 203 has an area for storing an image of the real space received from the objective camera 106 via the communication I / F controller 208. The RAM 203 has a work area used when the CPU 201 executes various processes. Thus, the RAM 203 can provide various areas as appropriate.

The video controller 206 is for performing display control of the right eye / left eye display 222, the display 210, and the external display 160. For the right-eye / left-eye display 222, for example, an external output terminal (for example, Digital Visual Interface) is used to output a video signal (image or character display signal). The display 210 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 201 using an image, text, or the like. Information such as various system settings can be input and displayed using the display 210.

The input controller 205 is for performing operation control of the input device 209. The input device 209 functions as a user interface such as a mouse and a keyboard, and can input various instructions to the CPU 201 by a user operation. For example, when the input device 209 is a touch panel, the user can input various instructions by pressing (touching with a finger or the like) an icon, a cursor, or a button displayed on the touch panel. The touch panel may be a touch panel that can detect a position touched by a plurality of fingers, such as a multi-touch screen.

The memory controller 207 is for controlling reading and writing of information to and from the external memory 211. The external memory 211 is a memory device such as a hard disk drive device, a flexible disk, a card type memory connected to the PCMCIA card slot via an adapter. The external memory 211 stores an OS (Operating System) and computer programs and data for causing the CPU 201 to execute or control each process described below as performed by the PC 100.

The computer program stored in the external memory 211 includes a computer program for causing the CPU 201 to execute processing according to each flowchart described below. The data stored in the external memory 211 includes information handled as known information in the following embodiments including the present embodiment, and data (drawing data) required for drawing various virtual objects including the CG model 130. )It is included. Computer programs and data stored in the external memory 211 are appropriately loaded into the RAM 203 under the control of the CPU 201 and are processed by the CPU 201. The external memory 211 may be a memory device such as a USB memory or an SD card.

The communication I / F controller 208 functions as an interface for receiving measurement results and real space images from the optical sensor 104. The communication I / F controller 208 functions as an interface for performing data communication with an external device via a network such as a LAN or the Internet. For example, the communication I / F controller 208 can perform Internet communication using TCP / IP. The communication I / F controller 208 also controls communication with the optical sensor 104 through Gigabit Ethernet (registered trademark) or the like. In FIG. 1, data communication between the optical sensor 104 and the external display 160 and the PC 100 is wired, but may be wireless. For example, the PC 100 can perform data communication with the optical sensor 104 and the external display 160 by various communication modes such as a LAN cable, a USB cable, and wireless communication.

The general-purpose bus 212, CPU 201, ROM 202, RAM 203, video controller 206, input controller 205, memory controller 207, and communication I / F controller 208 are all connected to the system bus 204.

In the present embodiment, the HMD 101 and the PC 100 are separate devices, but the HMD 101 and the PC 100 may be integrated. Further, the PC 100 may be handled as a server device.

Next, a module configuration example of the PC 100 will be described with reference to the block diagram of FIG. The operating system 401 controls input / output to / from the HMD 101, and transfers an image of the real space obtained from the right-eye / left-eye video camera 221 via the general-purpose bus 212 to the mixed reality platform 403. The operating system 401 also converts the mixed reality space image drawn by the graphic engine 402 via the video controller 206 to the right eye /
Output to the left eye display 222.

The graphic engine 402 generates a virtual object image using the virtual object drawing data stored in the external memory 211 and superimposes the generated virtual object image on the image captured by the right-eye / left-eye video camera 221. Thus, an image of the mixed reality space is generated. An engine used for generating (drawing) an image of a virtual object may be a widely used graphic engine such as OpenGL or DirectX, or a graphic engine developed independently. In the present embodiment, OpenGL is used as the graphic library.

The mixed reality platform (also referred to as MR platform) 403 obtains the position and orientation of the user's viewpoint as described above, and aligns the real space and the virtual space. These techniques can be realized by using existing techniques disclosed in, for example, Japanese Unexamined Patent Application Publication Nos. 2002-32784, 2006-072903, and 2007-166427.

A mixed reality application (also referred to as an MR application or a viewer application) 404 is required to draw a virtual object such as a viewpoint position and orientation, information on the color and shape of a virtual object, and a position and orientation of a virtual object from the mixed reality platform 403. The information is received, and a drawing command for a virtual object image is issued to the graphic engine 402. At this time, using the OpenGL API, a command in which the identification information and the position and orientation information of the virtual object to be drawn are set is issued.

Further, when the mixed reality platform 403 receives a captured image from the objective camera 106, the mixed reality platform 403 delivers the captured image to the mixed reality application 404. The correction module 405 performs a process for correcting the blur of the image drawn by the graphic engine 402. It is assumed that the mixed reality platform 403 and the mixed reality application 404 control whether or not to perform the blur correction process.

The correction module 405 can be realized by using an electronic camera shake correction technique. For example, a captured image is once read into a memory, and compared with the next captured image, the image shift is corrected by shifting the pixels. There are various techniques for blur correction, and any of the correction techniques may be used. The image to be corrected is an image output to the HMD 101.

The image corrected by the correction module 405 or the image not subjected to the vibration correction is output to the external display 160.

Next, processing performed by the HMD 101 and the PC 100 in order to generate a single (one frame) mixed reality space image and display it on the HMD 101 or the external display 160 will be described with reference to the flowchart of FIG.

In step S500, the PC 100 performs correction processing settings related to blur correction. The correction process setting is set in the HMD information 700 and the CG model information 720. Setting information is input from a setting screen displayed on the display 210 of the PC 100.

Here, FIG. 7 will be described. FIG. 7 is a diagram illustrating an example of setting information used in the present embodiment. It is assumed that the setting information is stored in the external memory 211.

The setting information includes HMD information 700 and CG model information 720. The HMD information 700 stores the settings of each HMD 101.

The CG model information 720 stores settings for each CG model in the virtual space.

The HMD information 700 includes information of HMDID 701, position 702, posture 703, external display display presence / absence 704, target 705, and setting value 706.

The HMDID 701 stores an ID for each HMD 101. The position 702 and the posture 703 store the position and orientation information of the HMD 101 calculated by the mixed reality platform 403. This information is constantly updated.

The external display display presence / absence 704 is information (flag) that sets whether to output an image to be output to the HMD 101 to the external display. In the case of ON, it is a setting for outputting to the external display 160, and is a setting for performing blur correction processing. In the case of OFF, this is a setting for not outputting to the external display 160, and is a setting for not performing the blur correction process. In the case of ON, the target 705 and the set value 706 are set.

Even when the external display display presence / absence 704 is ON, when the target 705 and the set value 706 are not set, the image is output to the external display 160, but the correction process is not performed. It may be.

The target 705 and the set value 706 are settings for performing blur correction processing, and a reference for performing blur correction processing is set.

The object 705 can set whether to cancel the shake correction process based on the relationship with the HMD 101 of another experiencer or whether to cancel the shake correction process based on the relationship with the CG. When canceling the shake correction process when approaching the HMD 101, the target HMD ID is set in the target 705, and the distance information is stored in the setting value 706.

Further, when the correction process is canceled when approaching the CG, the target CG is set as the target 705 and the distance information is stored in the set value 706.

Although the CG model ID may be set for the target 705, in the present embodiment, it is assumed that the CG to be subjected to the correction cancellation processing is set in the CG model information 720.

The CG model information 720 includes information on a model ID 721, a model name 722, a file path 723, a position 724, a posture 725, and a correction control release target 726.

The model ID 721 stores the ID of the CG model. The model name 722 stores the file name of the CG model itself. The file path 723 stores a location where a file serving as a CG model is stored.

The position 724 and the posture 725 have position and posture values on the virtual space to be displayed in the real space where the CG model is displayed.

The correction control cancellation target 726 stores information for setting whether to cancel the correction process. In the case of a CG model for canceling the correction process (correction control), ON is set. In the case of a CG model that does not cancel the correction process (correction control), OFF is set.

That is, when the correction control cancellation target 726 is ON and the target 705 is CG, the correction process is canceled if the HMD 101 is in a predetermined position, and the same blurring as the image displayed on the HMD 101 is present. A certain image is displayed on the external display 160.

In step S501, the setting information of FIG. 7 is read into the memory, and the HMD 101 is activated. When the HMD 101 is activated, the process proceeds to step S502.

In step S502, real image capturing is started using the functions of the right-eye / left-eye video camera 221.

In step S <b> 503, the right-eye / left-eye video camera 221 outputs the captured real space image (the image captured by the right-eye camera and the image captured by the left-eye camera) to the PC 100. Imaging and transmission are repeated, and MR images are sequentially received from the PC 100 and displayed on the right-eye / left-eye display 222 of the HMD 101.

In step S504, the CPU 201 receives the real-space image output from the right-eye / left-eye video camera 221 via the general-purpose bus 212. In step S505, the CPU 201 receives the received real-space image in the RAM 203 or the external memory 211. To store.

In step S506, the CPU 201 acquires the measurement result of the optical marker 103 by the optical sensor 104 via the communication I / F controller 208, and based on the acquired measurement result, the position and orientation of the user viewpoint as described above. (HMD 101 position and orientation) is obtained. The position and orientation of the user viewpoint obtained in step S506 are the position and orientation of the right viewpoint (right-eye video camera) and the position and orientation of the left viewpoint (left-eye video camera). As with the above, the right viewpoint is obtained by converting the position and orientation of the optical marker 103 using the “relative position and orientation relationship between the right viewpoint and the optical marker 103” obtained in advance. Can do. Similarly, for the left viewpoint, the position and orientation of the optical marker 103 are converted using the “relative position and orientation relationship between the left viewpoint and the optical marker 103” obtained in the same manner as described above. Can be obtained.

In step S507, the CPU 201 stores the viewpoint position and orientation obtained in step S506 in the external memory 211 in association with the identification information (ID) of the HMD 101. For example, as shown in the HMD information 700 of FIG. 7, IDs, positions, and postures are managed in association with each HMD.

In step S508, the CPU 201 reads out the virtual object drawing data stored in the external memory 211 to the RAM 203 based on the CG model information 720 in FIG. 7, and generates a virtual object based on the read drawing data. The generated virtual object is arranged in the virtual space. Then, the CPU 201 generates an image of the virtual object viewed from the viewpoint based on the position and orientation of the viewpoint obtained in step S506. Then, the CPU 201 generates a mixed reality space image (MR image) by superimposing the virtual object image on the real space image received in step S504. Strictly speaking, the CPU 201 superimposes the image of the virtual object generated based on the position and orientation of the right viewpoint on the image captured by the right-eye camera (image in the real space), so that the image of the mixed reality space for the right eye is superimposed. And the virtual object image generated based on the position and orientation of the left viewpoint is superimposed on the image captured by the left-eye camera (real space image). Generate.

Step S508 shows an example of a generation process for generating a synthesized image obtained by synthesizing the virtual object image displayed in accordance with the position and orientation related to the viewpoint of the user.

In step S509, the CPU 201 determines whether to display the MR image generated in step S508 on the external display based on the setting of the external display display presence / absence 704. If it is ON, the process proceeds to step S510. If it is OFF, the process proceeds to step S514.

In step S <b> 510, the CPU 201 performs correction control when outputting to the external display 160. Details of the correction control processing will be described in detail with reference to FIG.

In step S511, the CPU 201 outputs the MR image to the external display 160 via the video controller 206. This MR image may be an image that has undergone blur correction processing or an image that has not been subjected to blur correction processing (cancelled).

Step S511 shows an example of a second output process for outputting a composite image that has been shake-corrected using the composite image to the second display device. In addition, an example of the second output process for outputting the composite image generated in step S508 to the second display device when the condition for blur correction is satisfied based on the position of the experience person is shown. Furthermore, when the distance to the object is not satisfied, the shake-corrected composite image is output to the second display device, and when the distance to the object is not satisfied, the blur correction is canceled and generated in step S508. The synthesized image is output to the second display device.

In step S512, the external display 160 receives the MR image from the PC 100, and displays the received MR image in step S513.

The example displayed on the external display 160 in step S513 is FIG. 8 and FIG.

FIG. 8 shows a state where the user A is looking at the CG 802 to cancel the correction control, but is subjected to the blur correction process and displayed because it is outside the set value 706 distance (does not satisfy the condition).

FIG. 9 shows a state in which the user A is looking at the CG 902 to cancel the correction control, and the blur correction process is canceled and displayed because the user is within the distance of the set value 706 (satisfies the standard). A dotted line CG 902 indicates a model displayed by blurring. Note that the display of the dotted line 903 is for the purpose of explaining blurring for convenience, and is not limited to the dotted line display.

In addition, as in 901, in order to call attention to the user who is viewing the external display 160 other than the experienced person, "The same video as the experienced person is displayed. Please be careful. ”Is displayed. This display is information output from the PC 100 when the correction control is canceled. The display 901 can be arbitrarily set by setting information (not shown) in FIG.

In step S514, the CPU 201 outputs the MR image (the mixed reality space image for the right eye and the mixed reality space image for the left eye) generated in step S508 to the HMD 101 via the video controller 206.

Step S514 shows an example of a first output process for outputting the composite image generated in step S508 to the first display device used by the experience person.

In step S515, the right-eye / left-eye display 222 receives the right-eye mixed reality space image and the left-eye mixed reality space image transmitted from the PC 100 in step S514. In step S516, the right-eye / left-eye display 222 displays the right-eye mixed reality space image on the right-eye display.

Next, the correction control process in step S510 will be described with reference to the flowchart of FIG.

In step S601, the CPU 201 specifies the HMD 101 of another experience person or the target CG based on the setting information of FIG. In the case of the HMD 101 of another experienced person, it is desirable that the other experienced person is included in an image captured. That is, the blur correction control is canceled when a condition such as a set value is satisfied while viewing another experienced person.

In step S602, the CPU 201 calculates and acquires a distance from the HMD 101 of another experience person or the target CG. The distance can be calculated from the position of the HMD 101 of the target experiencer and the position of the HMD of another experience, or the position of the HMD 101 of the target experiencer and the position of the CG.

In step S <b> 603, the CPU 201 determines whether or not the acquired distance satisfies the distance of the set value 706. Specifically, it is determined whether or not the HMD 101 of the target experiencer exists at a position farther than the distance of the set value 706. If it is in a far position, it is determined that the condition is not satisfied, and the process proceeds to step S607. If it is close, the condition is satisfied and the process proceeds to step S604. That is, the above-described conditions are conditions relating to blur correction, and are conditions for canceling blur correction. It is also the distance between the position of the experience person and the object.

In step S604, the CPU 201 determines whether or not the blur correction has been canceled. That is, when the shake correction state is not performed (the release state), the process proceeds to step S605. If already released, the process proceeds to step S606.

In step S605, since blur correction is performed in a normal case, when the blur correction is canceled, the CPU 201 instructs the user other than the user to cancel that the correction process has been canceled. Information (notification) that the blur correction has not been performed is output to the external display 160. An example of alerting is 901 in FIG.

In this embodiment, control is performed so as to call attention when blur correction is canceled. However, warning information is output (output at predetermined intervals) while blur correction is cancelled. Also good.

Needless to say, when the warning display is displayed and the timing for correcting the blur is reached, control is performed to cancel the warning display.

Step S605 is an example of notification processing for notifying that blur correction has not been performed when the conditions relating to blur correction are satisfied based on the position of the experience person and the composite image generated in step S508 is output to the second display device. Is shown.

In step S606, the CPU 201 acquires the MR image generated in step S508 (an MR image without correction processing).

In step S607, blur correction is performed by the correction module 405 using the MR image generated in step S508.

In step S608, the CPU 201 acquires the MR image corrected in step S607.

The MR image acquired in step S606 or step S607 is output in step S511.

As described above, according to the present embodiment, the presence / absence of the blur correction process is switched according to the conditions of the HMD 101 and the display is performed on the external display. In particular, it is possible to reduce watching blurry video for a long time and prevent sickness.

The following forms are also conceivable as other embodiments. In the above-described embodiment, the blur correction process is canceled when the condition is satisfied. However, in other embodiments, the blur correction process is performed when the condition is satisfied (for example, the condition that the HMD 101 is outside the set distance is satisfied). Is executed, and the blur correction process is canceled when the condition is not satisfied (for example, the condition that the HMD 101 is within the set distance is satisfied).

In this way, it is only necessary to switch between execution and cancellation of the blur correction process, and any of the criteria that satisfy the condition (whether the reference is within or outside the distance) may be used.

Finally, a functional configuration example of the PC 100 will be described with reference to the block diagram of FIG. Each functional unit in FIG. 3 may be implemented by hardware or software (computer program). In the former case, for example, by installing such hardware in the HMD 101, the HMD 101 can also execute the processes described above as being performed by the PC 100. In the latter case, such a computer program is stored in the external memory 211, and the CPU 201 reads out this computer program to the RAM 203 and executes it appropriately, thereby realizing the function of the corresponding functional unit. The PC 100 can also be rephrased as an image processing apparatus.

The generation unit 301 is a functional unit that generates a combined image by combining virtual object images displayed in accordance with the position and orientation related to the viewpoint of the user. It is desirable that the image is a mixed reality space obtained by synthesizing the captured real image and CG.

The first output unit 302 is a functional unit that outputs the composite image generated by the generation unit 301 to a first display device (for example, the HMD 101) used by the experience person.

The second output unit 303 is a functional unit that outputs a composite image that has undergone shake correction using the composite image to a second display device (for example, the external display 160). In addition, the second output unit 303 displays the composite image generated by the generation unit 301 when a condition related to shake correction (specifically, a condition for canceling shake correction) is satisfied based on the position and orientation of the experience person. It is a functional unit that outputs to the second display device.

The notification unit 304 notifies that the shake correction is not performed when the condition for releasing the shake correction based on the position and orientation of the experience person is satisfied and the composite image generated by the generation unit 301 is output to the second display device. It is a functional part. An example of notification is 901.

  The condition for canceling the shake correction is a distance between the position and orientation of the experience person and the object.

  In addition, the second output unit 303 outputs the shake-corrected composite image to the second display device when the distance to the object is not satisfied, and cancels the shake correction when the distance to the object is not satisfied. Then, the composite image generated by the generation unit 301 is output to the second display device.

  In other words, the image processing apparatus according to the present embodiment can improve convenience in a form such as verification by outputting a composite image that is not subjected to shake correction depending on the position of the user when displaying the composite image on an external display. it can. Especially when motion is not necessary for verification, shake correction is performed. When verification is required (accurate display is required), control is performed so that blur correction is not performed. Verification from the outside can be facilitated while reducing poor physical condition.

It should be noted that the external display may be any device as long as it is a display device used by someone other than the experience person.

In the above-described embodiment, the HMD 101 and the PC 100 are separate devices. However, the HMD 101 and the PC 100 are integrated, and the HMD 101 performs not only the above-described processes performed by the own device but also the PC 100. The above-described processes described above may also be performed.

In the above-described embodiment, the HMD 101 is described as a video see-through head-mounted display device. However, an optical see-through head-mounted display device may be used as the HMD 101.

In addition to or instead of the HMD 101 mounted on the user's head, an image of the mixed reality space based on the user viewpoint is output to another display device, for example, a mobile terminal device such as a tablet terminal device or a smartphone. Anyway. In this case, the mobile terminal device can acquire the position and orientation of the mobile terminal device as a user viewpoint by using a sensor or application provided in the mobile terminal device. The image of the mixed reality space based on the user viewpoint may be generated by the PC 100 or may be generated on the mobile terminal device side.

In the above-described embodiment, the mixed reality presentation system (MR system) that presents the user with the mixed reality space obtained by synthesizing the real space and the virtual space has been described as an example. However, only the virtual space is presented to the user. You may apply to what is called a VR system. In that case, processing related to imaging of the real space and synthesis with the image of the real space is not necessary.

Further, in the above-described embodiment, in order to simplify the description, one HMD 101 is connected to one PC 100. However, two or more HMDs 101 are connected to one PC 100. Also good.

Although the embodiment of the present invention has been described above, the present invention can take an embodiment as a system, apparatus, method, program, recording medium, or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

The program according to the present invention is a program that allows a computer to execute the processing method of the flowchart, and the storage medium according to the present invention stores a program that allows the computer to execute the processing method.

As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by executing reading.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM, silicon A disk, solid state drive, or the like can be used.

Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention.

Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a communication line using a communication program, the system or apparatus can enjoy the effects of the present invention.

In addition, a part or all of each embodiment described above may be used in combination as appropriate, or a part or all of each embodiment described above may be selectively used.

  100 PC 101 HMD 103 Optical marker 102 Two-dimensional marker 104 Optical sensor 130 CG model 160 External display

Claims (6)

A generating unit configured to generate a combined image obtained by combining virtual object images displayed in accordance with a position and orientation related to the viewpoint of the user; and a first display device used by the user using the combined image generated by the generating unit. A first output means for outputting, and a second output means for outputting a composite image that has been shake-corrected using the composite image to a second display device, wherein the second output means is based on the position of the experience person, An image processing apparatus that outputs a composite image generated by the generation unit to a second display device when a condition relating to blur correction is satisfied. And a notification means for notifying that the shake correction is not performed when the condition relating to the shake correction is satisfied based on the position of the experience person and the composite image generated by the generation means is output to the second display device. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the condition relating to the shake correction is a condition for canceling the shake correction, and is a distance between the position of the experience person and the object.   The second output means outputs the blur-corrected composite image to the second display device when the distance to the object is not satisfied, and cancels the blur correction when the distance to the object is not satisfied. The image processing apparatus according to claim 3, wherein the composite image generated by the generation unit is output to the second display device. A generation step of generating a composite image obtained by combining virtual object images displayed in accordance with the position and orientation related to the viewpoint of the experience person, and the first display device used by the experience person using the composite image generated by the generation step. A first output step of outputting, and a second output step of outputting a composite image that has been shake-corrected using the composite image to a second display device, wherein the second output step is based on the position of the experience person A processing method of an image processing apparatus, wherein the composite image generated by the generation step is output to a second display device when a condition relating to blur correction is satisfied. Generating means for generating a composite image obtained by synthesizing an image of a virtual object displayed in accordance with a position and orientation related to the viewpoint of the experience person of the image processing device; and a first use of the composite image generated by the generation means for the experience person A first output means for outputting to one display device; and a second output means for outputting a composite image that has been shake-corrected using the composite image to a second display device. A program for causing a second display device to output a composite image generated by the generation unit when a condition relating to blur correction is satisfied based on a position.

Images