JP7148779B2 - Image processing device, its processing method, and program - Google Patents

Description

The present invention relates to a technique for blur correction of a composite image.

In recent years, the technology of mixed reality (hereinafter referred to as MR) has become widespread. By using mixed reality technology, users wearing a head-mounted display (HMD) are provided with a mixed reality image in which a CG model is arranged in a real image, and a mixed reality world that combines reality and virtuality ( (Mixed reality space) can be experienced. In generating a mixed reality image, a method of specifying the position of the HMD and the position of the CG model using sensors and two-dimensional markers is adopted.

Patent Literature 2 discloses a technique for synthesizing an image including the experiencer captured by an objective camera with a CG image projected while the experiencer is experiencing the experience, and displaying the result on a display.

In this way, it is common practice not only to simply display an image to the experiencer, but also to display the image the experiencer is looking at and the experience state of the experiencer.

JP-A-2003-308514 JP 2015-170232 A

As described above, in MR, the image that the user sees is displayed on the external display (the same image as that displayed on the HMD of the user is displayed), and the person who has not experienced it can check it from the outside. Sometimes.

In this case, since the user is moving, the real image (moving image) captured by the HMD is greatly blurred (many blurring occurs). Since the MR image generated using the real image is displayed on the HMD according to the user's movement, even if the display is blurred, there is no sense of discomfort. People often get sick (easily get motion sickness) because they are watching images that have nothing to do with movement (blurry images) on an external display.

In particular, the external display often uses a large display, and there is a problem that the image displayed on the HMD is not suitable for people who have not experienced the image displayed on the HMD.

On the other hand, even for those who have not experienced the experience, if it is necessary to see the actual work state of the experiencer (in the case of verification work, etc.), it is necessary to see the image displayed on the HMD of the experiencer as it is, so it is uniform There is also a problem in controlling blurring.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a user-friendly mechanism that controls blur correction of an image to reduce motion sickness of the user .

In order to achieve the above object, the image processing apparatus of the present invention includes:
controlling a specific image to be output to the first display means without performing specific blur correction for correcting image shift;
Control means for controlling the specific image to be subjected to the specific blur correction and output to the second display means
characterized by comprising

According to the present invention, it is possible to provide an easy-to-use mechanism that controls blur correction of an image and reduces sickness of the user .

The figure which shows the outline|summary of a system. FIG. 2 is a block diagram showing a hardware configuration example of each of the HMD 101 and PC 100; FIG. 2 is a block diagram showing a functional configuration example of a PC 100; FIG. FIG. 2 is a block diagram showing a module configuration example of the PC 100; 4 is a flowchart of processing performed by the HMD 101, PC 100, and external display 160; 4 is a flowchart of processing in step S510. 4 is a diagram showing a configuration example of information held in an external memory 211; FIG. FIG. 10 is a diagram showing a display example of the external display 160 during blur correction; FIG. 10 is a diagram showing a display example of the external display 160 when blur correction is canceled;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the embodiment described below is an example of a specific implementation of the present invention, and is one of specific examples of the configuration described in the claims.

The system according to the present embodiment displays a mixed reality space, which is a composite space of a real space and a virtual space, on a head-mounted display device such as an HMD, or on a display device provided separately from the head-mounted display device. It is a mixed reality presentation system that presents to. First, the outline of the system according to this embodiment will be described with reference to FIG.

An HMD 101 is worn on the user's head, and the display screen of the HMD 101 displays an image of a mixed reality space output from the PC 100 as an image processing device. The user experiences the mixed reality space by observing the image of the mixed reality space displayed on the display screen of the HMD 101 .

Moreover, in this embodiment, it is assumed that there are a plurality of users (experiencer A, experiencer B) who are experiencing.

An optical marker 103 detectable (measurable) by an optical sensor 104 is attached to the HMD 101 , and the measurement result of the optical marker 103 by the optical sensor 104 is sent to the PC 100 . Although the number of optical sensors 104 is one in FIG. 1 for the sake of simplification, a plurality of optical sensors 104 are actually arranged. The PC 100 obtains the position and orientation of the optical marker 103 based on the measurement result of the optical marker 103 by the optical sensor 104 . Since the technique for obtaining the position and orientation of the optical marker 103 based on the measurement result of the optical marker 103 by the optical sensor 104 is well known, detailed description of the technique will be omitted. For example, when the camera provided in the HMD 101 is used as the user's viewpoint (user's viewpoint), the PC 100 uses the "relative position and orientation relationship between the user's viewpoint and the optical marker 103" obtained in advance. , transforms the position and orientation of the optical marker 103 into the position and orientation of the user's viewpoint.

Note that the method of acquiring the position and orientation of the user's viewpoint is not limited to a specific method. For example, the position and orientation measurement sensor used to obtain the position and orientation of the user's viewpoint is not limited to the optical sensor 104, and other types of position and orientation measurement sensors such as a magnetic sensor and an ultrasonic sensor may be used to obtain the position and orientation of the user's viewpoint. can be obtained. Alternatively, the two-dimensional marker 102 placed in the physical space may be imaged by a camera provided in the HMD 101, and the position and orientation of the user's viewpoint may be obtained using the captured image obtained by the imaging. Also, a position and orientation measurement sensor and a two-dimensional marker may be used together to acquire the position and orientation of the user's viewpoint.

Then, the PC 100 generates a virtual object such as the CG model 130 and arranges it in the virtual space to generate an image of the virtual object seen from the user's viewpoint. Then, the PC 100 generates a composite image obtained by synthesizing the generated image of the virtual object and the image of the real space captured by the camera of the HMD 101 as an image of the mixed reality space, and generates the generated image of the mixed reality space. is transmitted to the HMD 101 . This allows the user to observe an image of the mixed reality space according to the position and orientation of his/her own viewpoint.

Also, in the real space, an external display 160 is arranged to display an image when the user is experiencing the mixed reality space. This external display 160 is used by a non-experienced user to check the video of the experiencer.

In this embodiment, the external display 160 shall be connected to each PC100 of HMD101. Images displayed by each HMD 101 are displayed on each external display 160 . The image of the HMD 101 of the experiencer A is displayed on the external display 160 for the experiencer A. FIG. Also, the image of the HMD 101 of the experiencer B is displayed on the external display 160 for the experiencer B. FIG.

Note that display control on the external display 160 is an example of the feature of this embodiment.

Next, hardware configuration examples of the HMD 101 and the PC 100 will be described with reference to the block diagram of FIG. Note that the configuration shown in FIG. 2 is an example of a hardware configuration applicable to the HMD 101 and the PC 100, and is not limited to the configuration shown in FIG.

First, the HMD 101 will be explained. The right-eye/left-eye video camera 221 includes a right-eye video camera for capturing an image of the real space provided to the right eye of the user, a left-eye video camera for capturing the image of the real space provided to the left eye of the user, , and images captured by the respective video cameras (images of the real space) are sent to the PC 100 .

Right-eye/left-eye display 222 includes a right-eye display for providing images (including images and text) to the user's right eye and a left-eye display for providing images (including images and text) to the user's left eye. , has The right-eye image transmitted from the PC 100 is displayed on the right-eye display, and the left-eye image is displayed on the left-eye display.

Next, the PC 100 will be explained. The general-purpose bus 212 functions as an interface for receiving real space images sent from the right-eye/left-eye video camera 221, and is configured by, for example, an external input terminal (eg, IEEE1394 terminal).

The CPU 201 executes various processes using computer programs and data stored in the ROM 202 and RAM 203 . As a result, the CPU 201 controls the operation of the entire PC 100 and executes or controls each process described later as what the PC 100 performs.

The ROM 202 stores computer programs and data that do not need to be rewritten, such as BIOS (Basic Input/Output System), which is a control program for the CPU 201 .

RAM 203 has an area for storing computer programs and data loaded from ROM 202 and external memory 211 . The RAM 203 also has an area for storing real space images received from the right-eye/left-eye video camera 221 via the general-purpose bus 212 . The RAM 203 also has an area for storing measurement results received from the optical sensor 104 via the communication I/F controller 208 . The RAM 203 also has an area for storing the real space image received from the objective camera 106 via the communication I/F controller 208 . The RAM 203 also has a work area used when the CPU 201 executes various processes. Thus, the RAM 203 can appropriately provide various areas.

The video controller 206 is for performing display control of the right-eye/left-eye display 222 , the display 210 and the external display 160 . For the right-eye/left-eye display 222, for example, an external output terminal (for example, Digital Visual Interface) is used to output a video signal (image or character display signal). A display 210 is composed of a CRT, a liquid crystal screen, or the like, and can display the results of processing by the CPU 201 in the form of images, characters, or the like. The display 210 can be used to input and display information such as various system settings.

The input controller 205 is for controlling the operation of the input device 209 . The input device 209 functions as a user interface such as a mouse and a keyboard, and can be operated by the user to input various instructions to the CPU 201 . For example, if the input device 209 is a touch panel, the user can input various instructions by pressing (touching with a finger or the like) an icon, cursor, or button displayed on the touch panel. The touch panel may be a touch panel, such as a multi-touch screen, capable of detecting positions touched by multiple fingers.

The memory controller 207 is for controlling reading and writing of information to the external memory 211 . The external memory 211 is a memory device such as a hard disk drive, a flexible disk, or a card-type memory connected to a PCMCIA card slot via an adapter. The external memory 211 stores an OS (operating system), and computer programs and data for causing the CPU 201 to execute or control each process described later as what the PC 100 performs.

The computer programs stored in the external memory 211 include computer programs for causing the CPU 201 to execute processing according to each flowchart described below. The data stored in the external memory 211 includes information handled as known information in each of the following embodiments including this embodiment, and data required for drawing various virtual objects including the CG model 130 (drawing data). )It is included. Computer programs and data stored in the external memory 211 are appropriately loaded into the RAM 203 under the control of the CPU 201 and are processed by the CPU 201 . Note that the external memory 211 may be a memory device such as a USB memory or an SD card.

The communication I/F controller 208 functions as an interface for receiving measurement results and real space images from the optical sensor 104 . Also, the communication I/F controller 208 functions as an interface for performing data communication with external devices via a network such as a LAN or the Internet. For example, the communication I/F controller 208 is capable of Internet communication using TCP/IP. The communication I/F controller 208 also controls communication with the optical sensor 104 through Gigabit Ethernet (registered trademark) or the like. Note that data communication between the optical sensor 104 and the external display 160 and the PC 100 is wired in FIG. 1, but may be wireless. For example, the PC 100 can perform data communication with the optical sensor 104 and the external display 160 using various communication modes such as LAN cable, USB cable, and wireless communication.

All of the general-purpose bus 212 , CPU 201 , ROM 202 , RAM 203 , video controller 206 , input controller 205 , memory controller 207 and communication I/F controller 208 are connected to system bus 204 .

Although the HMD 101 and the PC 100 are separate devices in this embodiment, the HMD 101 and the PC 100 may be integrated. Also, the PC 100 may be treated as a server device.

Next, a module configuration example of the PC 100 will be described with reference to the block diagram of FIG. The operating system 401 controls input/output with the HMD 101 , and transfers the real space image obtained from the right-eye/left-eye video camera 221 via the general-purpose bus 212 to the mixed reality platform 403 . The operating system 401 also transmits the image of the mixed reality space drawn by the graphic engine 402 to the right eye/eye through the video controller 206 .
Output to left eye display 222 .

The graphic engine 402 generates an image of the virtual object using drawing data of the virtual object stored in the external memory 211, and superimposes the generated image of the virtual object on the image captured by the right-eye/left-eye video camera 221. This generates an image of the mixed reality space. The engine used to generate (render) the image of the virtual object may be, for example, a widely used graphic engine such as OpenGL or DirectX, or a uniquely developed graphic engine. Note that OpenGL is used as a graphic library in this embodiment.

A mixed reality platform (also referred to as an MR platform) 403 obtains the position and orientation of the user's viewpoint as described above, and aligns the physical space with the virtual space. These techniques can be implemented using existing techniques disclosed in, for example, Japanese Patent Application Laid-Open Nos. 2002-32784, 2006-072903, and 2007-166427.

A mixed reality application (also referred to as an MR application or a viewer application) 404 receives from the mixed reality platform 403 the position and orientation of the viewpoint, the color and shape information of the virtual object, the position and orientation of the virtual object, and other information necessary for drawing the virtual object. It receives information and issues a command to draw a virtual object image to the graphic engine 402 . At this time, an OpenGL API is used to issue a command in which identification information and position/orientation information of a virtual object to be drawn are set.

Also, upon receiving the captured image from the objective camera 106 , the mixed reality platform 403 transfers the captured image to the mixed reality application 404 . A correction module 405 performs blur correction processing on the image rendered by the graphic engine 402 . It is assumed that the mixed reality platform 403 and the mixed reality application 404 control whether or not to perform blur correction processing.

Correction module 405 can be implemented using electronic image stabilization technology. For example, a captured image is once read into a memory, compared with the next captured image, and image misalignment is corrected by shifting pixels. There are various blur correction techniques, and any correction technique may be used for implementation. An image to be corrected is an image to be output to the HMD 101 .

The blur-corrected image or the non-blur-corrected image is output to the external display 160 by the correction module 405 .

Next, processing performed by the HMD 101 and the PC 100 to generate one (one frame) image of the mixed reality space and display it on the HMD 101 and the external display 160 will be described with reference to the flowchart of FIG.

In step S500, the PC 100 performs correction processing settings related to blur correction. Correction processing settings are set in the HMD information 700 and the CG model information 720 . Setting information is input from a setting screen displayed on the display 210 of the PC 100 .

Now, FIG. 7 will be described. FIG. 7 is a diagram showing an example of setting information used in this embodiment. It is assumed that the setting information is stored in the external memory 211 .

The setting information includes HMD information 700 and CG model information 720 . The HMD information 700 stores settings of each HMD 101 .

The CG model information 720 stores settings of each CG model in the virtual space.

The HMD information 700 includes HMD ID 701 , position 702 , orientation 703 , external display display presence/absence 704 , target 705 , and setting value 706 .

The HMDID 701 stores an ID for each HMD 101 . A position 702 and an orientation 703 store position and orientation information of the HMD 101 calculated by the mixed reality platform 403 . This information is constantly being updated.

The external display display presence/absence 704 is information (flag) that sets whether or not an image to be output to the HMD 101 is output to an external display. When it is ON, it is a setting for outputting to the external display 160 and is a setting for performing blur correction processing. If it is OFF, it is a setting for not outputting to the external display 160, and is a setting for not performing blur correction processing. Also, if it is ON, the target 705 and the set value 706 are set.

Note that even when the external display display presence/absence 704 is ON, if the target 705 and the setting value 706 are not set, the image is output to the external display 160 but the correction process is not performed. may be

A target 705 and a set value 706 are settings for performing blur correction processing, and a standard for performing blur correction processing is set.

The object 705 can set whether to cancel the blur correction processing depending on the relationship with the HMD 101 of the other experiencer or whether to cancel the blur correction processing depending on the relationship with the CG. When the blur correction processing is canceled when the HMD 101 is approached, the ID of the target HMD is set in the target 705 and the distance information is stored in the set value 706 .

Further, when the correction process is canceled when the CG is approached, the target CG is set in the target 705 and the distance information is stored in the set value 706 .

A CG model ID may be set for the object 705 , but in this embodiment, it is assumed that the CG model information 720 is used to set whether the correction is to be canceled or not.

The CG model information 720 includes model ID 721 , model name 722 , file path 723 , position 724 , orientation 725 , and correction control cancellation target 726 .

The model ID 721 stores the ID of the CG model. The model name 722 stores the file name of the CG model itself. A file path 723 stores a location where a file serving as a CG model is saved.

A position 724 and an orientation 725 have values of the position and orientation in the virtual space to be displayed in the physical space where the CG model is displayed.

The correction control cancellation target 726 stores information for setting whether or not to cancel the correction processing. ON is set in the case of a CG model for which correction processing (correction control) is to be canceled. OFF is set in the case of a CG model for which correction processing (correction control) is not canceled.

That is, when the correction control cancellation target 726 is ON and the target 705 is CG, the correction processing is canceled when the HMD 101 is at a predetermined position, and the image displayed by the HMD 101 is blurred. An image is displayed on the external display 160 .

In step S501, the setting information of FIG. 7 is read into the memory, and the HMD 101 is activated. When the HMD 101 is activated, the process moves to step S502.

In step S502, the function of the right-eye/left-eye video camera 221 is used to start capturing a real image.

In step S<b>503 , the right-eye/left-eye video camera 221 outputs the captured images of the physical space (the image captured by the right-eye camera and the image captured by the left-eye camera) to the PC 100 . Note that imaging and transmission are repeatedly performed, and MR images are sequentially received from the PC 100 and displayed on the right-eye/left-eye display 222 of the HMD 101 .

In step S504, the CPU 201 receives the physical space image output from the right-eye/left-eye video camera 221 via the general-purpose bus 212. In step S505, the CPU 201 stores the received physical space image in the RAM 203 or the external memory 211. store in

In step S506, the CPU 201 acquires the measurement result of the optical marker 103 by the optical sensor 104 via the communication I/F controller 208, and determines the position and orientation of the user's viewpoint based on the acquired measurement result as described above. (the position and orientation of the HMD 101) is obtained. The position and orientation of the user viewpoint obtained in step S506 are the position and orientation of the right viewpoint (video camera for right eye) and the position and orientation of the left viewpoint (video camera for left eye). As described above, the right viewpoint is obtained by converting the position and orientation of the optical marker 103 using the "relative position and orientation relationship between the right viewpoint and the optical marker 103" obtained in advance. can be done. Similarly, for the left viewpoint, the position and orientation of the optical marker 103 are converted using the "relative position and orientation relationship between the left viewpoint and the optical marker 103" obtained in advance. can be found at

In step S<b>507 , the CPU 201 stores the viewpoint position and orientation obtained in step S<b>506 in the external memory 211 in association with the identification information (ID) of the HMD 101 . For example, as shown in HMD information 700 in FIG. 7, ID, position, and orientation are associated and managed for each HMD.

In step S508, the CPU 201 reads the drawing data of the virtual object stored in the external memory 211 into the RAM 203 based on the CG model information 720 of FIG. 7, generates a virtual object based on the read drawing data, The generated virtual object is arranged in the virtual space. Then, the CPU 201 generates an image of the virtual object seen from the viewpoint based on the position and orientation of the viewpoint obtained in step S506. Then, the CPU 201 generates a mixed reality space image (MR image) by superimposing the image of the virtual object on the image of the physical space received in step S504. Strictly speaking, the CPU 201 superimposes the image of the virtual object generated based on the position and orientation of the right viewpoint on the image captured by the camera for the right eye (the image of the real space), thereby obtaining the mixed reality space image for the right eye. is generated, and an image of a virtual object generated based on the position and orientation of the left viewpoint is superimposed on the image captured by the camera for the left eye (image of the real space) to create an image of the mixed reality space for the left eye. Generate.

Step S508 represents an example of generation processing for generating a synthesized image obtained by synthesizing images of virtual objects displayed according to the position and orientation of the viewpoint of the experiencer.

In step S<b>509 , the CPU 201 determines whether or not to display the MR image generated in step S<b>508 on the external display based on the setting of the presence/absence of external display display 704 . If it is ON, the process proceeds to step S510, and if it is OFF, the process proceeds to step S514.

In step S<b>510 , the CPU 201 performs correction control when outputting to the external display 160 . Details of the correction control process will be described in detail with reference to FIG.

In step S<b>511 , the CPU 201 outputs MR images to the external display 160 via the video controller 206 . This MR image may be an image on which blur correction processing has been performed, or an image on which blur correction processing has not been performed (cancelled).

Step S511 shows an example of a second output process for outputting a synthesized image that has undergone blur correction using the synthesized image to the second display device. Also, it shows an example of second output processing for outputting the composite image generated in step S508 to the second display device when the conditions for blur correction are satisfied based on the position of the user. Further, if the distance to the object is not satisfied, the blur-corrected synthetic image is output to the second display device, and if the distance to the object is satisfied , the blur correction is canceled and the image generated in step S508 is displayed. The combined image is output to the second display device.

At step S512, the external display 160 receives the MR image from the PC 100, and at step S513 displays the received MR image.

Examples displayed on the external display 160 in step S513 are shown in FIGS. 8 and 9. FIG.

FIG. 8 shows a state in which the user A is viewing the CG 802 for which the correction control is to be canceled, but is outside the distance of the set value 706 (the conditions are not satisfied), so the blur correction process is performed and displayed.

FIG. 9 shows a state in which the user A is looking at the CG 902 for which correction control is to be canceled, and is within the distance of the set value 706 (which satisfies the criteria), so the blur correction processing is canceled and displayed. The dotted line CG of 902 shows the model displayed by blurring. Note that the dotted line display of 903 is for convenience of explanation of blurring, and is not limited to the dotted line display.

In addition, like 901, in order to call attention to the user who is looking at the external display 160 other than the experiencer, "The same image as the experiencer is displayed. There is no blur correction processing, so it may make you feel sick. Please be careful." is displayed. This display is information output from the PC 100 when the correction control is cancelled. Also, the display of 901 can be arbitrarily set by the setting information (not shown) in FIG.

In step S<b>514 , the CPU 201 outputs the MR images (right-eye mixed reality space image, left-eye mixed reality space image) generated in step S<b>508 to the HMD 101 via the video controller 206 .

Step S514 shows an example of first output processing for outputting the synthesized image generated in step S508 to the first display device used by the user.

In step S515, the right-eye/left-eye display 222 receives the mixed reality space image for the right eye and the mixed reality space image for the left eye transmitted from the PC 100 in step S514. Then, in step S516, the right-eye/left-eye display 222 displays the right-eye mixed reality space image on the right-eye display.

Next, the correction control process of step S510 will be described according to the flowchart of FIG.

In step S601, the CPU 201 identifies the HMD 101 of another experiencer or the target CG based on the setting information in FIG. In addition, in the case of the HMD 101 of another experiencer, it is desirable that the image is included in the captured image of the other experiencer. In other words, when conditions such as set values are satisfied while other users are watching, the blur correction control is canceled.

In step S602, the CPU 201 calculates and acquires the distance from the HMD 101 of another experiencer or the target CG. The distance can be calculated from the position of the target experiencer's HMD 101 and the position of another experiencer's HMD, or from the position of the target experiencer's HMD 101 and the position of the CG.

In step S<b>603 , the CPU 201 determines whether or not the acquired distance satisfies the distance of the setting value 706 . Specifically, it is determined whether or not the target experiencer's HMD 101 exists at a position farther than the distance of the set value 706 . If it is at a distant position, the condition is not satisfied, and the process proceeds to step S607. If the position is close, the condition is satisfied, and the process proceeds to step S604. That is, the above conditions are conditions related to blur correction, and conditions for canceling blur correction. It is also the distance between the position of the experiencer and the object.

In step S604, the CPU 201 determines whether or not blur correction has been cancelled. That is, when the blur correction state changes to the state in which the blur correction is not performed (released state), the process proceeds to step S605. If it is already in the canceled state, the process moves to step S606.

In step S605, since blur correction is normally executed, when blur correction is canceled, the CPU 201, in order to alert users other than the user that the correction process has been canceled, Information that blur correction is not performed is output (notified) to the external display 160 . An example of alerting is 901 in FIG.

In the present embodiment, control is performed so as to call attention when the blur correction is canceled. good too.

In addition, it goes without saying that when a warning is displayed and the timing for blur correction has come, control is performed to cancel the display of the warning.

Step S605 is an example of notification processing for notifying that blur correction has not been performed when the conditions related to blur correction are satisfied based on the position of the user and the composite image generated in step S508 is output to the second display device. is shown.

In step S606, the CPU 201 acquires the MR image (MR image without correction processing) generated in step S508.

In step S607, the correction module 405 performs blur correction using the MR image generated in step S508.

In step S608, the CPU 201 acquires the MR image corrected in step S607.

The MR image acquired in step S606 or step S607 is output in step S511.

As described above, according to the present embodiment, depending on the conditions of the HMD 101, the presence or absence of the blur correction process is switched to display on the external display, so that the system is easy to use. In particular, watching a blurred image for a long time can be reduced and motion sickness can be prevented.

As another embodiment, the following forms are also conceivable. In the above embodiment, the blur correction process is canceled when the condition is met. In other embodiments, the blur correction process is canceled when the condition is met (for example, the condition that the HMD 101 is out of the set distance). is executed, and the blur correction process is canceled when the condition is not satisfied (for example, the condition that the HMD 101 is within the set distance is satisfied).

In this way, it suffices to switch between execution and cancellation of blur correction processing, and the criterion for satisfying the conditions (whether the within-distance or out-of-distance criterion) may be either.

Finally, a functional configuration example of the PC 100 will be described with reference to the block diagram of FIG. Note that each functional unit in FIG. 3 may be implemented by hardware or may be implemented by software (computer program). In the former case, by installing such hardware in the HMD 101, for example, the HMD 101 can also execute each of the processes described above as those performed by the PC 100. FIG. In the latter case, such a computer program is stored in the external memory 211, and the CPU 201 appropriately reads out the computer program to the RAM 203 and executes it, thereby realizing the functions of the corresponding functional units. Note that the PC 100 can also be rephrased as an image processing device.

The generating unit 301 is a functional unit that generates a synthesized image by synthesizing images of virtual objects displayed according to the position and orientation of the viewpoint of the user. It is desirable that the image is a mixed reality space image obtained by synthesizing a captured real image and CG.

The first output unit 302 is a functional unit that outputs the synthesized image generated by the generation unit 301 to the first display device (for example, the HMD 101) used by the user.

The second output unit 303 is a functional unit that outputs a synthesized image that has undergone blur correction using the synthesized image to a second display device (eg, external display 160). In addition, the second output unit 303 outputs the synthesized image generated by the generation unit 301 based on the position and orientation of the user when the conditions related to blur correction (specifically, the conditions for canceling the blur correction) are satisfied. This is a functional unit that outputs to the second display device.

The notification unit 304 notifies that the blur correction has not been performed when the condition for canceling the blur correction is satisfied based on the position and orientation of the experiencer and the composite image generated by the generation unit 301 is output to the second display device. It is a functional part. An example of notification is 901.

The condition for canceling blur correction is the distance between the position and orientation of the user and the object.

Further, the second output unit 303 outputs the blur-corrected synthetic image to the second display device when the distance to the object is not satisfied , and cancels the blur correction when the distance to the object is satisfied. Then, the synthesized image generated by the generation unit 301 is output to the second display device.

In other words, the image processing apparatus according to the present embodiment, when displaying a synthesized image on an external display, outputs a synthesized image that is not shake-corrected according to the position of the user, thereby enhancing convenience in the form of verification or the like. can. In particular, in the case of motions that are unnecessary for verification, blur correction is performed, and in cases where verification is necessary (accurate display is required), blur correction is not performed. Verification from the outside can be facilitated while reducing poor physical condition.

Note that the external display may be any device as long as it is a display device used by a person other than the user.

In the above-described embodiment, the HMD 101 and the PC 100 are separate devices. Each of the processes described above may also be performed.

Further, in the above-described embodiment, the HMD 101 is described as a video see-through type head-mounted display device. However, an optical see-through type head-mounted display device may be used as the HMD 101 .

In addition to or instead of the HMD 101 worn on the user's head, an image of the mixed reality space based on the user's viewpoint is output to another display device, such as a tablet terminal device or a mobile terminal device such as a smartphone. You can do it. In this case, the mobile terminal device can acquire the position and orientation of the mobile terminal device as the user's viewpoint by using the sensors and applications provided in the mobile terminal device. The image of the mixed reality space based on the user's viewpoint may be generated by the PC 100 or may be generated by the mobile terminal device.

Further, in the above-described embodiment, a mixed reality presentation system (MR system) that presents the user with a mixed reality space that combines the real space and the virtual space has been described as an example, but only the virtual space is presented to the user. It may be applied to a so-called VR system. In that case, processing related to imaging of the real space and synthesis with the image of the real space are unnecessary.

In the above-described embodiment, one HMD 101 is connected to one PC 100 for the sake of simplicity of explanation. Also good.

Although the embodiments of the present invention have been described above, the present invention can be embodied as, for example, systems, devices, methods, programs, recording media, and the like. Specifically, it may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

Further, the program in the present invention is a program that allows a computer to execute the processing method of the flowchart, and the storage medium of the present invention stores the program that allows the computer to execute the processing method.

As described above, a recording medium recording a program for realizing the functions of the above-described embodiments is supplied to a system or device, and the computer (or CPU or MPU) of the system or device reads the program stored in the recording medium. Needless to say, the object of the present invention can also be achieved by reading and executing.

In this case, the program itself read from the recording medium implements the novel functions of the present invention, and the recording medium storing the program constitutes the present invention.

Examples of recording media for supplying programs include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon A disk, solid state drive, or the like can be used.

Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also based on the instructions of the program, the OS (operating system) and the like running on the computer are actually executed. Needless to say, a case where part or all of the processing is performed and the functions of the above-described embodiments are realized by the processing are included.

Furthermore, after the program read from the recording medium is written in the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is read according to the instruction of the program code. It goes without saying that a case where a CPU or the like provided in a function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

Moreover, the present invention may be applied to a system composed of a plurality of devices or to an apparatus composed of a single device. Further, it goes without saying that the present invention can be applied to the case where it is achieved by supplying a program to a system or apparatus. In this case, by loading a recording medium storing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

Furthermore, by downloading and reading out the program for achieving the present invention from a server, database, etc. on the communication line using the communication program, the system or apparatus can enjoy the effects of the present invention.

Some or all of the embodiments described above may be used in combination as appropriate, or some or all of the embodiments described above may be selectively used.

100 PC 101 HMD 103 Optical Marker 102 Two-Dimensional Marker 104 Optical Sensor 130 CG Model 160 External Display

Claims (22)

controlling a specific image to be output to the first display means without performing specific blur correction for correcting image shift;
a control means for controlling the specific image to be subjected to the specific blur correction and output to a second display means;
An image processing apparatus, wherein the specific image is an image including a virtual object and a photographed image. controlling a specific image to be output to the first display means without performing specific blur correction for correcting image shift;
a control means for controlling the specific image to be subjected to the specific blur correction and output to a second display means;
An image processing apparatus, wherein the image is a composite image of a virtual object and a photographed image. controlling a specific image to be output to the first display means without performing specific blur correction for correcting image shift;
a control means for controlling the specific image to be subjected to the specific blur correction and output to a second display means;
The control means outputs the specific image to the second display means without applying the specific blur correction to the specific image when the position of the apparatus having the first display means satisfies a predetermined condition. An image processing apparatus characterized by controlling to controlling a specific image to be output to the first display means without performing specific blur correction for correcting image shift;
a control means for controlling the specific image to be subjected to the specific blur correction and output to a second display means;
When the specific image is output to the second display means without being subjected to the specific blur correction, the control means controls to notify that the specific blur correction has not been performed. An image processing device characterized by: The control means controls the first display means to output an image based on the specific image, which is an image generated based on the position of the device provided with the first display means. The image processing apparatus according to any one of claims 1 to 4, characterized by 5. The image processing apparatus according to claim 2, wherein said specific image is an image containing a virtual object. 7. The image processing apparatus according to claim 2, wherein said specific image is an image including a photographed image. 8. The image processing apparatus according to claim 1, wherein said photographed image is an image photographed by image pickup means provided in an apparatus having said first display means. 5. The image processing apparatus according to claim 3, wherein said image is a composite image of a virtual object and a photographed image. 10. The image processing apparatus according to any one of claims 1 to 9, wherein the specific blur correction is a process of correcting image shift of the specific image by shifting pixels. 11. The image processing apparatus according to any one of claims 1 to 10, wherein the specific shake correction is a process of performing correction based on a result of comparing before and after the specific image. 12. The image processing apparatus according to any one of claims 1 to 11, wherein the specific blur correction is electronic blur correction. The control means outputs the specific image to the second display means without applying the specific blur correction to the specific image when the position of the apparatus having the first display means satisfies a predetermined condition. 5. The image processing apparatus according to claim 4, wherein the control is performed such that 14. The image processing apparatus according to claim 3, wherein said predetermined condition is a condition relating to a distance between a device having said first display means and an object. When the distance between the device having the first display means and the object does not satisfy the predetermined condition, the control means applies the specific blur correction to the specific image to display the image on the second display means. and when the distance between the device having the first display means and the object satisfies the predetermined condition, the specific image is displayed on the second display means without performing the specific blur correction. 15. The image processing apparatus according to claim 14, wherein control is performed so as to output. 16. The image processing apparatus according to any one of claims 1 to 15, wherein said first display means is provided in a head mounted display. 17. The image processing apparatus according to claim 16, wherein said second display means is a display means of a device used by a person other than a person who has experienced said head-mounted display. controlling a specific image to be output to the first display means without performing specific blur correction for correcting image shift;
a control step of performing control so that the specific image is subjected to the specific blur correction and output to a second display means;
A control method for an image processing device, wherein the specific image is an image including a virtual object and a photographed image. controlling a specific image to be output to the first display means without performing specific blur correction for correcting image shift;
a control step of performing control so that the specific image is subjected to the specific blur correction and output to a second display means;
A control method for an image processing device, wherein the image is a composite image of a virtual object and a photographed image. controlling a specific image to be output to the first display means without performing specific blur correction for correcting image shift;
a control step of performing control so that the specific image is subjected to the specific blur correction and output to a second display means;
In the control step, when the position of the device having the first display means satisfies a predetermined condition, the specific image is output to the second display means without being subjected to the specific shake correction. A control method for an image processing apparatus, characterized by controlling to controlling a specific image to be output to the first display means without performing specific blur correction for correcting image shift;
a control step of performing control so that the specific image is subjected to the specific blur correction and output to a second display means;
In the control step, when the specific image is output to the second display means without being subjected to the specific blur correction, it is controlled to notify that the specific blur correction has not been performed. A control method for an image processing apparatus characterized by: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 17.

Images