JP2011010126A - Image processing apparatus, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology for switching an image to be displayed on an HMD (Head-Mounted Display) or a display device other than the HMD, in accordance with the situation when displaying the image.SOLUTION: A mixed-reality image receiving section 151 captures a right-eye image and a left-eye image from an image processing apparatus 900. Display sections 102L, 102R display the left-eye image and the right-eye image, respectively. An image control section 105 determines, as an object to be output to a display device 30, any one of a panoramic image obtained by concatenating the left- and right-eye images horizontally arranged, the right-eye image and the left-eye image.

Description

  The present invention relates to an image display technique.

  Mixed reality technology, so-called MR (Mixed Reality) technology, that makes it appear as if an object (virtual object) drawn in CG exists in the real world is widely known. MR technology is realized by superimposing a real-space image and a virtual-space image generated by computer graphics (CG) modeled three-dimensionally and aligning and displaying each image. .

  A mixed reality presentation system using this technology is composed of the following devices. Real image capturing apparatus (for example, video camera) that captures the real world, and an image processing apparatus that generates a virtual space image as viewed from an imaging position in the real world and generates a mixed reality space image by combining the images. An image display device for displaying a mixed reality space image. The image processing apparatus is typically represented by a PC (personal computer), a workstation, or the like.

  As one of the mixed reality presentation systems described above, a system using a video see-through head mounted display (HMD = head mounted display device) is known. In this system, an object is imaged by an imaging unit in which the line of sight from the pupil position of the HMD wearer and the shooting direction are substantially matched, and a mixed reality space image generated using the taken image is provided to the HMD wearer as a stereoscopic image. To do.

  Here, the internal configuration of the video see-through HMD will be described with reference to FIG. In FIG. 10, reference numeral 101 denotes an imaging unit that images the real world, and reference numeral 102 denotes a display unit that displays an image. Reference numeral 15 denotes an imaging optical system including a lens for guiding the luminous flux of external light to the imaging device and forming an image on the imaging unit 101. Reference numeral 16 denotes an image displayed on the display unit 102, and the pupil 17 of the HMD wearer. Is a display optical system (free-form surface prism).

  With such a configuration, the imaging unit 101 performs imaging in the real space that is substantially coincident with the line-of-sight direction from the pupil position of the HMD wearer, and forms an image displayed on the display unit 102 on the pupil 17 of the HMD wearer. The mixed reality space image is presented to the HMD wearer.

  As a display device that displays a mixed reality space image, the wearer is suitable by using an HMD in which the line of sight from the wearer's pupil position substantially matches the imaging direction of the camera as described above, instead of a normal monitor. It is possible to project an image in the direction of the image on the HMD. Furthermore, since a CG image can be displayed when the wearer is facing the direction, it is possible to enhance the immersive feeling of the HMD wearer in the mixed reality space.

  Next, a basic configuration of the mixed reality system using the HMD will be described with reference to FIG. Such a system includes the following devices.

-HMD10 displaying mixed reality space image
-HMD controller 11 for controlling the HMD 10
An image processing apparatus 20 that generates a mixed reality space image by superimposing a virtual space image on a real space image captured by the HMD 10.
-Display device 30 for non-HMD wearers to share mixed reality experiences
In the configuration shown in FIG. 11, the HMD 10 and the HMD controller 11 are separate hardware, but each may be integrated by incorporating the HMD controller 11 in the HMD 10.

  The mixed reality presentation system displays the CG image and the real space image in a correct positional relationship even when the viewpoint position of the imaging unit of the HMD 10 changes, and for this purpose, the HMD 10 detects the viewpoint position and the line-of-sight position for detecting the line-of-sight direction. A posture detection mechanism (for example, a position and posture sensor) is provided.

  The image processing apparatus 20 places a virtual object that has been three-dimensionally modeled in a virtual space having the same scale as the real space, and is assumed to be observed from a viewpoint position or a line-of-sight direction detected by a position and orientation detection unit (not shown). Render the object and generate a CG image. By superimposing the CG image generated in this way and the real space image, the virtual object is correctly placed in the real space regardless of the viewpoint position and the line of sight observed by the imaging unit. Mixed reality space images can be generated and presented. With this configuration, the HMD wearer can experience the mixed reality world in which the real space image and the virtual space image (CG image) are superimposed through the HMD 10.

  In addition, there is an HMD controller 11 having an external output interface that can output the same image as the image viewed by the HMD wearer (an image viewed from the viewpoint of the HMD wearer) to the external display device 30.

  The HMD controller 11 having an external output interface outputs either the left-eye image or the right-eye image output from the image processing apparatus 20 to the HMD 10 to the external output interface using a selector. With such a configuration, an image that the HMD wearer is observing can be provided to the audience other than the HMD wearer through the display device 30, so that the mixed reality experienced by the HMD wearer can be displayed in a plurality of ways. It is possible for the audience to understand and share at the same time.

  However, in the HMD 10 capable of outputting an image to the spectator display device 30 described above, even when the mixed reality experience person returns the HMD 10 to a dedicated place or a hook after the mixed reality experience ends, the display device 30 Will continue to display. In this situation, since the image pickup unit attached to the HMD 10 continues to pick up meaningless images, a meaningless (insignificant for the audience) image is displayed on the display device 30 and the effect of the exhibition is diminished. There was a problem.

  On the other hand, a display control method for switching an image to be displayed on the audience display device 30 using a use state determination technique for determining whether the HMD 10 is used based on position and orientation information of the HMD 10 has been proposed. (Patent Document 1). In such a display control method, when the HMD 10 is used, the mixed reality space image input to the HMD 10 is displayed on the display device 30, and when the HMD 10 is not used, an alternative image that conveys a message that the HMD 10 is not used is displayed. 30. This makes it possible to increase the effect of the exhibition.

  The HMD 10 generally displays a pair of left and right stereo images having overlapping portions, but the display device 30 can display only the left-eye image or the right-eye image among the stereo images. Therefore, there is a problem that the observation area is different between the HMD wearer and the audience watching the display on the display device 30, and the observation images of both do not match.

  To solve this problem, by generating a two-dimensional image from a pair of left and right stereo images having overlapping portions, the image areas observed by the HMD wearer and the audience other than the HMD wearer match, and a mixed reality space image It is possible to further increase the feeling of sharing (Patent Document 2).

  The presentation of the mixed reality is performed by instructing the activation of the mixed reality presentation application on the operation screen (not shown) of the image processing apparatus 20 and operating it. As this application occupies the entire operation screen and operates, a mixed reality space image is displayed on the display unit of the HMD 10. When the mixed reality presentation is interrupted and various settings of the mixed reality presentation application and the HMD 10 and various operations of the image processing apparatus 20 are performed, the operation screen of the image processing apparatus 20 needs to be displayed. Therefore, the mixed reality presentation system having the display device 30 originally requires at least two display devices, that is, the display device 30 and the display device for operation of the image processing device 20. However, by using the display device 30 also as a display device for operating the image processing device 20, the number of display devices required for the system can be reduced by one.

  Next, when the wearer of the HMD 10 experiences mixed reality, the display of the mixed reality image displayed on the display unit of the HMD 10 on the display device 30 will be described with reference to FIG. explain. In FIG. 12A, 1201L and 1201R are a display unit for the left eye and a display unit for the right eye respectively provided in the HMD 10, and each display unit includes a desk 200 as a real object and a virtual object as a CG. 201, 202, and 203 are displayed. Such display enables display as if the virtual objects 201, 202, and 203 are arranged on the desk 200 as a real object. FIGS. 12B and 12C show the display screen of the display device 30 displaying the image displayed on the left-eye display unit 1201L and the image displayed on the right-eye display unit 1201R, respectively. The display screen of the display device 30 is shown.

  On the other hand, an operation screen displayed on the display device 30 when various operations of the image processing apparatus 20 are performed will be described with reference to FIG. As shown in FIG. 13A, windows 210 and 211 running on the image processing apparatus 20 are displayed across the display unit 1201L for the left eye and the display unit 1201R for the right eye. FIGS. 13B and 13C show the display screen of the display device 30 displaying the image displayed on the left-eye display unit 1201L and the image displayed on the right-eye display unit 1201R, respectively. The display screen of the display device 30 is shown.

  Here, it will be described with reference to FIGS. 12 and 13 that the operation of the image processing apparatus 20 is complicated. When experiencing mixed reality, as shown in FIG. 12A, a mixed reality image corresponding to the position and orientation of the HMD 10 is displayed on the display unit 1201R for the right eye and the display unit 1201L for the left eye. ing. On the other hand, as shown in FIGS. 12B and 12C, the mixed reality image displayed on one of the display unit 1201R for the right eye and the display unit 1201L for the left eye is displayed on the display device 30 as an HMD controller. 11 is selected and displayed. Thereby, the display which can share the mixed reality experience of the wearer of HMD10 with the audience other than the HMD wearer is made.

  On the other hand, when various operations of the image processing device 20 are performed, as shown in FIG. 13A, the operation screen of the image processing device 20 is a display screen of the display unit 1201R for the right eye and a display screen of the display unit 1201L for the left eye. Across both display screens. Since only the image displayed on either the left-eye display unit 1201L or the right-eye display unit 1201R can be displayed on the display device 30, as shown in FIGS. 13B and 13C, the left side of the operation screen is displayed. Or, only the right half is displayed. Therefore, the operator who views the display screen of the display device 30 cannot view the entire operation screen at once.

  Since only one half of the operation screen of the image processing device 20 is displayed on the display device 30, it is necessary to switch the output to the display device 30 with the HMD controller 11 in order to perform an operation on the other operation screen. there were. The conventional HMD 10 and the HMD controller 11 cannot solve the problem that the operation of the image processing apparatus 20 as described above is troublesome.

JP 2004-258123 A JP 2006-41795 A

  Since the operation screen covers the entire area of both the right-eye image and the left-eye image output to the HMD 10, only one of the right-eye image and the left-eye image is output to the display device 30. Only the left side or the right side of the operation screen is displayed on the display device 30. In order to display the other screen, it is necessary to switch the output of the selector. At the same time, the entire operation screen cannot be viewed, and the operation of the image processing apparatus 20 is troublesome.

  The present invention has been made in view of the above problems, and provides a technique for switching an image to be displayed on a display device according to a situation when an image is displayed on an HMD and a display device other than the HMD. With the goal.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement. That is, an acquisition unit that acquires an image for the right eye and an image for the left eye from an external device, a unit that displays the image for the right eye on the display unit for the right eye, and a unit that displays the image for the left eye on the display unit for the left eye Generating means for generating a single panoramic image by combining the left-eye image and the right-eye image side by side horizontally, and any of the right-eye image, the left-eye image, and the panoramic image Transmission means for selecting one of the images and transmitting the selected image to an external display device.

  According to the configuration of the present invention, when an image is displayed on the HMD and a display device other than the HMD, an image to be displayed on the display device can be switched depending on the situation.

The block diagram which shows the function structural example of the mixed reality presentation system which concerns on 1st Embodiment. The flowchart of the process which HMD800 performs. FIG. 6 is a diagram showing a display example on the display device 30. FIG. 6 is a diagram showing a display example on the display device 30. The block diagram which shows the function structural example of the mixed reality presentation system which concerns on 2nd Embodiment. The flowchart of the process which HMD801 performs. The flowchart of the process in step S901. The block diagram which shows the function structural example of the mixed reality presentation system which concerns on 3rd Embodiment. The flowchart of the process which concerns on 3rd Embodiment. The figure which shows the internal structure of a video see-through type HMD. The figure which shows the basic composition of the mixed reality system using HMD. FIG. 6 is a diagram showing a display example on the display device 30. FIG. 6 is a diagram showing a display example on the display device 30.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
The mixed reality presentation system according to the present embodiment outputs a mixed reality space image, which is a composite image of a real space image and a virtual space image, to the HMD, and also to a display device provided for a person other than the HMD wearer. Output. In such a configuration, the mixed reality presentation system switches the image to be output to the display device according to the situation. Hereinafter, the mixed reality presentation system according to the present embodiment will be described in more detail. In addition, in the reference numbers appearing in the drawings used in the following description, the reference numbers to which L and R are attached indicate components for the left eye and the right eye, respectively.

  First, a functional configuration example of the mixed reality presentation system according to the present embodiment will be described with reference to FIG. As shown in FIG. 1, the mixed reality presentation system according to the present embodiment includes a video see-through type HMD 800 as a head-mounted display device, an image processing device 900 represented by a PC (personal computer), a CRT, and a liquid crystal screen. And the like.

  First, the HMD 800 will be described. The imaging units 101L and 101R are for capturing a real space image provided to the left eye and the right eye of a user who wears the HMD 800 on the head, for example, a camera for capturing a moving image such as a video camera. . When the imaging units 101L and 101R capture a moving image in the real space, the imaging units 101L and 101R sequentially transmit the images of each frame (real space image) constituting the captured moving image to the subsequent captured image transmission unit 150. Hereinafter, the captured image captured by the image capturing unit 101L is referred to as “a left-eye real space image (captured image)”, and the captured image captured by the image capturing unit 101R is referred to as a “right-eye real space image (captured image)”. There is a case.

  The captured image transmission unit 150 transmits the captured images (the real space image for the left eye and the real space image for the right eye) received from the image capture units 101L and 101R to the image processing apparatus 900. The HMD 800 and the image processing apparatus 900 may communicate with each other by wired or wireless. The sensor unit 103 measures its own position and orientation as the position and orientation of the HMD 800 and transmits position and orientation information obtained by the measurement to the image processing apparatus 900.

  The mixed reality image receiving unit 151 receives the left-eye image and the right-eye image transmitted from the image processing device 900 (external device), sends them to the display units 102L and 102R, and sends them to the image control unit 105. Is also sent out. The right-eye image is a mixed reality space image for the right eye generated by processing described later on the image processing apparatus 900 side, or screen information of an area to be displayed on the display unit 102R in the operation screen in the image processing apparatus 900. Similarly, the left-eye image is a left-eye mixed reality space image generated by processing described later on the image processing apparatus 900 side, or screen information of an area displayed on the display unit 102L in the operation screen of the image processing apparatus 900. is there.

  The display units 102L and 102R are for presenting images to the user's left eye and right eye, respectively. When the user wears the HMD 800 on his / her head, the display units 102L and 102R are positioned in front of the user's left eye and right eye. And attached to the HMD 800. The display units 102L and 102R display the left-eye image and the right-eye image received by the mixed reality image receiving unit 151, respectively. As a result, the right-eye image and the left-eye image are displayed in front of the user's right eye and left eye, respectively.

  The instruction unit 106 is attached to the HMD 800 in order to selectively input a plurality of types of instructions such as switches. An operation instruction input by the user using the instruction unit 106 is transmitted from the instruction unit 106 to the image control unit. 105 is input. The instruction unit 106 may be a separate device from the HMD 800 without being attached to the HMD 800.

  In response to an instruction input from the instruction unit 106, the image control unit 105 selects one of the following three images as an output target to the object composition unit 107.

A single panoramic image obtained by combining the images (left-eye image and right-eye image) displayed on each of the display units 102L and 102R in the horizontal direction on the left and right. Left-eye image Right-eye image The image control unit 105 sends the selected image to the object composition unit 107. The object composition unit 107 composes a predetermined object such as information related to use of the HMD 800 with the image received from the image control unit 105, and sends the composited image to the display device image transmission unit 152. Of course, the composition of this object may be appropriately performed according to the user instruction or various situations, and may not always be performed. The display device image transmission unit 152 transmits an image signal based on the image received from the object composition unit 107 to the display device 30.

  The control unit 100 is for performing operation control of the above-described units constituting the HMD 800, and includes an execution control unit such as a CPU, a memory unit such as a RAM and a ROM, and the like. The execution control unit performs operation control of each unit constituting the HMD 800 using a computer program and data stored (or stored) in the memory unit.

  Next, the image processing apparatus 900 will be described. The captured image reception unit 153 receives the real space image for the right eye and the real space image for the left eye transmitted from the captured image transmission unit 150 on the HMD 800 side, and the captured image feature detection unit 120 in the subsequent stage receives each received image. To send.

  The captured image feature detection unit 120 performs the following processing on the real space image for the right eye and the real space image for the left eye. That is, feature points (for example, feature quantities such as artificial markers and natural features) are detected from the real space image. Then, the captured image feature detection unit 120 includes information on the feature points detected from each of the real space images together with the real space image for the right eye and the real space image for the left eye (the position and size of the feature points in the real space image, Information indicating the type) is sent to the mixed reality image generation unit 122.

  When the position and orientation detection unit 121 receives the position and orientation information transmitted from the sensor unit 103 on the HMD 800 side, the position and orientation detection unit 121 sends the received position and orientation information to the subsequent mixed reality image generation unit 122.

  The mixed reality image generation unit 122 corrects the position and orientation information by using the information related to the feature points received from the captured image feature detection unit 120 and the position and orientation information received from the position and orientation detection unit 121, and more. Highly accurate position and orientation information is generated. Since this technique is a well-known technique, a description thereof will be omitted. Note that the position and orientation information obtained here includes left-eye position and orientation information and right-eye position and orientation information, and each corrects the position and orientation relationship between the sensor unit 103 and the imaging units 101L and 101R. It can be obtained by adding to the subsequent position and orientation information. Since this is also a well-known technique, a description thereof will be omitted.

  The mixed reality image generation unit 122 generates an image of the virtual space viewed from the viewpoint having the position and orientation indicated by the position and orientation information, using the position and orientation information corrected in this way. More specifically, using the right-eye position and orientation information, a virtual space image viewed from the viewpoint (right eye) having the position and orientation indicated by the position and orientation information is generated as the right-eye virtual space image. Further, using the position and orientation information for the left eye, an image of the virtual space viewed from the viewpoint (left eye) having the position and orientation indicated by the position and orientation information is generated as the virtual space image for the left eye. Of course, it is assumed that various virtual objects are arranged in advance in this virtual space.

  The mixed reality image generation unit 122 combines the right-eye real space image received from the captured image feature detection unit 120 and the generated right-eye virtual space image to generate a right-eye mixed reality space image. . The mixed reality image generation unit 122 combines the left-eye real space image received from the captured image feature detection unit 120 and the generated left-eye virtual space image to generate a left-eye mixed reality space image. . There are various techniques for generating a mixed reality space image by combining a real space image and a virtual space image, and any technique may be used in the present embodiment.

  Then, the mixed reality image generation unit 122 sends the mixed reality space image for the left eye and the mixed reality space image for the right eye generated in this way to the mixed reality image transmission unit 154. The mixed reality image transmission unit 154 transmits the mixed reality space image for the right eye and the mixed reality space image for the left eye received from the mixed reality image generation unit 122 to the HMD 800 as a right eye image and a left eye image, respectively. To do.

  Note that the mixed reality image generation unit 122 does not generate the mixed reality space image, but transmits the real space images for the right eye and the left eye to the mixed reality image transmission unit 154. The mixed reality image transmission unit 154 A real space image may be sent to the HMD 800. This is realized by switching settings on the image processing apparatus 900 side. In the following description, the process of transmitting the mixed reality space image from the image processing apparatus 900 to the HMD 800 also indicates the process of transmitting the reality space image from the image processing apparatus 900 to the HMD 800.

  Here, it is assumed that the mixed reality presentation application (application for generating and transmitting a mixed reality image) executed by the image processing apparatus 900 is terminated or the system setting screen (operation screen) is shifted to. In this case, the screen information of the operation screen at this time is transmitted from the image processing device 900 (mixed reality image transmission unit 154) to the HMD 800. As described above, this screen information includes an area displayed on the display unit 102R and an area displayed on the display unit 102L, and is displayed over the entire display screen of each of the display units 102L and 102R. .

  In any case, information transmitted from the image processing apparatus 900 to the HMD 800 is image information (left-eye image and right-eye image) displayed on each of the display units 102L and 102R, and the image information is mixed reality. It may be a feeling image or screen information.

  With the HMD 800 and the image processing apparatus 900 described above, a user wearing the HMD 800 on the head can observe a mixed reality space image according to his / her movement. In the present embodiment, a single image processing apparatus 900 generates a right-eye image and a left-eye image. However, an image processing device for generating a right-eye image and an image processing device for generating a left-eye image may be used instead of the image processing device 900. In this case, the two image processing apparatuses are connected to the HMD 800, and each image processing apparatus transmits the generated image to the HMD 800.

  Next, the display device 30 will be described. The display device 30 is configured by a CRT, a liquid crystal screen, or the like, and is arranged so that other users who are not wearing the HMD 800 can observe, and displays an image based on an image signal transmitted from the HMD 800. . That is, the display device 30 is provided for an audience not wearing the HMD 800. By displaying the same image as the image displayed on the display unit 102L and / or the display unit 102R of the HMD 800, the audience can see the same image as the image viewed by the HMD wearer. Share a real-life experience. The display device 30 is used for providing images to an audience other than the wearer of the HMD 800, and also used for displaying screen information generated by the image processing device 900.

  Next, processing performed by the HMD 800 to determine an image to be output to the display device 30 and output the determined image to the display device 30 will be described with reference to FIG. Note that FIG. 2 does not touch anything other than such processing. However, in the following, the processing other than the processing performed for outputting an image to the display device 30 is not mentioned.

  When the user wants to switch the image to be output to the display device 30, the user operates the instruction unit 106 to set it to either ON (HMD observation image display) or OFF (operation screen display). When the user confirms that the mixed reality image is displayed on the display units 102L and 102R, the instruction unit 106 is set to ON so that the mixed reality space image is also displayed on the display device 30. To do. On the other hand, when the user confirms that screen information such as an operation screen is displayed on the display units 102L and 102R, the instruction unit 106 is used to display the screen information on the display device 30. Set to.

  When the user operates the instruction unit 106 to set ON or OFF, the instruction unit 106 sends a selection instruction (indicating either ON or OFF) by the user to the image control unit 105 in step S500. .

  In step S501, the image control unit 105 determines whether the selection instruction from the instruction unit 106 indicates ON or OFF. If it is determined that the selection instruction indicates ON, the process proceeds to step S502. If the selection instruction indicates OFF, the process proceeds to step S507. In step S <b> 502, the image control unit 105 sets one of the following as an output target image to the display device 30.

A single panoramic image generated by overlapping and combining overlapping portions of the right-eye image and the left-eye image received from the mixed reality image receiving unit 151. Either of the right-eye image or the left-eye image. On the other hand, which of the images to be output to the display device 30 may be determined based on an input instruction from the instruction unit 106 or another device, or may be determined in advance and may be included in the HMD 800 as setting data. It may be registered in the illustrated memory.

  If the panoramic image is to be output to the display device 30, the process proceeds to step S506. If either the right-eye image or the left-eye image is to be output to the display device 30, the process is performed. The process proceeds to step S503.

  In step S <b> 506, the image control unit 105 overlaps the overlapping portions (overlapping image portions) of the right-eye image and the left-eye image received from the mixed reality image receiving unit 151, thereby combining one image. Generate a panoramic image. Since the processing in this step, that is, the two-dimensional image generation processing by the composite image generation method in which the overlap portion is overlapped from the pair of left and right stereo images is a well-known technique, description thereof will be omitted.

  On the other hand, in step S <b> 503, the image control unit 105 selects an instructed or predetermined one of the right-eye image and the left-eye image as an output target image to the display device 30.

  On the other hand, in step S507, the image control unit 105 generates a single panoramic image by arranging the left-eye image and the right-eye image horizontally in the horizontal direction and combining them without overlap. In the generation of such a panoramic image, pixels are thinned out or interpolated in both the horizontal direction and the vertical direction of the image, and each panoramic image is arranged in the horizontal direction and written in an image memory (not shown) to generate one panoramic image. . Of course, the method of generating the panoramic image is not limited to this.

  Next, in step S504, the image control unit 105 changes the resolution of the “image to be output to the display device 30” obtained in any of steps S506, S503, and S507 to a resolution that the display device 30 can display. Various image adjustment processes are performed.

  In step S <b> 505, the image control unit 105 inputs the image adjusted in step S <b> 504 to the object composition unit 107. The object composition unit 107 composes the above-described object with respect to this image. Then, the object composition unit 107 outputs an image signal of the image obtained by the composition to the display device 30.

  Next, a display example on the display device 30 will be described with reference to FIG. FIG. 3A shows a display unit when displaying either the right-eye mixed reality image as the right-eye image or the left-eye mixed reality space image as the left-eye image on the display device 30. 102L, 102R, display examples on the display device 30 are shown.

  The display units 102L and 102R display a mixed reality space image 35 for the left eye and a mixed reality image 36 for the right eye, respectively. Each of the mixed reality space image 35 for the left eye and the mixed reality image 36 for the right eye includes a desk 200 as a real object, virtual objects 201, 202, 203, and a region 620. An area 620 is an area of the object synthesized by the object synthesis unit 107, and is an area for displaying information related to the use of the HMD 800, for example, continuous use time information after the use of the HMD 800 is started. The display position, display size, display timing, and the like of the area 620 may be set in the HMD 800 in advance, or may be changed as appropriate by the object composition unit 107 according to a user instruction or the like. In this case, however, either one of the mixed reality image 36 for the right eye and the mixed reality space image 35 for the left eye is displayed on the display device 30.

  On the other hand, FIG. 3B shows a process of generating a single panoramic image by overlapping and combining overlapping portions in the mixed reality space images displayed on the display units 102L and 102R. Yes. The display units 102L and 102R respectively display a mixed reality space image 37 for the left eye and a mixed reality image 38 for the right eye, and both images include real objects 600, 601, 602, and 603. Yes, but there are parts of each image that overlap each other.

  Japanese Patent Application Laid-Open No. 10-178564 discloses a composite image creation apparatus for an image in which an overlap region does not completely match due to a difference in field of view or a movement of an imaging target. The image control unit 105 generates one panoramic image 611 from the mixed reality space image 37 for the left eye and the mixed reality image 38 for the right eye by a technique equivalent to the technique. The panorama image 611 is subjected to object synthesis, image size adjustment processing for the screen size of the display device 30, and the like, thereby generating an image 130 to be finally sent to the display device 30.

  In the present embodiment, the position, size, time, and timing of the area 620 are set in advance in the HMD 800, but the position and size are fixed, and the presence or absence of display is switched by a switch or the like provided in the object composition unit 107. You may do it. In addition, the information displayed in the area 620 may be a product logo of the HMD 800 or the present system, an exhibition logo, or the like.

  Next, a case where screen information displayed on both the display screen of the display unit 102L and the display screen of the display unit 102R is displayed on the display device 30 will be described with reference to FIG. Screen information (operation screen) is displayed across both screens of the display units 102L and 102R, and an image area 480 on the left half of the operation screen is displayed in the screen of the display unit 102L. An image area 490 on the right half of the operation screen is displayed in the screen. Reference numerals 210 and 211 respectively denote a mixed reality application window and other windows running on the image processing apparatus 900. The mixed reality application is not in a mixed reality presentation operation over the entire operation screen, but is in a window screen display state for performing various settings.

  When the user sets OFF using the instruction unit 106, the image control unit 105 combines the image region 480 (left eye image) and the image region 490 (right eye image) side by side to form one panoramic image. 701 is generated.

  The left side of the third row in FIG. 4 shows a display example on the display device 30 when the aspect ratio of the panoramic image 701 is maintained. The right side of the third row in FIG. 4 shows a display example on the display device 30 when the aspect ratio of the panoramic image 701 is not maintained. The fourth row in FIG. 4 shows a display example when a partial area that can be displayed by the display device 30 is displayed in the entire panoramic image 701 without changing the resolution of the panoramic image 701. The display area can be set by the HMD 800 or by a peripheral device. Thus, the display method of the panoramic image 701 may be switched as appropriate.

  In the area 710L, the left-eye image immediately before being set OFF by the instruction unit 106 is displayed, and in the area 710R, the right-eye image immediately before being set OFF by the instruction unit 106 is displayed. Of course, other information may be displayed instead as long as the information shows the same contents.

  Conventionally, when displaying the operation image of the image processing device on the display device, only one of the left-eye image and the right-eye image can be displayed on the display device. It was not displayed, and the operation of the image processing apparatus was troublesome. However, according to the present embodiment, since the entire operation image of the image processing apparatus can be displayed on the display device, the operability is improved.

[Second Embodiment]
In this embodiment, the use state determination unit 108 is used instead of the instruction unit 106 to switch the image to be sent to the object composition unit 107. However, the system according to the present embodiment is the same as that of the first embodiment except that the use state determination unit 108 is used instead of the instruction unit 106 as shown in FIG.

  The system according to the present embodiment will be described with reference to FIG. The use state determination unit 108 detects a change in the position of a feature point in the image using a physical space image captured by at least one of the imaging units 101L and 101R, and based on the detection result, the HMD 801 Determine if it is being used. When it is determined that the HMD 801 is currently used, it is determined that the HMD observation image is displayed, and when it is determined that the HMD 801 is not used, it is determined that the operation screen is displayed. The subsequent steps are the same as in the first embodiment.

  Next, processing performed by the HMD 801 to determine an image to be output to the display device 30 and output the determined image to the display device 30 will be described with reference to FIG. Note that FIG. 6 does not touch anything other than this processing. However, in the following, the processing other than the processing performed for outputting an image to the display device 30 is not mentioned.

  When the HMD 801 is activated, the control unit 100 first sets an HMD status initialization process, that is, sets the HMD status to “not used” in step S900. Here, the HMD status is a variable indicating whether or not the HMD 801 is used.

  Next, in step S901, the use state determination unit 108 determines whether or not the HMD 801 is currently used using a real space image captured by either one of the imaging units 101L and 101R. Note that this determination may use real space images from both the imaging units 101L and 101R, and in that case, the determination result is adopted only when the same determination result is obtained for both images.

  Then, the use state determination unit 108 reflects the determination result in the HMD status. That is, when it is determined that the HMD 801 is used, the HMD status is changed to “in use”, and when it is determined that the HMD 801 is not used, the HMD status is updated to “not used”. Details of the processing in step S901 will be described later with reference to FIG. It should be noted that there is no change in the HMD status except when the use of the HMD is started or when the use is interrupted.

  Next, in step S902, the use state determination unit 108 determines whether or not the HMD status has been changed in the process in step S901. As a result of the determination, if it has been changed, the process proceeds to step S903, and if it has not been changed, the process proceeds to step S904.

  In step S <b> 903, the use state determination unit 108 determines whether the changed HMD status is “in use” or “not in use”. As a result of the determination, if it is “in use”, the process proceeds to step S502, and if it is “not used”, the process proceeds to step S507. Here, for the sake of explanation, it is assumed that the display state of the display device 30 does not change unless an image is newly output to the display device 30. As long as the system termination instruction is not input or the system termination condition is not satisfied, the process returns to step S901 via step S904.

  Next, details of the use state determination process of the HMD 801 in step S901 will be described with reference to FIG. In the present embodiment, in order to determine whether or not the HMD 801 is used, feature points (features) in the real space image used for alignment in the mixed reality presentation, particularly markers that are artificial features Is used. However, in the present embodiment, it is assumed that a plurality of markers are arranged in the imaging space by the imaging units 101L and 101R. Of course, the feature points are not limited to artificially arranged markers, and natural features such as corners and sides of real objects may be used.

  First, in step S1000, the use state determination unit 108 acquires a real space image captured by at least one of the imaging units 101L and 101R. Next, in step S1001, the use state determination unit 108 detects a marker from the real space image acquired in step S1000.

  Next, in step S1002, the use state determination unit 108 determines whether one or more markers have been detected in step S1001. As a result of the determination, if one or more markers are detected, the process proceeds to step S1003. If no marker can be detected, the process proceeds to step S1008.

  In step S1008, the use state determination unit 108 determines whether or not a predetermined time (for example, 3 seconds) has elapsed since the previous marker was detected. That is, it is determined whether or not a predetermined time has elapsed since the last marker was detected in the past frame. The time measurement may be performed by the use state determination unit 108 or the control unit 100.

  As a result of the determination, if the marker undetected state continues for a predetermined time or more, the process proceeds to step S1009. If the marker undetected state does not continue for the predetermined time or longer, the HMD status is not changed. The process in step S901 is completed.

  In step S1009, the use state determination unit 108 changes the HMD status to “not used”, and then completes the process in step S901. On the other hand, in step S1003, the use state determination unit 108 obtains the position (representative position) of the marker detected in step S1001. As the representative position, the center coordinate position of the marker is used.

  In step S <b> 1004, the use state determination unit 108 determines whether the representative position of the target marker in the current frame is within a threshold area centered on the representative position of the target marker in the previous frame. Processing is performed for all detected markers. In other words, all the detected markers are processed by determining whether the difference (distance) between the representative position of the target marker in the previous frame and the representative position of the target marker in the current frame is equal to or less than a threshold value. Do about. As a result of the determination, if the difference is greater than the threshold value for all markers or a predetermined number of markers or more, the process proceeds to step S1010, and if the difference is equal to or less than the threshold value, the process proceeds to step S1005.

  In step S <b> 1005, the use state determination unit 108 determines whether the representative position in the current frame is continuously located within the threshold area centered on the representative position in the immediately preceding frame for all markers or a predetermined number of markers or more. Is determined for a predetermined time (for example, 3 seconds) or more. In other words, for all markers or a predetermined number of markers or more, it is determined whether or not the time continuously determined that the difference is equal to or less than a threshold is equal to or longer than a predetermined time. As a result of the determination, if the time continuously determined that the difference is equal to or less than the threshold is equal to or longer than the predetermined time, the process proceeds to step S1009, and it is determined that the difference is equal to or less than the threshold. If the time is shorter than the predetermined time, the process proceeds to step S1006.

  In step S1006, the use state determination unit 108 obtains threshold area information (movement threshold area) indicating the position and size of the threshold area for each marker detected in step S1001. In step S1007, the use state determination unit 108 retains the threshold area information obtained for each marker for use in the next frame.

  In step S1010, the use state determination unit 108 determines whether the representative position in the current frame is continuously outside the threshold area centered on the representative position in the immediately preceding frame for all markers or a predetermined number of markers or more. Is determined over a predetermined time (for example, 2 seconds). As a result of the determination, if the time of continuous positioning is equal to or longer than the predetermined time, the process proceeds to step S1011. If the time of continuous positioning is shorter than the predetermined time, the process proceeds to step S1006. Proceed to In step S1011, the use state determination unit 108 updates the HMD status to “in use”.

  In this embodiment, the movement in the image between frames is obtained based on the change in the representative position of the marker. However, the present invention is not limited to the use of the marker. Etc.) The amount of motion between frames may be obtained by performing comparison processing.

  In this embodiment, the center position of the marker is used as the representative position of the marker. However, the present invention is not limited to this, and the position of the marker corner or a combination of the center point of the marker and a predetermined point is representative. It is good also as a position.

  In the present embodiment, the use state determination unit 108 performs the HMD use state determination process using the real space image, but the position of the HMD 801 is similar to the method disclosed in Japanese Patent Application Laid-Open No. 2004-258123. An HMD usage state determination process using posture information may be performed. In addition, the HMD 801 may be determined not to be used when the fluctuation amount of the position / orientation information falls within a preset range, and the HMD 801 may be determined to be in use when it exceeds the variation amount.

  In addition, when the user holds the HMD 801 on a dedicated holder or hook by a method using the position and orientation information of the HMD 801 and the like, the usage state of the HMD 801 is “not used”. If not, it may be determined as “in use”.

  As described above, according to the present embodiment, since the output image to the display device 30 is switched based on the usage state of the HMD 801, the display on the display device 30 is automatically performed based on a series of conditions when the HMD 801 is used. Images can be switched according to the usage.

[Third Embodiment]
In this embodiment, in order to suppress fatigue of the HMD wearer due to continuous use of the HMD, presentation of the mixed reality space image is controlled based on the continuous use time of the HMD. As the presentation control of the mixed reality space image, when the continuous use time of the HMD elapses for a predetermined time or more, the presentation of the mixed reality image is interrupted and an image other than the mixed reality space image such as an operation screen is presented.

  The system according to the present embodiment will be described with reference to FIG. The timer 109 measures the continuous use time of the HMD 802, and the control unit 100 monitors the time that the timer 109 is timing. There are various methods for obtaining the continuous use time of the HMD 802. For example, as in the second embodiment, the time when the HMD status is continuously “in use” may be set as the continuous use time, or the mixed reality space image is continuously transmitted from the image processing apparatus 901. It is good also considering time as continuous use time.

  The control unit 100 instructs to stop transmission of the mixed reality space image from the image processing device 901 side (stop instruction) when the length of time counted by the timer 109 becomes equal to or longer than a predetermined time. Is transmitted to the image processing apparatus 901 via the control signal transmission unit 155. Furthermore, when the length of time counted by the timer 109 exceeds a predetermined time, the control unit 100 notifies the HMD object composition unit 110 to that effect.

  When receiving a stop instruction from the HMD 802, the control signal receiving unit 156 causes the mixed reality image generation unit 122 to transmit screen information instead of the mixed reality space image. The HMD object composition unit 110 sends the left-eye image and the right-eye image received from the mixed reality image receiving unit 151 to the display units 102L and 102R, respectively. When the HMD object composition unit 110 is notified from the control unit 100 that “the length of continuous use time is equal to or longer than a predetermined time”, the HMD object composition unit 110 will soon add to at least one of the left-eye image and the right-eye image. An object for ending the HMD experience and prompting for a break is superimposed.

  In the present embodiment, a request is made via the control signal transmission unit 155 and the control signal reception unit 156. However, the same effect can be obtained in the embodiment in which the captured image is interrupted via the captured image transmission unit 150 and the captured image reception unit 153 to determine by so-called offline detection on the image processing apparatus 901 side.

  Next, the operation of the HMD 802 will be described. The HMD 802 normally performs processing for displaying the mixed reality space image for the left eye and the mixed reality space image for the right eye transmitted from the image processing apparatus 901 on the display units 102L and 102R, respectively. Such processing is as described in the above embodiment. In parallel with this process, the process according to the flowchart of FIG. 9 is executed by another thread.

  First, in step S1200, the control unit 100 monitors the time measured by the timer 109 and determines whether or not the continuous use time of the HMD 802 is equal to or longer than a predetermined time. As a result of the determination, if the continuous use time of the HMD 802 is equal to or longer than the predetermined time, the process proceeds to step S1201. If the continuous use time of the HMD 802 is less than the predetermined time, the process returns to step S1200.

  In step S1201, the control unit 100 notifies the HMD object composition unit 110 that the length of time counted by the timer 109 has reached a predetermined time. In response to this, the HMD object composition unit 110 superimposes an object for ending the HMD experience and prompting a break on at least one of the left-eye image and the right-eye image received from the mixed reality image receiving unit 151. . Thereafter, the HMD object composition unit 110 transmits the left-eye image and the right-eye image to the display units 102L and 102R, respectively. As a result, the HMD wearer sees the message indicated by this object, and recognizes that the mixed reality presentation application will be terminated soon.

  Next, if a predetermined time (for example, 10 seconds) elapses, the process proceeds to step S1203 via step S1202, and if not elapses, the process waits in step S1202. In step S1203, the control unit 100 transmits an instruction (stop instruction) for stopping transmission of the mixed reality space image from the image processing device 901 side to the image processing device 901 via the control signal transmission unit 155. To do.

  In step S1204, the image control unit 105 arranges the left-eye image and the right-eye image, which are transmitted from the image processing device 901 and received by the mixed reality image receiving unit 151, horizontally in the horizontal direction and combine without overlapping. Thus, one panoramic image is generated.

  In step S1205, the image control unit 105 performs various image adjustment processes such as changing the resolution of the panoramic image to a resolution that can be displayed by the display device 30. Then, the image control unit 105 inputs the image after the image adjustment to the object composition unit 107. The object composition unit 107 composes the above-described object with respect to this image. Then, the object composition unit 107 outputs an image signal of the image obtained by the composition to the display device 30.

  As described above, by using the continuous use restriction process of the HMD 802 and the image output process to the display device 30, the user viewing the display device 30 can check the continuous use time. When the continuous use time of the HMD 802 reaches a predetermined time or longer, the image processing apparatus 901 stops transmitting the mixed reality image and transmits a screen other than the mixed reality image, such as an operation screen, to the HMD 802. This makes it possible to quickly perform operations such as preparation for the next use of the HMD 802 and termination of the mixed reality presentation system.

Claims (9)

Acquisition means for acquiring an image for the right eye and an image for the left eye from an external device;
Means for displaying the right-eye image on a display unit for the right eye and displaying the left-eye image on a display unit for the left eye;
Generating means for generating a single panoramic image by combining the left-eye image and the right-eye image side by side horizontally;
An image processing apparatus comprising: a transmission unit that selects any one of the right-eye image, the left-eye image, and the panoramic image, and transmits the selected image to an external display device. Furthermore,
Instructing means for instructing either transmission of the operation screen or transmission of the mixed reality space image,
When instructed to send the operation screen,
The generating unit generates the panoramic image by combining the left-eye image and the right-eye image horizontally and horizontally without combining them,
When instructed to send a mixed reality space image,
The generating means generates the panoramic image by overlapping the left eye image and the right eye image overlapping each other and horizontally combining them,
When instructing either the operation screen transmission or the mixed reality space image transmission,
The image processing apparatus according to claim 1, wherein the transmission unit transmits the panoramic image to the external display device. The instructing means uses at least one of the real space image for the right eye and the real space image for the left eye to determine whether or not a predetermined amount has changed in the position of the feature amount in the image. Prepared,
When it is determined that there has been a change in a predetermined amount, an instruction to transmit a mixed reality space image is given, and when it is determined that there has not been a change in a predetermined amount, an instruction to transmit an operation screen is given. The image processing apparatus according to claim 2. When instructed to send a mixed reality space image,
The image processing apparatus according to claim 2, wherein the transmission unit transmits any one of the right-eye image and the left-eye image to the external display device. Furthermore,
Means for acquiring a real space image for the right eye and a real space image for the left eye;
Means for transmitting the real space image for the right eye and the real space image for the left eye to the external device;
The acquisition means includes a mixed reality space image for the right eye generated by the external device using the real space image for the right eye, a mixed reality space image for the left eye generated using the real space image for the left eye, The image processing apparatus according to claim 1, wherein the image processing apparatus acquires the image as the right-eye image and the left-eye image from the external apparatus, respectively.   The acquisition unit acquires, from the external device, an operation screen displayed across the display screen of the right-eye display unit and the display screen of the left-eye display unit. The image processing apparatus according to any one of the above. The acquisition means includes
Means for measuring the continuous use time of the image processing device;
When the continuous use time is equal to or longer than a predetermined time, instead of the mixed reality space image for the right eye and the mixed reality space image for the left eye, an operation screen for the right eye, an operation screen for the left eye Means for requesting said external device for transmission;
Means for acquiring the right-eye operation screen and the left-eye operation screen transmitted from the external device in response to the request as the right-eye image and the left-eye image, respectively.
The generating means generates a single panoramic image by horizontally combining the right-eye operation screen and the left-eye operation screen horizontally and horizontally,
The image processing apparatus according to claim 5, wherein the transmission unit transmits the panoramic image to the external display device.   The image processing apparatus according to claim 1, wherein the image processing apparatus is a head-mounted display device. An acquisition unit that acquires an image for the right eye and an image for the left eye from an external device;
Displaying the right-eye image on the right-eye display unit and displaying the left-eye image on the left-eye display unit;
A generating step for generating a single panoramic image by horizontally combining the left-eye image and the right-eye image side by side;
And a transmission step of selecting one of the right-eye image, the left-eye image, and the panoramic image and transmitting the selected image to an external display device. An image processing method performed by a processing apparatus.

Images