JP5161544B2 - Image display system - Google Patents

Description

  The present invention relates to an image display system including a display device with an imaging function and an image processing device.

  Mixed reality, so-called MR (Mixed Reality) technology is known as a technology that seamlessly fuses the real world and the virtual world in real time. Conventionally, in an MR system using an HMD (Head Mounted Display) that is a display device with an imaging function, a captured image acquired by an imaging unit of the HMD is taken into a PC (Personal Computer). Then, after combining with the CG image generated in the PC and superimposing OSD (On Screen Display) display as necessary, the image is displayed on the display unit of the HMD.

  The HMD that is a display device with an imaging function may be a hand-held device such as binoculars, and is not limited to a head-mounted device.

  FIG. 9 is a functional block diagram of a general MR system using an HMD. The outline of the MR system will be described with reference to FIG.

  The HMD 90 includes an I / F 903 with an imaging unit 901, a display unit 902, and an image processing apparatus 91.

  The imaging unit 901 captures an external observation image that substantially matches the line-of-sight position of the HMD wearer. Generally, it is composed of two image sensors for the right eye and left eye for generating a stereo image, an optical system, and a DSP for performing image processing at the subsequent stage. Here, DSP is an abbreviation for Digital Signal Processor.

  The display unit 902 is a display unit for displaying a composite image, and is also composed of two display devices for right eye and left eye and an optical system. As the display device, a small liquid crystal display or a retina scan type device using MEMS is used. Here, MEMS is an abbreviation for Micro Electro Mechanical Systems.

  The I / F 903 is an I / F for transmitting a captured image acquired by the imaging unit 901 to the image processing apparatus 91 and transmitting a composite image received from the image processing apparatus 91 to the HMD 90. For this I / F 903, an optical fiber such as USB, IEEE 1394 metal wire, Gigabit Ethernet (registered trademark) or the like that has real-time properties and can transmit a large amount of data is used.

  The image processing apparatus 91 includes an I / F 904 with the HMD 90, a position information generation unit 905, a CG image drawing unit 906, an image composition unit 907, an OSD display superimposition unit 908, an OSD display data generation unit 909, and a selector 910. The image processing apparatus 91 is generally realized by an apparatus having a high-performance arithmetic processing function or graphic display function such as a PC or a workstation.

  The I / F 904 is an I / F of the image processing apparatus 91 corresponding to the I / F 903 of the HMD 90, and has the same specifications as the I / F 903 of the HMD 90. The position information generation unit 905 generates position and orientation information of the HMD wearer based on the captured image received from the HMD 90. For example, the position / orientation information is generated from a captured image acquired by the imaging unit 901 of the HMD 90. In addition, there are a method of calculating from a captured image acquired by an objective viewpoint camera installed in the surroundings, and a method of calculating using various sensor information such as a gyro sensor and an acceleration sensor attached to the HMD. Further, by using a combination of the above methods, it is possible to generate more accurate position and orientation information.

The CG image drawing unit 906 draws a CG based on the position and orientation information calculated by the position information generation unit 905. The image synthesis unit 907 synthesizes the captured image and the CG image. The OSD display data generation unit 909 generates a message for notifying the HMD wearer of various status information such as setting information and usage time, error contents when an error occurs in the system, a warning, and the like. The OSD display superimposing unit 908 superimposes the message generated by the OSD display data generating unit 909 on the synthesized image synthesized by the image synthesizing unit 907 as an OSD display. The selector 910 switches and outputs either a composite image obtained by combining the captured image and the CG image, or an image obtained by superimposing the OSD display on the composite image.
JP-A-11-88913

  However, in such a conventional MR system (for example, Patent Document 1), a frame delay occurs when a captured image is captured by the image processing apparatus, which makes the observation image presented to the HMD wearer feel uncomfortable. In addition, since the image processing, position information generation, CG image generation, image synthesis, OSD display superposition, and many other processes are executed simultaneously in the image processing apparatus, the processing load on the image processing apparatus increases. As a result, a processing delay occurs in the image processing apparatus, and there is a problem that the time delay from when the image is captured by the HMD until the composite image is displayed to the HMD wearer is further increased.

  In such a configuration, since the OSD display is performed on the image processing apparatus side, it is necessary to notify the PC of the error content when an error occurs on the HMD side. In addition, OSD display cannot be performed when communication between the HMD and the PC is interrupted. For this reason, it is desirable to add an OSD display function to the HMD side as well, but this not only causes an increase in circuit scale but also causes a new problem of increasing complexity by controlling both OSD displays.

  The present invention has been made in view of such a problem, and an object of the present invention is to provide an image display system capable of reducing the processing load of each device constituting the image display system and improving the total throughput of the system. .

In order to achieve the above object, an image display system according to the present invention comprises the following arrangement. That is,
An image display system including a display device with an imaging function and an image processing device,
The display device with an imaging function is:
Imaging means for imaging a real space;
Display means for displaying an image;
First communication means for communicating with the image processing device;
Storage means for storing a plurality of types of OSD display images to be superimposed on an image displayed by the display means;
OSD control means for selecting the OSD display image from the plurality of types of OSD display images according to the input control information;
According to the input control information, the real space image captured by the imaging unit, the image input from the image processing device via the first communication unit, and the OSD display image selected by the OSD control unit, First combining means for selecting any two and combining;
According to a set operation mode, the first mode control means for generating control information for the OSD control means and the first synthesis means, and controlling the operation of the OSD control means and the first synthesis means,
The image processing apparatus includes:
Second communication means for communicating with the display device with an imaging function;
Image rendering means for rendering a virtual space image to be superimposed on the real space image on the memory;
Second combining means for combining the real space image input from the display device with an imaging function via the second communication means and the virtual space image drawn by the image drawing means in accordance with the input control information. When,
Superimposing means for superimposing an OSD display image generated in accordance with input control information on an image to be transmitted to the display device with an imaging function;
In accordance with the set operation mode, the second synthesis means and the superimposing means are generated control information, and a second mode control means for controlling the operation of the second synthesizing means and the superimposing means,
Mode input means for setting the operation mode in at least one of the display device with an imaging function and the image processing device.

  Preferably, one of the operation modes is a mode in which a real space image captured by the imaging unit of the display device with an imaging function is directly displayed on the display unit.

  Preferably, one of the operation modes is a mode in which the first space unit of the display device with an imaging function combines the real space image and the virtual space image and displays the image on the display unit. .

  Preferably, one of the operation modes is a mode in which, in the second synthesizing unit of the image processing apparatus, the real space image and the virtual space image are synthesized and displayed on the display unit.

  Preferably, one of the operation modes is an image to be transmitted to the display device with an imaging function by the superimposing unit of the image processing device, and the OSD display image generated according to the input control information. It is a mode to superimpose.

  Preferably, one of the operation modes is a mode for superimposing an OSD display image in the first combining unit of the display device with an imaging function.

  Preferably, one of the operation modes is a mode in which an OSD display image is superimposed by both the superimposing unit of the image processing apparatus and the first synthesizing unit of the display device with an imaging function.

  Preferably, when a mode for superimposing an OSD display image is selected by both the superimposing unit of the image processing apparatus and the first combining unit of the display device with an imaging function, the OSD display superimposed by each unit is selected. Control means for controlling the display position so that the images do not overlap each other is further provided.

Preferably, when the OSD display image is superimposed, the superimposing unit of the image processing apparatus converts the OSD display information including display position information of the OSD display image into the OSD control unit of the display device with an imaging function. To
The OSD control means of the display device with an imaging function is synthesized by the first synthesis means of the display device with an imaging function based on display position information in the OSD display information received from the image processing device. The display position of the display image is determined.

Preferably, the OSD display information is any one of information on priority of display contents, information on superimposed device names, information on fonts used, information on colors, information on sizes, or the like Including multiple information
When performing OSD display on the display device with an imaging function, the image processing device and the imaging function are controlled by performing control so that the OSD display is performed by a display method different from the OSD display method performed by the image processing device. Even when the OSD display is simultaneously superimposed on the attached display device, the user can easily recognize the state of each device.

  Preferably, at the time of setting the operation mode by the mode input means, the display device with an imaging function and the image processing device negotiate, and a display position at which each device performs OSD display is determined in advance.

  ADVANTAGE OF THE INVENTION According to this invention, the image display system which can reduce the processing load of each apparatus which comprises an image display system, and can improve the total throughput of a system can be provided.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  The present invention relates to a mixed reality presentation technique using an HMD (Head Mounted Display) which is a display device with an imaging function. In particular, the present invention relates to an image display system that controls the respective image synthesis functions and OSD display functions of the HMD and the image processing apparatus and that can execute a plurality of operation modes.

  In the present invention, the image composition on the HMD side is made possible by adding an image composition function capable of OSD superposition in the HMD. As a result, the frame delay due to the capture of the captured image on the image processing device side and the processing delay (time delay from imaging to display) on the image processing device side are reduced, and at the HMD side with a minimum circuit scale. An image display system capable of performing OSD display is realized.

(Embodiment 1)
FIG. 1 is a diagram showing a configuration of an image display system according to the first embodiment of the present invention.

  A video see-through type HMD 201 (display device with an imaging function) has an imaging unit for acquiring an image of a real space observed by an HMD wearer. The HMD 201 displays a composite image (MR image) obtained by superimposing a CG image (virtual space image) generated by the image processing device 203 on a real space image, a captured real space image, an output image from the image processing device 203, and the like. A display unit for providing an image to the HMD wearer;

  The HMD 201 communicates with the controller 202 connected by wire. The communication between the HMD 201 and the controller 202 is not limited to wired communication. For example, a wireless communication system that configures a small-scale network such as a WLAN (Wireless Local Area Network) or a WPAN (Wireless Personal Area Network) may be used. The HMD 201 can be driven by power supply from the controller 202 or can be driven by a battery.

  The image processing apparatus 203 connected to the controller 202 by wire has an image processing unit that draws a CG image. The image processing apparatus 203 communicates with the HMD 201 via the controller 202. In the HMD 201, the received CG image and the real space image captured by the imaging unit in the HMD 201 are combined, and the combined image (MR image) is displayed to the HMD wearer by the display unit.

  In FIG. 1, the image processing apparatus 203 and the controller 202 are shown as separate hardware configurations, but the present invention is not limited to this. For example, all the functions of the controller 202 may be mounted and integrated in the image processing apparatus 203, or the functions of the image processing apparatus 203 and the controller 202 may be integrated to form a dedicated image processing apparatus. Is possible. The image composition function does not necessarily have to be provided on the HMD 201 side, and may be configured to be installed on the image processing apparatus 203 or the controller 202 side.

  The image processing apparatus 203 is realized by standard components (eg, CPU, memory (RAM, ROM, etc.), hard disk, external storage device, network interface, display, keyboard, mouse, etc.) mounted on a general-purpose computer. Is done.

  In the following description, from a functional viewpoint, a combination of functions of the HMD 201 and the controller 202 will be described as HMD.

  Next, the functional configuration of the image display system according to the first embodiment will be described with reference to FIG.

  FIG. 2 is a block diagram showing a functional configuration of the image display system according to the first embodiment of the present invention.

  The image display system according to the first embodiment includes an HMD 10 and an image processing device 11.

  The HMD 10 includes an imaging unit 101, a display unit 102, an I / F 103, a chroma key composition unit 104, an OSD display data storage unit 105, a mode control unit 106, an OSD control unit 115, and a selector 116.

  The image processing apparatus 11 includes an I / F 107, a position information generation unit 108, a CG image drawing unit 109, an image composition unit 110, an OSD display data generation unit 117, an OSD display superimposition unit 111, a selector 112, a mode input unit 113, and mode control. Part 114.

  The imaging unit 101 of the HMD 10 acquires a captured image of the real space at the viewpoint of the HMD wearer. The display unit 102 displays an image to the HMD wearer. An I / F 103 (first communication unit) and an I / F 107 (second communication unit) of the image processing apparatus 11 are communication I / Fs between the HMD 10 and the image processing apparatus 11.

  The chroma key composition unit 104 (first composition unit) extracts a specific background color from one of the two input image data, and performs chroma key composition to replace the background color part with the other image data. The input to the chroma key combining unit 104 includes three inputs: a captured image acquired by the image capturing unit 101, a composite image or CG image transmitted from the image processing apparatus 11, and an OSD display image transmitted from the OSD control unit 115. The chroma key composition unit 104 performs a process of selecting and composing two images out of these three image inputs according to the set mode.

  The OSD display data storage unit 105 stores a plurality of types of OSD display data for OSD display performed by the HMD 10. The OSD control unit 115 determines the OSD corresponding to the message image (OSD display image) to be notified to the HMD wearer from the OSD display data stored in the OSD display data storage unit 105 according to the state of the HMD 10. Select the display data. The OSD control unit 115 performs control to output the selected OSD display data to the chroma key composition unit 104.

  The mode control unit 106 (first mode control unit) controls the function of the HMD 10 according to the setting input by the mode input unit 113 of the image processing apparatus 11. Details of the setting will be described later. The selector 116 selects an image to be output to the display unit 102 according to the set mode.

  The position information generation unit 108 of the image processing apparatus 11 calculates the position and posture of the HMD wearer from the real space image data acquired by the imaging unit 101 of the HMD 10 and generates three-dimensional position and posture information. The CG image drawing unit 109 draws a CG image to be combined with the captured image on the memory based on the three-dimensional position and orientation information generated by the position information generation unit 108. Here, the CG image is more conceptually expressed as a virtual space image for superimposing and synthesizing an artificially generated image on a virtual space virtually defined on a captured image that is a real space image. is there.

  The image synthesis unit 110 (second synthesis unit) synthesizes the captured image acquired by the imaging unit 101 and the CG image generated by the CG image drawing unit 109. The OSD display data generation unit 117 generates OSD display data for OSD display. The OSD display superimposing unit 111 superimposes the OSD display data generated by the OSD display data generating unit 117 on the image transmitted to the HMD 10.

  The selector 112 selects an output image according to the set mode and outputs it to the I / F 107. The mode input unit 113 is an input unit for selecting a mode for performing image synthesis and OSD function control. The mode control unit 114 (second mode control unit) controls the function of the image processing apparatus 11 according to the setting input by the mode input unit 113.

  The image display system according to the first embodiment includes a chroma key composition unit 104, an OSD display data storage unit 105, a CG image drawing unit 109, an image composition unit 110, an OSD display superimposition unit, according to the setting input by the mode input unit 113. The selector 112 is controlled. And by this control, it becomes possible to perform the function demonstrated below.

  In the first embodiment, the mode input unit 113 is provided in the image processing apparatus 11, but the present invention is not limited to this. For example, the mode input unit 113 is provided in the HMD 10, and setting information is transmitted to the mode control unit 114 of the image processing apparatus 11 via the mode control unit 106 and I / F 103 of the HMD 10 and the I / F 107 of the image processing apparatus 11. You can also. A configuration in which the setting I / F is provided in the HMD 10 or the image processing apparatus 11 and the setting is performed from an external apparatus is also conceivable.

  Next, an example of a display image in the image display system according to the first embodiment will be described with reference to FIG.

  FIG. 3 is a diagram for explaining an example of a display image in the image display system according to the first embodiment of the present invention.

  The captured image 301 is an image obtained by capturing the real space observed by the HMD wearer with the imaging unit 101 of the HMD 201. In the captured image 301, it is assumed that the marker 302 installed in the real space is previously given absolute position and orientation information in the MR space or relative position and orientation information with respect to an arbitrary absolute position and orientation.

  The position information generation unit 108 in the image display system detects the marker 302 from the captured image 301. Next, the position information generation unit 108 relates to the relative positional relationship of the HMD 10 main body based on the position and orientation information of the marker 302 from the marker information of the marker 302 and the direction in which the HMD wearer is observing the marker 302. Three-dimensional position and orientation information can be calculated. Here, the marker information is information related to the marker, such as the size and shape of the marker 302, the fill pattern, and the like.

  A plurality of markers can be used at the same time. By using a plurality of markers, it is possible to expand an area where position and orientation information can be calculated. Further, when a plurality of markers are used at the same time, the relationship between the relative positions and orientations of the markers is calculated in advance, and the position and orientation information of each marker is registered. Information can be calculated.

  When using a plurality of markers at the same time, it is not a marker that can be identified up to the direction by the internal filling pattern as shown in FIG. 3, for example, a color marker using a specific color or a light emitting element such as an LED. A marker having no directionality can also be used. Also, instead of markers prepared in advance, extract contours and feature points such as tables, walls, and floors in the image, or any color, and use their natural features as indicators instead of markers, Three-dimensional position and orientation information can also be generated.

  Furthermore, three-dimensional position and orientation information is generated with higher accuracy by using a plurality of the same type of markers, using several types of markers at the same time, or using a combination of marker information and natural feature information in the image. You can also. In addition, by grasping the relative positional relationship of multiple markers and natural features and registering them in the system, the position of each marker and natural feature can be obtained even if not all markers or feature points are displayed in the image. It is also possible to estimate the posture.

  In order to generate the three-dimensional position and orientation information of the HMD in the image display system, there is a method of using sensing information from various sensing devices in addition to a method of using image information captured by the imaging unit 101 (subjective camera) built in the HMD 10. is there. As an example of the sensing device, there is a type of device that is used by being attached to the HMD 10 itself, such as a magnetic sensor, a gyro sensor, and an acceleration sensor. Or the device of the type installed and used in surrounding environments other than HMD10 like an infrared camera (objective camera) etc. is mentioned.

  When an objective camera is used, a marker is also attached to the HMD 10 main body as a detection target. These sensing devices can be used alone or in combination of a plurality of types. In addition, image information captured by a subjective camera built in the HMD 10 and sensing information acquired by a sensing device can be used in combination. In such a configuration, even when the marker is not captured by the subjective camera, the 3D position and orientation information can be generated without interruption by supplementing with the sensing information of the sensing device.

  The CG image 303 is a CG image drawn based on the captured image 301 or 3D position and orientation information generated based on sensing information. The size, orientation, etc. of the CG object 304 in the CG image 303 are determined according to the position and orientation of the HMD wearer, and are drawn by the CG image drawing unit 109. Parts other than the CG object 304 are filled with a color determined as a background color.

  The composite image 305 is an image after the captured image 301 and the CG image 303 are combined, and the HMD wearer can experience the MR space by viewing the composite image displayed on the HMD 10. In the image display system, the display image can be switched, and the HMD wearer can observe not only the composite image 305 but also the captured image 301 and the CG image 303 with the HMD 10.

  Furthermore, it is also possible to superimpose a message image for notifying the HMD wearer as an OSD display image on each of the captured image 301, the CG image 303, and the composite image 305, and display these images on the HMD 10. . Here, the message image includes various status information such as setting information and usage time, error contents when an error occurs in the image display system, a warning, and the like.

  OSD superimposed images 306, 307, and 308 are obtained by superimposing a message image to be notified to the HMD wearer as an OSD display on each of the captured image 301, the CG image 303, and the composite image 305, respectively. In FIG. 3, “usage”, “for”, “hour”, “between”, “3”, “0”, “minute”, “time” are selected from the plurality of stored OSD display images (character images). ”And“ over ”OSD display images (character images) are selected and combined to generate an OSD display image that displays“ usage time of 30 minutes ”.

  Next, an example of a mode in the image display system will be described with reference to FIG.

  FIG. 4 is a diagram showing modes of the image display system according to the first embodiment of the present invention.

  The “composite” column is a column indicating which of the chroma key combining unit 104 of the HMD 10 and the image combining unit 110 of the image processing apparatus 11 is used to combine the captured image and the CG image. The setting in which “None” is written in the combination column indicates a system in which an image captured by the imaging unit 101 of the HMD 10 is directly displayed on the display unit 102.

  The column “OSD display” indicates whether or not the OSD display superimposing function of the HMD 10 and the image processing apparatus 11 is used.

  In addition to the above-described presence / absence of the synthesis function and the difference between apparatuses that execute the function, the mode of the image display system is further divided depending on the presence / absence of OSD display and the difference between apparatuses that execute the function. Here, this mode is an operation mode of the image display system for controlling the image composition function and the OSD display function.

  Next, an example of an image handled in each mode of the image display system will be described with reference to FIG.

  FIG. 5 is a diagram illustrating an example of an image handled in each mode of the image display system according to the first embodiment of the present invention.

  The operation of each mode shown in FIG. 4 will be described below using the functional block diagram of FIG. 2 and the example of the image of FIG. In addition, display examples of modes 2, 6, and 8 to be described are clear from display examples of other modes, and thus are omitted in FIG.

  Modes 1 and 2 indicate systems in which the display unit 102 directly displays the captured image 501 captured by the imaging unit 101 of the HMD 10 without performing image synthesis.

  In mode 1, the OSD display image 502 is superimposed on the captured image acquired by the imaging unit 101 of the HMD 10 by the chroma key combining unit 104 to generate a captured image 503 with OSD display and displayed on the display unit 102. The OSD display image 502 to be superimposed on the captured image 501 is selected from the data stored in the OSD display data storage unit 105 in the HMD 10 according to the state of the HMD 10 and contents to be notified to the HMD wearer. Use.

  In mode 2 (not shown), the captured image 501 is directly displayed on the display unit 102 without superimposing the OSD display image 502.

  In mode 3 to mode 6, the captured image captured by the imaging unit 101 of the HMD 10 and the CG image generated by the CG image rendering unit 109 of the image processing apparatus 11 are combined by the image combining unit 110 of the image processing apparatus 11 and displayed. A system to be displayed by the unit 102 is shown.

  In mode 3, the OSD display superimposing unit 111 superimposes the OSD display image on the composite image generated by the image combining unit 110 of the image processing apparatus 11. The composite image 504 on which the OSD display is superimposed is further combined with the OSD display image 505 by the chroma key combining unit 104 of the HMD 10, and the composite image 506 with OSD display is displayed on the display unit 102. In mode 3, OSD display can be performed in the image processing apparatus 11 and the HMD 10 according to the respective states.

  In mode 4, the synthesized image 507 generated by the image synthesizing unit 110 of the image processing apparatus 11 is synthesized with the OSD display image 508 by the chroma key synthesizing unit 104 of the HMD 10, and the synthesized image 509 with OSD display is displayed on the display unit 102. Is displayed.

  In mode 5, the OSD display superimposing unit 111 superimposes the OSD display image on the composite image generated by the image combining unit 110 of the image processing apparatus 11 to generate the composite image 510. On the other hand, the HMD 10 displays a composite image 512 with the OSD display (the same image as the composite image 510) on the display unit 102 without performing composition or OSD display.

  In mode 6 (not shown), the composite image 507 generated by the image compositing unit 110 of the image processing apparatus 11 is directly displayed on the composite image 507 on the display unit 102 without performing OSD display on the image processing apparatus 11 and the HMD 10. indicate.

  In modes 7 and 8, the captured image captured by the imaging unit 101 of the HMD 10 and the CG image generated by the CG image rendering unit 109 of the image processing apparatus 11 are combined by the chroma key combining unit 104 of the HMD 10 and displayed on the display unit 102. The system is shown.

  In mode 7, the OSD display superimposing unit 111 superimposes the OSD display image on the CG image generated by the CG image drawing unit 109 of the image processing apparatus 11. The CG image 513 on which the OSD display is superimposed is combined with the captured image 514 by the chroma key combining unit 104 of the HMD 10, and the combined image 515 with OSD display is displayed on the display unit 102.

  In mode 8 (not shown), the CG image generated by the CG image drawing unit 109 of the image processing apparatus 11 is combined with the captured image by the chroma key combining unit 104 of the HMD 10 without performing OSD display on the image processing apparatus 11. And the composite image is displayed on the display unit 102.

  Next, a processing flow when setting the mode of the image display system will be described with reference to FIG.

  FIG. 6 is a flowchart for explaining a processing flow at the time of mode setting of the image display system according to the first embodiment of the present invention.

  When the power is turned on, the image processing apparatus 11 first transmits a connection request to the HMD 10 in order to check whether or not it is connected to the HMD 10 (step S601). When the HMD 10 receives the connection request, the HMD 10 transmits a connection completion notification to the image processing apparatus 11 to notify that it is connected to the image processing apparatus 11 (step S602).

  When the image processing apparatus 11 receives the connection completion notification from the HMD 10 and confirms the connection with the HMD 10, the image processing apparatus 11 shifts to a mode input waiting state (step S603). If mode input is performed from the mode input unit 113 while waiting for mode input (step S604), the mode control unit 114 transmits mode setting information to the HMD 10 (step S605). Subsequently, the mode control unit 106 sends control information corresponding to the input mode to the functional blocks of the CG image drawing unit 109, the image synthesis unit 110, the OSD display superimposing unit 111, and the selector 112 in the image processing apparatus 11. Transmit (step S606).

  In accordance with the control information, the mode control unit 114 sets each functional block in the image processing apparatus 11 (step S607). Thereafter, each functional block transmits a setting completion notification to the mode control unit 114 (step S608). When receiving a setting completion notification from each functional block, the mode control unit 114 shifts to a mode setting completion notification waiting state until receiving a mode setting completion notification from the HMD 10 (step S609).

  On the other hand, the HMD 10 receives mode setting information from the image processing apparatus 11. In response to this, the mode control unit 106 transmits control information corresponding to the received mode setting information to the function blocks of the chroma key combining unit 104, the OSD control unit 115, and the selector 116 (step S610). The mode control unit 106 sets each functional block in the HMD 10 according to the control information (step S611).

  Thereafter, each functional block transmits a setting completion notification to the mode control unit 106 (step S612). When receiving the setting completion notification, the mode control unit 106 transmits a mode setting completion notification to the image processing apparatus 11 (step S613). Upon receiving the mode setting completion notification from the HMD 10, the image processing apparatus 11 exits the mode setting completion notification waiting state and ends the mode setting (step S614).

  As described above, according to the first embodiment, it is possible to add an image composition function and an OSD display superimposing function on the HMD side while minimizing an increase in circuit scale. Further, by performing image composition on the HMD side, an image display system with little time delay from imaging to display can be realized.

  In addition, by executing the OSD display function on the HMD side using the chroma key composition unit, the circuit scale can be reduced, and the size and power consumption can be reduced compared to simply adding the chroma key composition function and the OSD display function to the HMD separately. Can be suppressed.

(Embodiment 2)
Compared with the image display system of the first embodiment, the image display system of the second embodiment has a position information generation unit on the HMD side, from the OSD display superimposing unit of the image processing apparatus to the OSD control unit of the HMD. The OSD display information on the image processing apparatus side is transmitted.

  FIG. 7 is a block diagram showing a functional configuration of the image display system according to the second embodiment of the present invention.

  The image display system according to the second embodiment includes an HMD 70 and an image processing device 71.

  The HMD 70 includes an imaging unit 701, a display unit 702, an I / F 703, a chroma key composition unit 704, an OSD display data storage unit 705, an OSD control unit 706, a position information generation unit 707, a mode control unit 708, a selector 709, and a position and orientation sensor 717. Have

  The image processing apparatus 71 includes an I / F 710, a CG image drawing unit 711, an image composition unit 712, an OSD display superimposing unit 713, a selector 714, a mode input unit 715, a mode control unit 716, an OSD display data generation unit 718, and an OSD control unit. 719.

  In FIG. 7, the detailed description of functional blocks having the same names as those in FIG. 2 is omitted. Therefore, hereinafter, a functional block newly added in the second embodiment and a portion where the function is different will be described.

  The position / orientation sensor 717 acquires sensing information for generating three-dimensional position / orientation information of the HMD 70. The position information generation unit 707 calculates the position and orientation of the HMD wearer using the captured image and sensing information acquired by the imaging unit 701, and generates 3D position and orientation information. Then, the position information generation unit 707 transmits the three-dimensional position and orientation information to the OSD control unit 706 of the HMD 70 and the OSD control unit 719 and the CG image drawing unit 711 of the image processing apparatus 71.

  Next, the OSD control unit 706 of the HMD 70 generates a message image to be notified to the HMD wearer from a plurality of types of OSD display data stored in the OSD display data storage unit 705 according to the state of the HMD 70. Select the corresponding OSD display data. Next, the OSD control unit 706 includes the OSD display information transmitted from the OSD control unit 719 of the image processing apparatus 71, the three-dimensional position and orientation information generated by the position information generation unit 707, and the CG transmitted from the CG image drawing unit 711. Refer to drawing range information. Then, the OSD control unit 706 controls the selected OSD display data based on the various information referred to, and transmits the OSD display data to the chroma key composition unit 704.

  The OSD display information is OSD display information including the display position of the OSD display executed by the image processing apparatus 71.

  The CG image drawing unit 711 of the image processing device 71 executes drawing of a CG image to be combined with the captured image based on the three-dimensional position and orientation information generated by the position information generation unit 707 of the HMD 70. Next, the CG image drawing unit 711 transmits the drawn CG image to the image composition unit 712, and transmits the CG drawing range information to the OSD control unit 719 and the OSD control unit 706 of the HMD 70.

  The OSD control unit 719 of the image processing apparatus 71 refers to the three-dimensional position and orientation information generated by the position information generation unit 707 and the CG drawing range information from the CG image drawing unit 711, and generates the OSD display data generation unit 718. Controls the overlay position of OSD display data. It is desirable to control the display position of the OSD display data so that the OSD display data and the CG image do not overlap each other. However, when the CG drawing range frequently changes from frame to frame, the OSD display position changes greatly from frame to frame, so this display position control method is not limited to this.

  The OSD control unit 719 transmits display data subjected to display position control to the OSD display superimposing unit 713 and OSD display information to the OSD control unit 706 of the HMD 70 via the I / Fs 710 and 703, respectively.

  By having the above system configuration, the image display system of the second embodiment can perform all operations in the mode shown in FIG. 4 as in the first embodiment.

  Since the image display system according to the second embodiment includes the position information generation unit 707 as the HMD 70, for example, during operation in mode 7 or mode 8, the captured image used for position information generation from the HMD 70 to the image processing apparatus 71 is acquired. There is no need to send. Further, it is not necessary to transmit the position / orientation information from the image processing apparatus 71 to the HMD 70 during the operation in the mode in which the OSD display is performed by the HMD 70 such as mode 3 or mode 4. As described above, the image display system according to the second embodiment has an effect that the amount of data transmitted and received between the HMD 70 and the image processing apparatus 71 can be reduced.

  Furthermore, even when communication between the HMD 70 and the image processing apparatus 71 is interrupted, the HMD 70 can grasp its own position, so that it is possible to perform OSD display according to the position of the HMD wearer. For example, when the HMD 70 and the image processing apparatus 71 are wired I / F, it is detected that the HMD wearer is about to move away from the image processing apparatus beyond the length of the cable, and this is indicated on the OSD display. It is possible to notify. Further, in the case of a wireless I / F, it is possible to detect that the mobile terminal is about to move out of the wireless communication range and notify the fact by OSD display.

  Next, effects of transmitting OSD display information from the image processing apparatus 71 will be described below.

  FIG. 8 is a diagram for explaining an example of a display image in the image display system according to the second embodiment of the present invention.

  Of the modes shown in FIG. 4, mode 3 is a mode in which OSD display can be performed by both the HMD 70 and the image processing apparatus 71. In this case, the HMD 70 monitors the state of the HMD 70 and the image processing device 71 monitors the state of the image processing device 71, and each device can perform OSD display as necessary. Therefore, even when communication between the HMD 70 and the image processing apparatus 71 cannot be performed due to an error, it is possible to perform OSD display on the HMD 70 and notify the HMD wearer that an error has occurred.

  However, when the image processing apparatus 71 and the HMD 70 perform OSD display on the same image, the OSD display overlaps as in the image 801, and the HMD wearer may not be able to recognize the message.

  Therefore, in the second embodiment, when OSD display is performed by the OSD display superimposing unit 713 in the image processing apparatus 71, OSD display information including display position information is transmitted to the HMD 70. Then, the OSD control unit 706 in the HMD 70 refers to the display position information, and controls to perform OSD display at a position where the display does not overlap, thereby solving this problem.

  As another method, when the mode (mode 3) in which OSD display is executed by both the HMD 70 and the image processing apparatus 71 is set at the time of mode setting after startup, negotiation is performed between the HMD 70 and the image processing apparatus 71. Then, based on the negotiation, a control method for determining in advance the display positions of the OSD displays to be executed can be considered so that the display positions for performing the OSD display do not overlap.

  By performing the display position control as described above, it is possible to perform display so that the OSD display of each device does not overlap as shown in the image 802. Further, the OSD display information can include information such as the priority of the notification message (OSD display image), the name of the device that has notified the notification message, the font color, and the size as information other than the display position information.

  That is, the OSD display information includes information on the priority of display contents, information on the superimposed device name, information on the font used, information on color, information on size, or a plurality of them. Contains information.

  As a result, it is possible to display OSD superimposed images such as images 803, 804, and 805, and the HMD wearer can easily recognize the state of each device by the user.

  As described above, according to the second embodiment, an image composition function and an OSD display superimposing function can be added on the HMD side. Further, by having the position information generation unit on the HMD side, it is possible to reduce the amount of transmission data between apparatuses, even in some modes. Further, in the mode (mode 3) in which the OSD display is simultaneously superimposed on the HMD and the image processing apparatus, it is possible to prevent the OSD display from overlapping. At the same time, it is possible to realize an OSD display in which the HMD wearer can easily recognize the priority of the message and which device is reporting the message.

  Although the embodiment has been described in detail above, the present invention can take an embodiment as a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowchart shown in the drawing) that realizes the functions of the above-described embodiments is directly or remotely supplied to a system or apparatus. In addition, this includes a case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  Examples of the recording medium for supplying the program include a floppy (registered trademark) disk, a hard disk, and an optical disk. Further, as a recording medium, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), etc. is there.

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. Then, the computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from a homepage of the connection destination to a recording medium such as a hard disk. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Further, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  Furthermore, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

It is a figure which shows the structure of the image display system of Embodiment 1 of this invention. It is a block diagram which shows the function structure of the image display system of Embodiment 1 of this invention. It is a figure for demonstrating an example of the display image in the image display system of Embodiment 1 of this invention. It is a figure which shows the mode of the image display system of Embodiment 1 of this invention. It is a figure which shows the example of the image handled by each mode of the image display system of Embodiment 1 of this invention. It is a flowchart explaining the processing flow at the time of the mode setting of the image display system of Embodiment 1 of this invention. It is a block diagram which shows the function structure of the image display system of Embodiment 2 of this invention. It is a figure for demonstrating an example of the display image in the image display system of Embodiment 2 of this invention. It is a functional block diagram which shows the system configuration | structure of the conventional MR system.

Explanation of symbols

10 HMD
DESCRIPTION OF SYMBOLS 11 Image processing apparatus 101 Imaging unit 102 Display unit 103 I / F (HMD)
104 Chroma key composition unit 105 OSD display data storage unit 106 Mode control unit (HMD)
107 I / F (image processing apparatus)
108 position information generation unit 109 CG image drawing unit 110 image composition unit 111 OSD display superimposing unit 112 selector (image processing apparatus)
113 mode input unit 114 mode control unit (image processing apparatus)
115 OSD controller 116 Selector (HMD)
117 OSD display data generator

Claims (11)

An image display system including a display device with an imaging function and an image processing device,
The display device with an imaging function is:
Imaging means for imaging a real space;
Display means for displaying an image;
First communication means for communicating with the image processing device;
Storage means for storing a plurality of types of OSD display images to be superimposed on an image displayed by the display means;
OSD control means for selecting the OSD display image from the plurality of types of OSD display images according to the input control information;
According to the input control information, the real space image captured by the imaging unit, the image input from the image processing device via the first communication unit, and the OSD display image selected by the OSD control unit, First combining means for selecting any two and combining;
According to a set operation mode, the first mode control means for generating control information for the OSD control means and the first synthesis means, and controlling the operation of the OSD control means and the first synthesis means,
The image processing apparatus includes:
Second communication means for communicating with the display device with an imaging function;
Image rendering means for rendering a virtual space image to be superimposed on the real space image on the memory;
Second combining means for combining the real space image input from the display device with an imaging function via the second communication means and the virtual space image drawn by the image drawing means in accordance with the input control information. When,
Superimposing means for superimposing an OSD display image generated in accordance with input control information on an image to be transmitted to the display device with an imaging function;
In accordance with the set operation mode, the second synthesis means and the superimposing means are generated control information, and a second mode control means for controlling the operation of the second synthesizing means and the superimposing means,
An image display system comprising: mode input means for setting the operation mode in at least one of the display device with an imaging function and the image processing device. The image display system according to claim 1, wherein one of the operation modes is a mode in which a real space image captured by the imaging unit of the display device with an imaging function is directly displayed by the display unit. . One of the operation modes is a mode in which the real space image and the virtual space image are combined and displayed on the display unit in the first combining unit of the display device with an imaging function. The image display system according to claim 1. One of the operation modes is a mode in which, in the second synthesizing unit of the image processing apparatus, the real space image and the virtual space image are synthesized and displayed on the display unit. 2. The image display system according to 1. One of the operation modes is a mode in which the superimposing unit of the image processing apparatus superimposes an OSD display image generated according to the input control information on an image to be transmitted to the display device with an imaging function. The image display system according to claim 2, wherein the image display system is an image display system. 5. The image display system according to claim 2, wherein one of the operation modes is a mode in which an OSD display image is superimposed in the first combining unit of the display device with an imaging function. 5. One of the operation modes is a mode in which an OSD display image is superimposed by both the superimposing unit of the image processing apparatus and the first synthesizing unit of the display device with an imaging function. The image display system described in 1. When the mode for superimposing the OSD display image is selected by both the superimposing unit of the image processing apparatus and the first combining unit of the display device with an imaging function, the OSD display images superimposed by the respective units do not overlap each other. The image display system according to claim 7, further comprising control means for controlling the display position as described above. When the OSD display image is superimposed, the superimposing unit of the image processing apparatus transmits OSD display information including display position information of the OSD display image to the OSD control unit of the display device with an imaging function.
The OSD control means of the display device with an imaging function is synthesized by the first synthesis means of the display device with an imaging function based on display position information in the OSD display information received from the image processing device. The image display system according to claim 8, wherein a display position of the display image is determined. The OSD display information includes information on priority of display contents, information on superimposed device names, information on fonts used, information on colors, information on sizes, or a plurality of information among them. Including
When performing OSD display on the display device with an imaging function, the image processing device and the imaging function are controlled by performing control so that the OSD display is performed by a display method different from the OSD display method performed by the image processing device. The image display system according to claim 9, wherein the user can easily recognize the state of each device even when the OSD display is simultaneously superimposed on the attached display device. The display device with an imaging function and the image processing device negotiate at the time of setting the operation mode by the mode input means, and a display position at which each device performs OSD display is determined in advance. 9. The image display system according to 8.