JP2015064476A - Image display device, and method of controlling image display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display device that can integrate the types of the display of peripheral external devices.SOLUTION: An image display device includes: a creation unit that creates an integrated image including a first display area that displays, as a list, a plurality of identification images for mutually identifying a plurality of external devices connected with the image display device, and a second display area that displays a display image, which is an image displayed by one external device selected from the plurality of external devices; and an image display unit that allows a user of the image display device to visually recognize the integrated image.

Description

  The present invention relates to an image display device.

  A head-mounted display device (Head Mounted Display (HMD)), which is a display device mounted on the head, is known. The head-mounted display device, for example, generates image light representing an image using a liquid crystal display and a light source, and uses the generated image light to a user's eye using a projection optical system, a light guide plate, or the like. By guiding, the user is made to recognize the virtual image. Such a head-mounted display device is connected to an external device such as a smartphone by a wired interface such as MicroUSB (Universal Serial Bus), for example, and conforms to a standard such as MHL (Mobile High Definition Link). The video signal from can be received. Similarly, the head-mounted display device is connected to an external device through a wireless interface such as a wireless LAN, and can receive a video signal from the external device in accordance with a standard such as Miracast. The head-mounted display device can allow the user of the head-mounted display device to visually recognize the same virtual image as the screen displayed on the display screen of the external device, based on the video signal received in this way.

  Patent Document 1 describes an AV (Audio Visual) system capable of integrating and displaying a current program video, information about the current program video, and program information of other channels on one screen. ing. Patent Document 2 describes an image display device in which a virtual screen is provided adjacent to a display screen of a plurality of computer devices, and a user of each computer device can operate a window on the virtual screen.

Japanese Patent No. 3968467 Japanese Patent No. 5082722

  In the technique described in Patent Document 1, an analog broadcast receiving unit and a digital broadcast receiving unit are illustrated as AV devices connected to the AV system. That is, in the technique described in Patent Document 1, there is a problem that versatility is low since the video is between specific devices. In addition, the technology described in Patent Document 2 only provides a display means different from the display screen of the computer device, that is, a virtual screen, and cannot display the display screens of a plurality of computer devices. There was a problem.

  For this reason, there has been a demand for an image display device that can integrate display in peripheral external devices (peripheral devices). In addition, there have been various demands for image display devices such as improved versatility, improved usability, improved convenience, improved reliability, and reduced manufacturing costs.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to an aspect of the present invention, an image display device is provided. The image display device includes: a first display area for displaying a list of a plurality of identification images for mutually identifying a plurality of external devices connected to the image display device; and a selection from the plurality of external devices A generating unit that generates an integrated image including a second display region that displays a display image that is an image displayed by the one external device; an image that allows a user of the image display device to visually recognize the integrated image A display unit. According to the image display device of this aspect, the image display unit displays a plurality of identification images of a plurality of external devices connected to the image display device and one external device selected from the plurality of external devices. The user can visually recognize an integrated image including a display image that is an image to be displayed. Therefore, it is possible to provide an image display device that can integrate display on an external device around the image connection device, in other words, an external device connected to the image display device.

(2) In the image display device according to the aspect described above, the acquisition unit that acquires the display image from the one external device; and the second display area of the integrated image using the acquired display image And an update unit for updating. According to the image display device of this aspect, the image display unit can make the user visually recognize the integrated image in which the display screen is updated for one external device selected from among the plurality of external devices.

(3) In the image display device of the above aspect; the identification image is assigned to a display image at one point in time of the external device, an image representing information for identifying the external device, and the external device And at least one of the images representing the symbols may be included. According to the image display device of this aspect, the first display area of the integrated image is assigned to the external device, a display image at one point in time of the external device, an image representing information for identifying the external device, and the external device. And / or an image representing a given symbol.

(4) In the image display device according to the aspect described above, further: an external device that is estimated to be the user's attention is identified from the plurality of external devices, and the identified external device is identified as the one external device. An external device specifying unit may be provided. According to this form of the image display device, the external device specifying unit specifies an external device that is estimated to have been noticed by the user, and sets the specified external device as one external device. For this reason, the user can select one external device from among a plurality of external devices only by paying attention to the external device using, for example, the line of sight or the movement of the head. As a result, convenience for the user can be improved.

(5) In the image display device according to the above aspect, the image display device further includes: a distance acquisition unit that acquires distances between the image display device and the plurality of external devices; and the generation unit includes the acquired distances. In response, at least one of the display order of the plurality of identification images in the first display area and the size of the identification images in the first display area may be determined. According to the image display device of this aspect, the image display unit includes the display order and size of the plurality of identification images in the first display area according to the distance between the image display device and the plurality of external devices. The user can visually recognize the integrated image in which at least one of them is changed. As a result, the user can know the distance between the image display device and the plurality of external devices according to the display order or size in the first display area.

(6) In the image display device according to the above aspect, the generation unit may update the first display area. According to the image display device of this aspect, the generation unit can also update the first display area.

(7) In the image display device according to the above aspect, the image display unit further includes a display drive unit that generates image light representing the integrated image, and emits the generated image light to the eyes of the user. The integrated image may be visually recognized by the user. According to this form of the image display device, the image display device can be configured as a head-mounted display device.

(8) In the image display device according to the aspect described above, the generation unit sets an aspect ratio of the second display region as an aspect ratio of the region where the image light can be generated by the display driving unit. Generating the integrated image in which the first display area is arranged outside; the image display unit; when there is no operation by the user, the second unit of the integrated image with respect to the display driving unit; The image light of a portion corresponding to a display area is generated; and when there is an operation by the user, the image of at least a portion corresponding to the first display area of the integrated image with respect to the display driving unit Light may be generated. According to the image display device of this aspect, when there is no operation by the user, the image display unit causes the user to visually recognize a portion corresponding to the second display area of the integrated image, and there is an operation by the user Causes the user to visually recognize the portion corresponding to the first display area of the integrated image. As a result, according to the image display device of this aspect, the virtual screen (first display area) is provided outside the normal screen (second display area), and the normal screen and the virtual screen are operated by the user. The display can be switched. As a result, convenience for the user can be improved.

(9) In the image display device of the above aspect, further; for additional presentation for providing augmented reality to the user when it is presumed that the user has paid attention to the one external device An additional presentation information generating unit for generating the information; and the image display unit causes the display driving unit to generate image light representing the generated additional presentation information, and the generated image light is transmitted to the display driving unit. You may make the said user visually recognize the information for the additional presentation by injecting into a user's eyes. According to this form of the image display device, the image display unit visually recognizes additional presentation information for providing augmented reality to the user when the user pays attention to one external device. Can be made. As a result, convenience for the user can be improved.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with a new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including a part or all of the two elements of the generation unit and the image display unit. That is, this apparatus may or may not have the generation unit. Moreover, this apparatus may or may not have an image display unit. Such an apparatus can be realized as an image display apparatus, for example, but can also be realized as an apparatus other than the image display apparatus. Any or all of the technical features of each form of the image display device described above can be applied to this device.

  Note that the present invention can be realized in various modes, for example, an image display device, a control method for the image display device, an image display system using the image display device, a head-mounted display device, a head In the form of a control method for a wearable display device, an image display system using a head-mounted display device, a computer program for realizing the functions of these methods, devices or systems, a recording medium on which the computer program is recorded, etc. Can be realized.

It is explanatory drawing which shows schematic structure of the image display system 1000 in one Embodiment of this invention. 2 is a block diagram functionally showing a configuration of a head mounted display 100. FIG. It is explanatory drawing which shows an example of the area | region information. It is explanatory drawing which shows an example of the virtual image visually recognized by the user. It is a sequence diagram which shows the procedure of an integrated image display process. It is explanatory drawing which shows an example of the integrated image produced | generated by step S126. It is a sequence diagram which shows the procedure of an integrated image update process. It is explanatory drawing which shows a mode that the integrated image in which the 2nd display area was updated is displayed on the head mounted display. It is a block diagram which shows functionally the composition of the head mounted display 100a of a 2nd embodiment. It is a sequence diagram which shows the procedure of the integrated image display process of 2nd Embodiment. It is explanatory drawing which shows an example of the integrated image produced | generated by step S126 of the integrated image display process of 2nd Embodiment. It is a sequence diagram which shows the procedure of the integrated image update process in 2nd Embodiment. It is explanatory drawing which shows a mode that the integrated image in which the 1st display area and the 2nd display area were updated is displayed in the head mounted display 100a. It is a block diagram which shows functionally the structure of the head mounted display 100b of 3rd Embodiment. It is explanatory drawing which shows an example of the area | region information 122b in 3rd Embodiment. It is explanatory drawing for demonstrating an example of operation | movement of the image process part 160b. It is explanatory drawing which shows the structure of the external appearance of the head mounted display in a modification.

A. First embodiment:
A-1. Image display system configuration:
FIG. 1 is an explanatory diagram showing a schematic configuration of an image display system 1000 according to an embodiment of the present invention. The image display system 1000 includes a head-mounted display device 100, a computer 300 (PC 300), a home appliance 400, an AV (Audio Visual) device 500, a portable terminal 600, and a game device 700. Hereinafter, the head-mounted display device 100 is also referred to as an “image display device”. Similarly, the PC 300, the home appliance 400, the AV device 500, the mobile terminal 600, and the game device 700 are collectively referred to as “external device”. The image display system 1000 is a system for causing the head-mounted display device 100 to display an “integrated image” obtained by integrating displays on an external device connected to the head-mounted display device 100.

  The head-mounted display device 100 is an image display device mounted on the head, and is also called a head mounted display (HMD). The head mounted display 100 according to the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. PC 300 is an information communication terminal. In the present embodiment, a notebook personal computer is illustrated as the PC 300, but the PC 300 may be configured by another information communication terminal such as a desktop personal computer. The home appliance 400 is a household electric product. In the present embodiment, a refrigerator is illustrated as the home appliance 400, but the home appliance 400 may be configured by other electric products such as an air conditioner, a microwave oven, a washing machine, a telephone, a fax machine, a printer, and a digital camera.

  The AV device 500 is a generic name for video devices that record, reproduce, and convert video, and audio devices that record, reproduce, and convert audio. In the present embodiment, a hard disk player is exemplified as the AV device 500, but the AV device 500 is configured by other video devices and audio devices such as a television, a DVD (Digital Versatile Disk) player, component audio, and a digital audio player. Also good. The portable terminal 600 is a portable information communication terminal. In the present embodiment, a smartphone is exemplified as the mobile terminal 600, but the mobile terminal 600 may be configured by other mobile information communication terminals such as a mobile phone, a PDA (Personal Digital Assistant), and a tablet terminal. The game device 700 is a device for operating a computer game.

  The external devices of the PC 300, the home appliance 400, the AV device 500, the portable terminal 600, and the game device 700 described above are connected to the head mounted display 100 so as to be capable of wireless communication or wired communication.

A-2. Configuration of head mounted display device:
FIG. 2 is a block diagram functionally showing the configuration of the head mounted display 100. As shown in FIGS. 1 and 2, the head mounted display 100 includes an image display unit 20 that allows the user to visually recognize a virtual image while being mounted on the user's head, and a control unit (controller) that controls the image display unit 20. ) 10. The image display unit 20 and the control unit 10 are connected by a connection unit 40 and transmit various signals through the connection unit 40. A metal cable or an optical fiber can be used for the connection part 40.

A-2-1. Configuration of control unit:
The control unit 10 is a device for controlling the head mounted display 100. The control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a wireless communication unit 132, a GPS module 134, a CPU 140, an interface 180, and transmission units (Tx) 51 and 52. These parts are connected to each other by a bus (not shown) (FIG. 2).

  The input information acquisition unit 110 acquires a signal corresponding to an operation input to an input device such as a touch pad, a cross key, a foot switch, gesture detection, line-of-sight detection, touch detection, or a microphone. Specifically, the foot switch is a switch operated by a user's foot. In gesture detection, a user's gesture is detected using a camera or the like. In this case, the gesture and the command are associated in advance. At the time of gesture detection, a user's fingertip, a ring attached to the user's hand, a tool held by the user, or the like may be used as a mark for motion detection. In the gaze detection, the user's gaze is detected using an infrared sensor or the like. In this case, the movement of the line of sight and the command are associated in advance. In the touch detection, the user touches the contact sensor provided at any location of the housing of the head mounted display 100 (for example, outside the right holding unit 21 and the left holding unit 23 (FIG. 1) of the image display unit 20). Detect operations.

  As described above, the input information acquisition unit 110 acquires signals corresponding to operation inputs to various input devices. In particular, if it is assumed that the operation input by the foot switch, line-of-sight detection, and microphone is acquired, it is difficult for the user to operate using the hand (for example, work scenes such as housework, work scenes in the medical field, construction) The user's convenience when the head mounted display 100 is used can be greatly improved in work scenes that require manual work such as industry and manufacturing.

  The storage unit 120 includes a ROM, a RAM, a DRAM, a hard disk, and the like. The storage unit 120 includes area information 122 and an identification image storage unit 124. The area information 122 is information that defines an “area” included in the integrated image generated in the integrated image display process (FIG. 5). The identification image storage unit 124 is a storage unit for storing identification images of external devices collected in the integrated image display process. Here, the integrated image display process is a process of generating an integrated image and displaying it on the head mounted display 100. The integrated image is an identification image of a plurality of external devices connected to the head mounted display 100, and a display image (hereinafter referred to as an image displayed by one external device selected by the user from the plurality of external devices). , Simply referred to as “display image”). The integrated image update process is a process of dynamically updating the display image of one external device selected by the user among the integrated images and displaying the updated image on the head mounted display 100.

The “external device identification image” is an image for mutually identifying external devices connected to the head mounted display 100. As an identification image of the external device in the present embodiment, for example, an image representing information exemplified in the following a1 to a3 can be used.
(A1) A display image at one time of the external device.
(A2) Display image of the operation screen or setting screen of the external device.
(A3) A shortened moving image of a display image in one period of the external device.

  FIG. 3 is an explanatory diagram illustrating an example of the region information 122. The area information 122 has a rectangular area (hereinafter also referred to as “area 122”). The region 122 preferably has the same aspect ratio as the aspect ratio of the display element of the head mounted display 100 (FIG. 2: right LCD 241 and left LCD 242). The area 122 includes six thumbnail display areas B1 to B6 and an active display area BG. The thumbnail display areas B1 to B6 are areas for displaying a list of identification images of the plurality of external devices by arranging the identification images of the plurality of external devices in the integrated image display process (FIG. 5). The thumbnail display areas B1 to B6 are also referred to as “first display areas”. The active display area BG is an area in which a display image of one external device selected by the user from among a plurality of external devices is arranged in the integrated image display process. The active display area BG is also referred to as a “second display area”.

  In the example of FIG. 3, the thumbnail display areas B <b> 1 to B <b> 3 are arranged adjacent to each other at the left end of the area 122 in the vertical direction. The thumbnail display areas B4 to B6 are arranged at the right end of the area 122 so as to be adjacent to each other in the vertical direction. The thumbnail display areas B1 to B6 are areas having the same size. In the example of FIG. 3, the active display area BG is an area that occupies a portion excluding the thumbnail display areas B <b> 1 to B <b> 6 from the area 122. In FIG. 3, the thumbnail display areas B1 to B6 are hatched with dots, and the active display area BG is hatched.

  Note that the arrangement and size of the thumbnail display areas B1 to B6 shown in FIG. 3 are merely examples, and can be arbitrarily determined. For example, the thumbnail display areas B <b> 1 to B <b> 6 may be arranged at the upper end or the lower end of the area 122. The thumbnail display areas B1 to B6 may be arranged so as to surround the four sides of the area 122. The thumbnail display areas B1 to B6 may have different sizes. Furthermore, the thumbnail display areas B1 to B6 may each overlap at least part of the area. However, in order to improve the visibility of the active display area BG, it is preferable that the thumbnail display areas B1 to B6 are arranged at the end of the area 122 that does not hinder the user's view.

  The power supply 130 supplies power to each part of the head mounted display 100. As the power supply 130, for example, a secondary battery can be used. The wireless communication unit 132 communicates with an external device in accordance with a predetermined wireless communication standard (for example, infrared, short-range wireless communication exemplified by Bluetooth (registered trademark), wireless LAN exemplified by IEEE 802.11, etc.). Wireless communication with The GPS module 134 detects the current position of the user of the head mounted display 100 by receiving a signal from a GPS satellite, and generates current position information representing the current position information of the user. The current position information can be realized by coordinates representing latitude and longitude, for example.

  The CPU 140 reads out and executes a computer program stored in the storage unit 120, thereby generating a generation unit 142, an update unit 144, an operating system (OS) 150, an image processing unit 160, an audio processing unit 170, and a display control unit 190. Function as.

  The generation unit 142 executes integrated image display processing (FIG. 5) to generate an integrated image. The generation unit 142 selects a first display area that displays a list of a plurality of identification images for mutually identifying the plurality of external devices and a plurality of external devices selected from the plurality of external devices by the user. A function as a “generating unit” for generating an integrated image including a second display area for displaying a display image of one external device is realized. The update unit 144 executes integrated image update processing (FIG. 7) to update the integrated image. The update unit 144 uses a function as an “acquisition unit” that acquires a display image of one external device selected by the user from among a plurality of external devices, and uses the acquired display image to obtain the first image of the integrated image. A function as an “update unit” for updating the display area 2 is realized.

  The image processing unit 160 generates a signal based on a video signal input via the OS 150 from the generation unit 142, the interface 180, the wireless communication unit 132, or the like. The image processing unit 160 controls the display on the image display unit 20 by supplying the generated signal to the image display unit 20 via the connection unit 40. The signal supplied to the image display unit 20 differs between the analog format and the digital format.

  For example, in the case of a digital format, a video signal in a state where a digital R signal, a digital G signal, a digital B signal, and a clock signal PCLK are synchronized is input. The image processing unit 160 performs various color tone correction processes such as a known resolution conversion process and brightness saturation adjustment on image data Data composed of a digital R signal, a digital G signal, and a digital B signal. Image processing such as stone correction processing is executed. Thereafter, the image processing unit 160 transmits the clock signal PCLK and the image data Data via the transmission units 51 and 52.

  In the case of an analog format, a video signal including an analog R signal, an analog G signal, an analog B signal, a vertical synchronization signal VSync, and a horizontal synchronization signal HSync is input. The image processing unit 160 separates the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the input signal, and generates a clock signal PCLK using a PLL circuit (not shown) or the like according to their periods. The image processing unit 160 converts the analog R signal, the analog G signal, and the analog B signal into digital signals using an A / D conversion circuit or the like. The image processing unit 160 performs well-known image processing on the image data Data composed of the converted digital R signal, digital G signal, and digital B signal as necessary, and then outputs the clock signal PCLK and the image data Data. The vertical synchronization signal VSync and the horizontal synchronization signal HSync are transmitted via the transmission units 51 and 52. Hereinafter, the image data Data transmitted through the transmission unit 51 is also referred to as “right eye image data Data1”, and the image data Data transmitted through the transmission unit 52 is referred to as “left eye image data Data2”. Also called.

  The display control unit 190 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24 included in the image display unit 20. The control signals include driving ON / OFF of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, driving ON / OFF of the left LCD 242 by the left LCD control unit 212, This is a signal for individually switching driving ON / OFF of the left backlight 222 by the left backlight control unit 202. The display control unit 190 controls the generation and emission of image light in the right display drive unit 22 and the left display drive unit 24 based on these control signals. The display control unit 190 transmits the generated control signal via the transmission units 51 and 52.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and supplies the acquired audio signal to a speaker (not shown) of the right earphone 32 and a speaker (not shown) of the left earphone 34.

  The interface 180 is a predetermined wired communication standard (for example, MicroUSB (Universal Serial Bus), USB, HDMI (High Definition Multimedia Interface, registered trademark), DVI (Digital Visual Interface), VGA (Video Graphics Array), composite, RS- In accordance with 232C (Recommended Standard 232), a wired LAN exemplified by IEEE 802.3), wireless communication is performed with an external device.

A-2-2. Configuration of image display:
The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right optical image display unit 26 (FIG. 1), a left optical image display unit 28 (FIG. 1), and a 9-axis sensor 66. Contains.

  The right display drive unit 22 and the left display drive unit 24 are arranged at positions facing the user's temple when the user wears the image display unit 20. The right display driving unit 22 and the left display driving unit 24 in the present embodiment generate and emit image light representing an image using a liquid crystal display (hereinafter referred to as “LCD”) or a projection optical system. . The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are included.

  The receiving unit 53 receives data transmitted from the transmitting unit 51. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the input clock signal PCLK, right-eye image data Data1, the vertical synchronization signal VSync, and the horizontal synchronization signal HSync. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby converting the illumination light emitted from the right backlight 221 into an image. Modulate into effective image light to represent. The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state.

  The left display drive unit 24 has substantially the same configuration as the right display drive unit 22 and operates in the same manner as the right display drive unit 22. That is, the left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, and a left LCD control unit 212 that functions as a display element. And a left LCD 242 and a left projection optical system 252. Detailed description is omitted. In this embodiment, the backlight method is adopted, but image light may be emitted using a front light method or a reflection method.

  The right optical image display unit 26 and the left optical image display unit 28 are disposed so as to be positioned in front of the left and right eyes of the user when the user wears the image display unit 20 (see FIG. 1). The right optical image display unit 26 includes a right light guide plate 261 and a light control plate (not shown). The right light guide plate 261 is formed of a light transmissive resin material or the like. The right light guide plate 261 guides the image light output from the right display driving unit 22 to the right eye RE of the user while reflecting the image light along a predetermined optical path. For the right light guide plate 261, a diffraction grating or a transflective film may be used. The light control plate is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20. The light control plate protects the right light guide plate 261 and suppresses damage to the right light guide plate 261 and adhesion of dirt. Further, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The left optical image display unit 28 has substantially the same configuration as the right optical image display unit 26 and operates in the same manner as the right optical image display unit 26. That is, the left optical image display unit 28 includes a left light guide plate 262 and a light control plate (not shown), and guides the image light output from the left display drive unit 24 to the left eye LE of the user. Detailed description is omitted.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of detecting the movement of the user's head of the head mounted display 100 is detected. Functions as a detection unit. Here, the movement of the head includes changes in the speed, acceleration, angular velocity, direction, and direction of the head.

  FIG. 4 is an explanatory diagram illustrating an example of a virtual image visually recognized by the user. FIG. 4 illustrates the visual field VR of the user. As described above, the image light guided to both eyes of the user of the head mounted display 100 forms an image on the retina of the user, so that the user can visually recognize the virtual image VI. In the example of FIG. 4, the virtual image VI is an OS standby screen of the head mounted display 100. Further, the user views the outside scene SC through the right optical image display unit 26 and the left optical image display unit 28. As described above, the user of the head mounted display 100 of the present embodiment can see the virtual image VI and the outside scene SC behind the virtual image VI for the portion of the visual field VR where the virtual image VI is displayed. Further, the user can directly see the outside scene SC through the right optical image display unit 26 and the left optical image display unit 28 for the portion of the visual field VR where the virtual image VI is not displayed. In the present specification, “the head mounted display 100 displays an image” includes allowing the user of the head mounted display 100 to visually recognize a virtual image.

A-3. Integrated image display processing:
FIG. 5 is a sequence diagram showing the procedure of the integrated image display process. The integrated image display process of the present embodiment is a process of generating an integrated image and displaying it on the head mounted display 100. The integrated image is an image including both identification images of a plurality of external devices connected to the head mounted display 100 and a display image of one external device selected by the user from the plurality of external devices. It is. The integrated image display process is mainly executed by the generation unit 142.

  In step S100, an application for executing the integrated image display process is activated. The activation of the application in step S100 may be triggered by the input information acquisition unit 110 detecting the activation operation by the user, or may be triggered by detection of an activation command from another application. By the activation of the application in step S100, the functions of the generation unit 142 and the update unit 144 are realized for the CPU 140.

  In step S102, the wireless communication unit 132 or the interface 180 detects that the PC 300 as an external device is connected. Hereinafter, a case where the head mounted display 100 and the PC 300 communicate using a wireless LAN compliant with IEEE 802.11 will be exemplified. Thereafter, the generation unit 142 establishes a connection with the PC 300.

In step S104, the PC 300 acquires an image of the display screen of the PC 300. Specifically, the PC 300 acquires any one of the frame images shown in the following b1 to b3. The following b1 corresponds to the above a1, the following b2 corresponds to the above a2, and the following b3 corresponds to the above a3.
(B1) A frame image obtained by rendering a screen displayed on the PC 300 at one point of time, such as content being played back on the PC 300 or an application GUI (Graphical User Interface).
(B2) A frame image obtained by rendering an operation screen (that is, a standby screen) or a setting screen (that is, an OS or BIOS setting screen) of the PC 300.
(B3) A plurality of frame images obtained by rendering a screen displayed on the PC 300 during one period, such as content being played back on the PC 300 and application GUI.

  In step S <b> 106, the PC 300 transmits the acquired frame image to the head mounted display 100. The generation unit 142 of the head mounted display 100 acquires this frame image via the wireless communication unit 132. At this time, the generation unit 142 and the wireless communication unit 132 function as an “acquisition unit”.

  In step S108, the generation unit 142 causes the identification image storage unit 124 to store the acquired frame image as an “identification image of the external device”. Note that the generation unit 142 may store the frame image in the identification image storage unit 124 after reducing the frame image. Then, the storage capacity consumed by the identification image storage unit 124 can be reduced. In step S108, the generation unit 142 may disconnect the connection with the PC 300.

  In step S <b> 110, the generation unit 142 performs the same processing as steps S <b> 102 to S <b> 108 with the home appliance 400. Similarly, the generation unit 142 performs similar processing with the AV device 500 in step S112, with the mobile terminal 600 in step S114, and with the game device 700 in step S116. As a result of steps S <b> 110 to S <b> 116, identification images of all external devices connected to the head mounted display 100 are stored in the identification image storage unit 124.

  In step S118, the generation unit 142 acquires an image of the display screen of the head mounted display 100. Specifically, any one of the frame images shown in b1 to b3 for the head mounted display 100 is acquired.

  In step S120, the generation unit 142 causes the identification image storage unit 124 to store the acquired frame image as an “identification image of the image display device”. Note that the generation unit 142 may store the frame image in the identification image storage unit 124 after reducing the frame image. As a result of steps S118 and S120, the identification image storage unit 124 stores an identification image of the head mounted display 100 itself.

  In step S124, the generation unit 142 acquires a plurality of identification images stored in the identification image storage unit 124, and sorts the acquired identification images according to a predetermined condition. The sorting conditions are, for example, the ID (for example, product number or serial number) of the external device from which the identification image is acquired, the name of the external device (for example, the name of the product) from which the identification image is acquired, and the identification image storage unit 124. The order in which the identification images are acquired, the distance between the external device that is the acquisition source of the identification images and the head mounted display 100, and the like can be used. The distance between the external device and the head mounted display 100 is, for example, the current position information of the head mounted display 100 detected by the GPS module 134 by the OS 150 and the current position information of the external device acquired from the external device. , Can be obtained by comparing. In this case, the OS 150 functions as a “distance acquisition unit”. Note that the “distance between the external device and the head mounted display 100” in the present embodiment includes a rough distance.

  In step S126, the generation unit 142 generates an integrated image using the sorted identification image. Specifically, the generation unit 142 arranges the plurality of identification images sorted in step S124 in the thumbnail display areas B1 to B6 (first display areas) of the area information 122, respectively. The generation unit 142 also arranges the black image in the active display area BG (second display area) of the area information 122. In this active display area BG, a display image of one external device selected by the user in an integrated image change process described later is inserted.

  In step S126, for each identification image, the generation unit 142 changes the shape of the identification image, changes the transmittance of the identification image, color conversion processing to the identification image, changes the size of the identification image, and the identification image. It may be possible to give decorations (characters, figures, symbols, etc.), etc. For example, the generation unit 142 may determine the size of the identification image according to the distance between the external device and the head mounted display 100 when changing the size of the identification image. For example, if the identification image of the external device that is closer to the head-mounted display 100 is made larger, it is possible for the user who has viewed the integrated image to connect the head-mounted display 100 and each external device. It is possible to visually grasp the distance. Similarly, the shape, the transmittance, the content of the color conversion process, and the content of the decoration may be determined according to the distance between the head mounted display 100 and the external device.

  FIG. 6 is an explanatory diagram illustrating an example of the integrated image generated in step S126. In the thumbnail display area B1 of the integrated image GI, a display image G1 at one time of the head mounted display 100 is arranged as an identification image of the head mounted display 100. Similarly, a display image G2 at one point of the PC 300 is displayed in the thumbnail display area B2 as an identification image of the PC 300, and a display image G3 of the setting screen of the household appliance 400 is displayed as an identification image of the household appliance 400 in the thumbnail display area B3. In the thumbnail display area B4, the display image G4 of the operation screen of the AV device 500 is displayed as an identification image of the AV device 500, and the identification image of the mobile terminal 600 is displayed in the thumbnail display region B5 at one point of time of the mobile terminal 600. A display image G5 is displayed in the thumbnail display area B6 as a display image G6 at one point in time of the game device 700 as an identification image of the game device 700. In the example of FIG. 6, the aspect ratio of the thumbnail display areas B1 to B6 does not match the aspect ratio of the identification images G1 to G6 of the external devices. In step S126 of FIG. 5, the generation unit 142 may perform a process of matching the aspect ratios of the identification images G1 to G6 of the external devices with the aspect ratios of the thumbnail display areas B1 to B6.

  In step S128 of FIG. 5, the generation unit 142 displays an integrated image on the head mounted display 100. Specifically, the generation unit 142 transmits the integrated image generated in step S126 to the image processing unit 160. The image processing unit 160 that has received the integrated image performs the above-described display processing. As a result, the image light guided to both eyes of the user of the head mounted display 100 forms an image on the retina of the user, so that the user of the head mounted display 100 visually recognizes the virtual image of the integrated image in the field of view. Can do. In other words, the head mounted display 100 can display an integrated image.

A-4. Integrated image update processing:
FIG. 7 is a sequence diagram illustrating the procedure of the integrated image update process. The integrated image update process of the present embodiment is a process of dynamically updating the display image of one external device selected by the user among the integrated images and causing the head mounted display 100 to display it. The integrated image update process is mainly executed by the update unit 144.

  In step S <b> 200, the update unit 144 acquires a selection operation for selecting one external device from a plurality of external devices via the input information acquisition unit 110. This selection operation is an operation performed by the user of the head mounted display 100. The user can perform a selection operation using an input device (for example, a touch pad, a cross key, a foot switch, gesture detection, line-of-sight detection, touch detection, a microphone, or the like) included in the head mounted display 100.

  In step S202, the update unit 144 transmits a request for a display image to the one external device selected by the user. In the figure, an example in which the external device selected by the user is the PC 300 is shown.

  The external device (PC 300) that has received the request for the display image in step S204 rendered the image of the display screen of the PC 300, that is, the screen currently displayed on the PC 300, such as the content being played back on the PC 300 or the application GUI. Get a frame image.

  In step S206, the PC 300 transmits the acquired frame image to the head mounted display 100. The update unit 144 of the head mounted display 100 acquires the first image via the wireless communication unit 132. At this time, the update unit 144 and the wireless communication unit 132 function as an “acquisition unit”.

  In step S208, the update unit 144 updates the active display region BG (second display region) in the integrated image generated in step S126 of the integrated image display process (FIG. 5) using the acquired frame image. To do.

  In step S210, the update unit 144 displays the updated integrated image. Details are the same as step S128 of the integrated image display process (FIG. 5).

  In step S212, the update unit 144 performs the processing in steps S202 to S210 for the selected external device (PC 300) until a selection operation of another external device different from the external device acquired in step S200 is acquired. repeat. On the other hand, when the selection operation of another external device is acquired, the updating unit 144 performs the processes of steps S202 to S212 for the other external device.

  FIG. 8 is an explanatory diagram showing a state where an integrated image in which the second display area is updated is displayed on the head mounted display 100. As shown in FIG. 8, the user of the head mounted display 100 is selected from the plurality of identification images of the plurality of external devices connected to the head mounted display 100 and the plurality of external devices in the visual field VR. An integrated image GI including a display image of one external device can be visually recognized as a virtual image VI. In the example of FIG. 8, the plurality of identification images of the plurality of external devices are the identification image G1 of the head mounted display 100 described in FIG. 6, the identification image G2 of the PC 300, the identification image G3 of the home appliance 400, and the AV device. An identification image G4 of 500, an identification image G5 of the portable terminal 600, and an identification image G6 of the game machine 700. In the example of FIG. 8, the display image of the selected one external device is the display image IM that is an image displayed by the PC 300. The head mounted display 100 of the present embodiment is a transmissive head mounted display device. Therefore, as described in FIG. 4, the user can visually recognize the background through the virtual image VI. About this background, illustration in FIG. 8 is abbreviate | omitted for convenience of illustration.

  As described above, according to the integrated image display process (FIG. 5) and the integrated image update process (FIG. 7) of the first embodiment, the image display unit 20 is connected to the head mounted display 100 (image display apparatus). A plurality of identification images G2 to G7 of a plurality of external devices (PC 300 to game device 700) and a display image IM of one external device (PC 300 in the above example) selected from the plurality of external devices. The integrated image GI can be visually recognized by the user. For this reason, the head mounted display 100 which can integrate the display in the periphery of the head mounted display 100, in other words, the external device connected to the head mounted display 100 can be provided.

  Further, according to the integrated image update process (FIG. 7) of the first embodiment, the image display unit 20 is one external device (PC300 in the above example) selected from a plurality of external devices (PC300 to game device 700). The user can visually recognize the integrated image GI whose display screen is dynamically updated. According to the integrated image update process and the integrated image display process (FIG. 5) described above, the head mounted display 100 dynamically updates only the active display area BG (second display area) in the integrated image, and thumbnails. The display areas B1 to B6 (first display area) are not updated. Therefore, the head mounted display 100 only needs to acquire the identification image once from each external device, and communication traffic between the head mounted display 100 and each external device can be reduced. In addition, the display image of one external device selected by the user can be dynamically updated.

A-5. Variations in integrated image update processing:
The integrated image update process (FIG. 7) may be modified as follows. The variations shown below may be added alone or in combination.

A-5-1. First variation:
In the first variation, it is possible to perform an “operation for selecting one device from external devices” only by paying attention to the external device intended by the user. In this variation, the CPU 140 of the head mounted display 100 further includes an external device specifying unit.

The external device specifying unit specifies an external device that is estimated to have been noticed by the user, and notifies the update unit 144 of the specified external device in step S200 of FIG. The external device specifying unit can specify, for example, the external device that is estimated to be the user's attention by the following methods c1 to c3.
(C1) A directional wireless antenna is provided at a position corresponding to the eyebrows of the user outside the housing of the head mounted display 100 (in other words, between the right optical image display unit 26 and the left optical image display unit 28). The configuration. Then, it can be estimated that the external device that has received the radio wave from the wireless antenna is an external device viewed by the user. For this reason, the external device identifying unit identifies the external device that is estimated to have received the user's attention by receiving a message to that effect from the external device that has received the radio wave from the directional wireless antenna. Can do.

(C2) An infrared light emitting unit is provided at a position corresponding to the eyebrows of the user outside the housing of the head mounted display 100. Further, the external device includes an infrared light receiving unit. Then, it can be estimated that the external device that has received the infrared rays from the head mounted display 100 is an external device viewed by the user. Therefore, the external device specifying unit can specify the external device that is estimated to have received the user's attention by receiving a message to that effect from the external device that has received the infrared rays. (C3) The external device specifying unit obtains the orientation of the head mounted display 100 (the orientation of the user's head) obtained by the 9-axis sensor 66 (FIG. 2) and the current position information obtained by the GPS module 134 (FIG. 2). The position information of each external device can be used to identify the external device that is estimated to have received the user's attention.

  Thus, according to the first variation, the external device specifying unit specifies and specifies an external device that is presumed that the user has paid attention from a plurality of external devices (PC 300 to game device 700). The external device is one external device selected by the user (FIG. 7, step S200). For this reason, the user can select one external device from among a plurality of external devices only by paying attention to the intended external device using, for example, the line of sight or the movement of the head. As a result, convenience for the user can be improved. In addition, said c1-c3 are an example to the last, and the external apparatus specific | specification part can identify the external apparatus which the user turned attention according to a motion of a user's head, a user's eyes | visual_axis, etc. .

A-5-2. Second variation:
In the second variation, a tag for providing augmented reality (AR) is attached to one external device selected by the user (FIG. 7, the external device described in step S200). Is possible. In this variation, the CPU 140 of the head mounted display 100 further includes an additional presentation information generation unit.

  The additional presentation information generation unit provides augmented reality to the user when it is estimated that the user has paid attention to the one external device selected by the user in step S200 of FIG. Generate information for additional presentation. Here, the method of estimating that “the user has paid attention to a specific external device” uses a directional antenna, infrared rays, current position information, and head orientation, similar to the above c1 to c3. can do. Further, for example, it may be estimated that the user has paid attention to a specific external device by recognizing an image of an outside scene captured by a WEB camera or the like. The information for additional presentation means adding, deleting, emphasizing, or attenuating some information with respect to the external device. Specifically, the information for additional presentation is, for example, a balloon or a note-shaped tag including a character string informing that the external device is selected by the user, or an external device selected by the user. Such as an icon (a picture, a character, a symbol, etc.) for informing the user, a frame for informing that the user has selected an external device (a frame arranged so as to decorate the external device), or the like.

  The additional presentation information generation unit transmits the generated additional presentation information to the image processing unit 160. The image processing unit 160 that has received the integrated image performs the above-described display processing. As a result, the user of the head mounted display 100 can visually recognize a virtual image of additional presentation information (tag) in the field of view.

  As described above, according to the second variation, the image display unit 20 is expanded to the user when the user pays attention to the one external device selected by the user in step S200 of FIG. Additional presentation information for providing a sense of reality can be viewed. As a result, convenience for the user can be improved.

B. Second embodiment:
In the second embodiment of the present invention, a configuration in which an image different from the image exemplified in the first embodiment is used as the identification image of the external device will be described. Below, only the part which has a different structure and operation | movement from 1st Embodiment is demonstrated. In the figure, the same components as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment described above, and detailed description thereof is omitted.

B-1. Image display system configuration:
The image display system of the second embodiment is the same as that of the first embodiment shown in FIG. 1 except that a head mounted display 100a is provided instead of the head mounted display 100.

B-2. Configuration of head mounted display device:
FIG. 9 is a block diagram functionally showing the configuration of the head mounted display 100a of the second embodiment. The difference from the first embodiment shown in FIG. 2 is that a control unit 10 a is provided instead of the control unit 10. The control unit 10a includes a generation unit 142a instead of the generation unit 142, and an update unit 144a instead of the update unit 144. The generation unit 142a differs from the first embodiment shown in FIG. 5 in the content of the integrated image display process. The update unit 144a differs from the first embodiment shown in FIG. 7 in the content of the integrated image update process.

In the integrated image display process and the integrated image update process of the second embodiment, instead of the “identification image of the external device” exemplified in a1 to a3 of the first embodiment, one of the following a4 and a5 is replaced with “external Used as “identification image of device”.
(A4) An image representing information for identifying an external device. The information for identifying the external device includes, for example, an external device ID (for example, product number or serial number), an external device name (for example, product name), an external device status (for example, product property), and the like. .
(A5) An image representing a symbol assigned to an external device. Here, the “symbol” is, for example, an icon representing the presence of an external device, a graphic or symbol representing the external device in an abstract manner, and the like.

B-3. Integrated image display processing:
FIG. 10 is a sequence diagram illustrating a procedure of the integrated image display process according to the second embodiment. The integrated image display process of the second embodiment is a process of generating an integrated image and displaying it on the head mounted display 100a, as in the first embodiment (FIG. 5). The difference from the first embodiment is that the images listed in a4 and a5 are used as the identification image of the external device. The integrated image display process is mainly executed by the generation unit 142a.

  First, as in the first embodiment described with reference to FIG. 5, an application is activated (step S100), and a connection between the head mounted display 100a and the PC 300 is established (step S102).

In step S300, the PC 300 acquires device information of the PC 300. Specifically, the PC 300 acquires any one of the following information b4 and b5 (not necessarily an image). The following b4 corresponds to the above a4, and the following b5 corresponds to the above a5.
(B4) Information for identifying the PC 300. For example, information indicating the status of the PC 300 such as the manufacturing number and serial number, the name of the PC 300, the properties of the PC 300, and the like.
(B5) An image representing the symbol of the PC 300. For example, an icon image of the PC 300.

  In step S302, the PC 300 transmits the acquired device information to the head mounted display 100a. The generation unit 142a of the head mounted display 100a acquires this device information via the wireless communication unit 132. At this time, the generation unit 142a and the wireless communication unit 132 function as an “acquisition unit”.

  In step S303, the generation unit 142a generates an image representing the acquired device information. The generation unit 142a stores the generated image in the identification image storage unit 124 as an “identification image of the external device”. Note that when the device information received from the PC 300 is in the form of an image as in b5 above, generation of an image representing the device information can be omitted.

  In step S304, the generation unit 142a disconnects the connection between the head mounted display 100a and the PC 300. In this way, communication traffic and power consumption can be reduced by immediately disconnecting unnecessary connections.

  In step S <b> 306, the generation unit 142 performs the same processing as in steps S <b> 102 and steps S <b> 300 to S <b> 304 with the home appliance 400. Similarly, the generation unit 142 performs similar processing with the AV device 500 in step S308, with the mobile terminal 600 in step S308, and with the game device 700 in step S310. As a result, the identification image storage unit 124 stores identification images of all external devices connected to the head mounted display 100a.

  In step S314, the generation unit 142a acquires device information of the head mounted display 100. Specifically, any information shown in b4 and b5 above for the head mounted display 100a is acquired.

  In step S316, the generation unit 142a generates an image representing the acquired device information and stores it in the identification image storage unit 124 as an “identification image of the image display device”. As a result of steps S314 and S316, the identification image storage unit 124 stores the identification image of the head mounted display 100a itself.

  Thereafter, as in the first embodiment described with reference to FIG. 5, the generation unit 142a sorts a plurality of identification images stored in the identification image storage unit 124 (step S124), and generates an integrated image (step S126). ), An integrated image is displayed (step S128). Details are as described in FIG.

  FIG. 11 is an explanatory diagram illustrating an example of the integrated image generated in step S126 of the integrated image display process according to the second embodiment. In the thumbnail display area B1 of the integrated image GI, an image J1 representing the product number of the head mounted display 100 is arranged as an identification image of the head mounted display 100. Similarly, an icon image J2 of the PC 300 is displayed in the thumbnail display area B2, and an icon image J2 of the PC 300 is displayed in the thumbnail display area B3. An image J3 representing the name of the product of the home appliance 400 is displayed in the thumbnail display area B3. In B4, as an identification image of the AV device 500, an icon image J4 of the AV device 500 is displayed. In a thumbnail display region B5, as an identification image of the mobile terminal 600, an image J5 representing the product number of the mobile terminal 600 is displayed in the thumbnail display region B6. The image J6 representing the product number of the game machine 700 is arranged as an identification image of the game machine 700, respectively.

B-4. Integrated image update processing:
FIG. 12 is a sequence diagram illustrating a procedure of the integrated image update process in the second embodiment. The integrated image update process of the second embodiment is a process of dynamically displaying the second display area of the integrated image, as in the first embodiment (FIG. 7). The difference from the first embodiment is that the images listed in a4 and a5 are used as the identification image of the external device, and the display image acquired from the external device is used to update the second display area. Thus, the identification image of the external device in the identification image storage unit 124 is updated. The integrated image update process is mainly executed by the update unit 144a.

  First, as in the first embodiment described with reference to FIG. 7, a selection operation for selecting one external device from a plurality of external devices is acquired (step S200), and the display screen is displayed from the selected one external device. An image (frame image) is acquired (steps S202 to S206).

  In step S400, the update unit 144a stores the acquired frame image in the identification image storage unit 124 as an “identification image of the external device”. At this time, the update unit 144a overwrites the identification image of the same external device already stored in step S303 of the integrated image display process (FIG. 10). In step S400, the acquired frame image may be stored in the identification image storage unit 124 after being reduced.

  In step S402, the update unit 144a acquires a plurality of identification images stored in the identification image storage unit 124, and sorts the acquired identification images according to a predetermined condition. Details are the same as step S124 of the integrated image display process (FIG. 5).

  In step S404, the update unit 144a updates the thumbnail display areas B1 to B6 (first display areas) of the integrated image using the sorted identification images. Details are the same as in step S126 of the integrated image display process (FIG. 5).

  In step S408, the update unit 144a updates the active display region BG (second display region) in the integrated image updated in step S404 using the frame image acquired in step S206.

  In step S210, the update unit 144a displays the updated integrated image. Details are the same as step S128 of the integrated image display process (FIG. 5).

  In step S410, the update unit 144a performs steps S202, S204, and S206 on the selected external device (PC 300) until a selection operation of another external device different from the external device acquired in step S200 is acquired. , S408 and S210 are repeated. On the other hand, when the selection operation of another external device is acquired, the updating unit 144 performs a series of processes of steps S202 to S410 on the other external device.

  FIG. 13 is an explanatory diagram showing a state in which an integrated image in which the first display area and the second display area are updated is displayed on the head mounted display 100a. As shown in FIG. 13, the user of the head mounted display 100a is selected from the plurality of identification images of the plurality of external devices connected to the head mounted display 100a and the plurality of external devices in the visual field VR. An integrated image GI including a display image of one external device can be visually recognized as a virtual image VI. In the example of FIG. 13, the identification images of a plurality of external devices are the identification image J1 of the head mounted display 100a described in FIG. 11, the identification image J2 of the PC 300, the identification image J3 of the home appliance 400, and the AV device. The identification image J4 of 500, the identification image J5 of the portable terminal 600, and the identification image J6 of the game machine 700. Here, for the identification image J2 of the PC 300, the updated identification image in step S400 of the integrated image update process (FIG. 12) is displayed. In the example of FIG. 13, the display image of the selected one external device is the display image IM that is an image displayed by the PC 300. As in FIG. 8, in FIG. 14, the background that originally transmits the virtual image VI is not shown.

  As described above, according to the integrated image display process (FIG. 10) and the integrated image update process (FIG. 12) of the second embodiment, the same effects as those of the first embodiment can be obtained. Furthermore, in the integrated image display process and the integrated image update process of the second embodiment, an image of a different type from the image exemplified in the first embodiment can be used as the identification image of the external device.

B-5. Variations in integrated image update processing:
Also in the integrated image update process (FIG. 12) of the second embodiment, the first variation and the second variation described above can be applied.

C. Third embodiment:
In the third embodiment of the present invention, a configuration in which thumbnail display areas B1 to B6 (first display areas) of combined images are assumed to be virtual display areas will be described. Below, only the part which has a different structure and operation | movement from 1st Embodiment is demonstrated. In the figure, the same components as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment described above, and detailed description thereof is omitted.

C-1. Image display system configuration:
The image display system of the third embodiment is the same as that of the first embodiment shown in FIG. 1 except that a head mounted display 100b is provided instead of the head mounted display 100.

C-2. Configuration of head mounted display device:
FIG. 14 is a block diagram functionally showing the configuration of the head mounted display 100b of the third embodiment. The difference from the first embodiment shown in FIG. 2 is that a control unit 10 b is provided instead of the control unit 10. The control unit 10b includes region information 122b instead of the region information 122, and an image processing unit 160b instead of the image processing unit 160.

  FIG. 15 is an explanatory diagram illustrating an example of the region information 122b in the third embodiment. Similarly to the first embodiment shown in FIG. 3, the area information 122b of the third embodiment includes six thumbnail display areas B1 to B6 (first display area) and an active display area BG (second display area). ) And. The use of each area is as described in FIG. The thumbnail display areas B1 to B3 and the thumbnail display areas B4 to B6 are disposed outside the left and right in the horizontal direction with the active display area BG interposed therebetween. The aspect ratio of the active display area BG is the same as the aspect ratio of the area where image light can be generated by the display drive unit, in other words, the aspect ratio of the display element (FIG. 2: right LCD 241 and left LCD 242) of the head mounted display 100b. is there.

  In the example of FIG. 15, the thumbnail display areas B1 to B3 are respectively arranged adjacent to each other in the vertical direction outside the left side of the area 122b. The thumbnail display areas B4 to B6 are arranged outside the right side of the area 122 and adjacent to each other in the vertical direction. The thumbnail display areas B1 to B6 are areas having the same size. In FIG. 15, the thumbnail display areas B1 to B6 are hatched with dots, and the active display area BG is hatched.

  Note that the arrangement and size of the thumbnail display areas B1 to B6 shown in FIG. 15 are examples, and can be arbitrarily determined. It is sufficient that the thumbnail display areas B1 to B6 are arranged outside the active display area BG (second display area). For this reason, for example, the thumbnail display areas B1 to B6 may be arranged outside the upper side or the lower side of the active display area BG. Further, the thumbnail display areas B1 to B6 may be arranged so as to surround the outer four sides of the active display area BG. The thumbnail display areas B1 to B6 may have different sizes. Furthermore, the thumbnail display areas B1 to B6 may each overlap at least part of the area.

C-3. Integrated image display processing:
The procedure of the integrated image display process in the third embodiment is the same as that of the first embodiment shown in FIG. 5 except for the following steps d1 and d2.
(D1) In step S126, the generation unit 142 generates an integrated image using the region information 122b described in FIG.
(D2) In step S128, the image processing unit 160b corresponds to the active display area BG (second display area, hatched area in FIG. 15) of the integrated image received from the generation unit 142. Are displayed on the head mounted display 100b. Then, the image processing unit 160b is a portion corresponding to the thumbnail display areas B1 to B6 (first display area, areas in which dots are hatched in FIG. 15) triggered by obtaining a predetermined operation from the user. Is displayed on the head mounted display 100b. The predetermined trigger uses, for example, a user's operation by an input device (for example, touch pad, cross key, foot switch, gesture detection, gaze detection, touch detection, microphone, etc.) provided in the head mounted display 100b. Can do.

  FIG. 16 is an explanatory diagram for explaining an example of the operation of the image processing unit 160b. For example, the image processing unit 160b uses, as a trigger, that the user has touched the left holding unit 23 of the head mounted display 100b as a trigger to display an integrated image range displayed on the head mounted display 100b as illustrated in FIG. Slide to the left. In this way, the image processing unit 160b allows the user to visually recognize the first display area (a part thereof) and the second display area (a part thereof). Similarly, the image processing unit 160b slides the range of the integrated image to be displayed on the head mounted display 100b to the right, triggered by the user touching the right holding unit 21 of the head mounted display 100b. In the area information 122b, when the thumbnail display areas B1 to B6 (first display areas) are provided above and below the active display area BG (second display area), the user can use the case of the head mounted display 100b. The same processing can be performed with the touching of the top and bottom of as a trigger.

C-4. Integrated image update processing:
The procedure of the integrated image update process in the third embodiment is the same as that of the first embodiment shown in FIG. 7 except for the procedure of the image processing unit 160b in step S210 (same as d2 above).

  As described above, according to the integrated image display process (FIG. 5) and the integrated image update process (FIG. 7) of the third embodiment, the same effects as those of the first embodiment can be obtained. Furthermore, according to the integrated image display process and the integrated image update process of the third embodiment, the image display unit 20 displays the active image area BG (second display area) of the integrated image when there is no operation by the user. When the user visually recognizes the corresponding part and there is an operation by the user, the user can visually recognize the part corresponding to the thumbnail display areas B1 to B6 (first display area) of the integrated image. As a result, according to the head mounted display 100b (image display device) of this embodiment, the virtual screen (thumbnail display areas B1 to B6, first display) outside the normal screen (active display area BG, second display area). Area) and switching between the normal screen and the virtual screen can be performed by the user's operation. As a result, convenience for the user can be improved.

C-5. Variations in integrated image update processing:
Also in the integrated image update process (FIG. 7) of the third embodiment, the first variation and the second variation described above can be applied.

D. Variation:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

・ Modification 1:
In the said embodiment, it illustrated about the structure of the image display system. However, the configuration of the image display system can be arbitrarily determined without departing from the gist of the present invention. For example, addition, deletion, conversion, and the like of each device configuring the image display system can be performed. In addition, it is possible to change the network configuration of the devices constituting the image display system.

  For example, as the image display device, a projector, a monitor, a television, or the like can be employed instead of the head mounted display.

  For example, a cloud server via the Internet INT may be used as one of the external devices in the above embodiment. Even in this case, the generation unit and the update unit can obtain the same effect as that of the first to third embodiments by performing the same processing as that of the first to third embodiments.

Modification 2
In the said embodiment, it illustrated about the structure of the head mounted display. However, the configuration of the head-mounted display can be arbitrarily determined without departing from the gist of the present invention. For example, each component can be added, deleted, converted, and the like.

  The allocation of components to the control unit and the image display unit in the above embodiment is merely an example, and various aspects can be employed. For example, the following aspects may be adopted. (I) A mode in which processing functions such as a CPU and a memory are mounted on the control unit, and only a display function is mounted on the image display unit. (Ii) Processing functions such as a CPU and a memory are provided in both the control unit and the image display unit. (Iii) a mode in which the control unit and the image display unit are integrated (for example, a mode in which the control unit is included in the image display unit and functions as a glasses-type wearable computer), (iv) instead of the control unit A mode in which a smartphone or a portable game machine is used, and (v) a mode in which the connection unit (code) is eliminated by adopting a configuration in which the control unit and the image display unit can perform wireless communication and wireless power feeding. (Vi) A mode in which the touch pad is eliminated from the control unit and the touch pad is provided in the image display unit.

  In the above embodiment, for convenience of explanation, the control unit includes a transmission unit, and the image display unit includes a reception unit. However, each of the transmission unit and the reception unit of the above-described embodiment has a function capable of bidirectional communication, and can function as a transmission / reception unit. Further, for example, the control unit illustrated in FIG. 5 is connected to the image display unit via a wired signal transmission path. However, the control unit and the image display unit may be connected by connection via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth.

  For example, the configurations of the control unit and the image display unit illustrated in FIG. 2 can be arbitrarily changed. Specifically, for example, the control unit includes various input devices (for example, operation sticks, keyboards) in addition to the above-described various input devices (touch pad, cross key, foot switch, gesture detection, line of sight detection, microphone). A mouse, etc.). Moreover, in the said embodiment, although the secondary battery was used as a power supply, as a power supply, not only a secondary battery but a various battery can be used. For example, a primary battery, a fuel cell, a solar cell, a thermal cell, or the like may be used.

  For example, the head mounted display is a binocular transmissive head mounted display, but may be a monocular head mounted display. Further, it may be configured as a non-transmissive head mounted display in which the transmission of the outside scene is blocked when the user wears the head mounted display. In the above embodiment, it is assumed that the image display unit is a head-mounted display that is worn like glasses, but, for example, a head that employs an image display unit of another shape such as an image display unit that is worn like a hat. It is good also as a mount display. The earphone may be an ear-hook type or a headband type, or may be omitted. Furthermore, for example, it may be configured as a head-up display (HUD, Head-Up Display) mounted on a vehicle such as an automobile or an airplane. In addition, it may be configured as a head mounted display built in a body protective device such as a helmet.

  FIG. 17 is an explanatory diagram showing an external configuration of a head-mounted display in a modified example. In the example of FIG. 17A, the image display unit 20c includes a right optical image display unit 26c instead of the right optical image display unit 26, and includes a left optical image display unit 28c instead of the left optical image display unit 28. ing. The right optical image display unit 26c and the left optical image display unit 28c are formed smaller than the optical member of the first embodiment, and are disposed obliquely above the right and left eyes of the user when the head mounted display is mounted. Yes. In the example of FIG. 17B, the image display unit 20d includes a right optical image display unit 26d instead of the right optical image display unit 26, and includes a left optical image display unit 28d instead of the left optical image display unit 28. ing. The right optical image display unit 26d and the left optical image display unit 28d are formed smaller than the optical member of the first embodiment, and are disposed obliquely below the right and left eyes of the user when the head mounted display is mounted. Yes. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit completely covers the user's eye. It can also be realized as an unmounted head mounted display.

  For example, in the above embodiment, the display driving unit is configured using a backlight, a backlight control unit, an LCD, an LCD control unit, and a projection optical system. However, the above aspect is merely an example. The display drive unit may include a component for realizing another method together with these components or instead of these components. For example, the display driving unit may include an organic EL (Organic Electro-Luminescence) display, an organic EL control unit, and a projection optical system. Further, for example, the display drive unit can use a digital micromirror device or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head-mounted display device.

  For example, each function unit such as a generation unit, a notification unit, an image processing unit, a display control unit, and an audio processing unit is realized by a CPU developing and executing a computer program stored in a ROM or a hard disk on the RAM. It was described as a thing. However, these functional units may be configured using an ASIC (Application Specific Integrated Circuit) designed to realize the function.

・ Modification 3:
In the said embodiment, an example of the integrated image display process was shown. However, the procedure of the integrated image display process is merely an example, and various modifications can be made. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

  For example, the generation unit may dynamically change the sizes of the thumbnail display areas B1 to B6 according to the distance between the external device from which the identification image is acquired and the head mounted display. In this way, the user who has visually recognized the integrated image has a head-mounted display (image display device) and a plurality of external devices depending on the size of each image in the thumbnail display regions B1 to B6 (first display region). You can know the distance between.

  In the above embodiment, the generation unit does not update the thumbnail display areas B1 to B6 (first display areas). However, the generation unit may update the thumbnail display areas B1 to B6 (first display area) upon occurrence of a predetermined trigger. The predetermined trigger can be arbitrarily determined. For example, the occurrence of a requested operation by the user, the occurrence of a predetermined operation by the user, the elapse of a predetermined time since the previous integrated image generation, the OS of the head mounted display Or an internal request from another application. In this way, the generation unit can update the first display area only when a predetermined trigger occurs. For this reason, it is possible to reduce communication traffic between the head mounted display (image display device) and the external device.

  For example, the identification image and the integrated image described in the above embodiment are assumed to be two-dimensionally expressed images. However, the image processing unit may express the identification image and the integrated image three-dimensionally by using different data for the right-eye image data and the left-eye image data.

  In the embodiment described above, an integrated image including the first display area and the second display area is generated in the control unit and displayed on the image display unit. However, the integrated image only needs to be displayed by the image display unit, and the control unit does not necessarily have to “generate” the integrated image. Specifically, for example, an image representing the first display area and an image representing the second display area are separately generated in the control unit, and these separate images are displayed together on the image display unit. An integrated image may be displayed.

-Modification 4:
In the said embodiment, an example of the integrated image update process was shown. However, the procedure of the integrated image update process is merely an example, and various modifications can be made. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

-Modification 5:
In the above embodiment, an example of the region information has been shown. The area information is merely an example, and various changes can be made. For example, a component can be added, deleted, or changed.

  For example, a plurality of different area information may be stored in the storage unit. In this case, the integrated image is obtained from a plurality of different region information depending on arbitrary conditions such as the user's preference (setting) of the head mounted display, the movement of the user's line of sight, the movement of the user's head, the ambient brightness, etc. One area information used for generation may be selected.

Modification 6:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Control part 20 ... Image display part 22 ... Right display drive part 24 ... Left display drive part 26 ... Right optical image display part 28 ... Left optical image display part 32 ... Right earphone 34 ... Left earphone 40 ... Connection part 51 ... Transmission Unit 52 ... Transmitter 53 ... Receiver 100 ... Head mounted display (head-mounted display device)
110: Input information acquisition unit (operation acquisition unit)
DESCRIPTION OF SYMBOLS 120 ... Memory | storage part 122 ... Area | region information 124 ... Identification image memory | storage part 130 ... Power supply 132 ... Wireless communication part (acquisition part)
140 ... CPU
142 ... generating unit (acquiring unit, generating unit)
144 ... Update unit (update unit)
DESCRIPTION OF SYMBOLS 160 ... Image processing part 170 ... Sound processing part 180 ... Interface 190 ... Display control part 201 ... Right backlight control part 202 ... Left backlight control part 221 ... Right backlight 222 ... Left backlight 251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate 300 ... Computer (external device)
400: Home appliance (external device)
500 ... Equipment (external device)
600 ... Mobile terminal (external device)
700 ... Game equipment (external device)
1000 ... Image display system Data ... Image data SC ... Outside view RE ... Right eye LE ... Left eye VI ... Virtual image VR ... Visual field J1-J6 ... Identification image B1-B6 ... Thumbnail display area (first display area)
G1 to G6 ... identification image BG ... active display area (second display area)
GI ... Integrated image INT ... Internet

Claims (10)

An image display device,
A first display area for displaying a list of a plurality of identification images for mutually identifying a plurality of external devices connected to the image display device; and one external device selected from the plurality of external devices A generating unit that generates an integrated image including a second display area that displays a display image that is an image displayed by
An image display unit for allowing a user of the image display device to visually recognize the integrated image;
An image display device comprising: The image display device according to claim 1, further comprising:
An acquisition unit for acquiring the display image from the one external device;
An update unit that updates the second display area of the integrated image using the acquired display image;
An image display device comprising: The image display device according to claim 1 or 2,
The identification image includes at least one of a display image at one time of the external device, an image representing information for identifying the external device, and an image representing a symbol assigned to the external device. An image display device including any one of them. The image display device according to any one of claims 1 to 3, further comprising:
An image display device comprising: an external device identifying unit that identifies an external device that is presumed to be noted by the user from among the plurality of external devices, and that uses the identified external device as the one external device. The image display device according to any one of claims 1 to 4, further comprising:
A distance acquisition unit that acquires distances between the image display device and the plurality of external devices,
The generation unit includes a display order of the plurality of identification images in the first display area, and a size of the identification image in the first display area according to the acquired distances. An image display device that determines at least one of them. The image display device according to any one of claims 1 to 5,
The generation unit is an image display device that updates the first display area. The image display device according to any one of claims 1 to 6,
The image display unit further includes a display driving unit that generates image light representing the integrated image, and the integrated image is visually recognized by the user by emitting the generated image light to the user's eyes. An image display device. The image display device according to claim 7,
The generation unit uses an aspect ratio of the second display area as an aspect ratio of the image light generating area by the display driving unit, and the first display area is arranged outside the second display area. Generating the integrated image;
The image display unit
When there is no operation by the user, the display driving unit is caused to generate the image light corresponding to the second display area of the integrated image,
When there is an operation by the user, the image display device causes the display driving unit to generate at least a portion of the image light corresponding to the first display area of the integrated image. The image display device according to claim 7 or 8, further comprising:
An additional presentation information generating unit configured to generate additional presentation information for providing augmented reality to the user when it is estimated that the user has paid attention to the one external device; ,
The image display unit causes the display driving unit to generate image light representing the generated additional presentation information, and emits the generated image light to the eyes of the user to perform the additional presentation. An image display device that allows the user to visually recognize information. A method for controlling an image display device, comprising:
(A) a first display area for displaying a list of a plurality of identification images for mutually identifying a plurality of external devices connected to the image display device, and one selected from the plurality of external devices Generating an integrated image including a second display area that displays a display image that is an image displayed by the external device;
(B) allowing the user of the image display device to visually recognize the integrated image;
A method comprising:

Images