JP2018085595A - Head-mounted display device and method for controlling the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology that can appropriately adjust an amount of data to be processed depending on various scenes in a head-mounted display device.SOLUTION: A head-mounted display device comprises: an image display part that enables a user to visually recognize a virtual image; an image display control part that controls operation of the image display part; an outside scene imaging camera that picks up an image of an outside scene; and an imaging processing part that performs image processing on data of a picked-up image picked-up by the outside scene imaging camera to create processed image data, the imaging processing part settable to either one of a first processing mode or a second processing mode of processing a larger amount of data per unit time than the first processing mode. The imaging processing part includes an object determination part that determines whether or not a predetermined object is present in an image indicated by the processed image data; when the object determination part determines the object to be present, the imaging processing part switches a processing mode from the first processing mode to the second processing mode.SELECTED DRAWING: Figure 11

Description

  The present invention relates to a head-mounted display device.

  A head-mounted display device that is mounted on a user's head and forms a virtual image on an image display unit positioned in front of the user's eyes is known (for example, Patent Document 1). In addition, a technique called augmented reality (AR) in which information is additionally presented using a computer in a real environment is known (for example, Patent Document 1).

JP 2003-296757 A

  In recent years, a head-mounted display device may include a camera that captures an outside scene. The captured image data captured by the camera is subjected to various processing in order to analyze it or display an image on an image display unit or an external display unit. Examples of various processes include resolution conversion processing, brightness adjustment (gamma correction), noise removal processing, 2D / 3D conversion processing, and removal of the background of an object to extract the object. And background removal processing.

  However, in various scenes, if the head-mounted display device processes captured image data under the same conditions, the data processing amount may be unnecessarily high. If the amount of data processing is unnecessarily high, there may be a problem that the processing time of the captured image data becomes long or the power consumption for processing the captured image data increases. Therefore, a technique that can appropriately adjust the data processing amount in accordance with various scenes is desired.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to one aspect of the present invention, a head-mounted display device is provided. The head-mounted display device includes an image display unit that allows a user to visually recognize a virtual image, an image display control unit that controls the operation of the image display unit, an outside scene imaging camera that captures an outside scene, and the outside scene imaging unit. An imaging processing unit that performs image processing on captured image data captured by a camera to generate processed image data, the first processing mode for processing the captured image data, and a unit time than the first processing mode A second processing mode with a large amount of data processing, and an imaging processing unit that can be set to any one of the second processing mode for processing the captured image data, the imaging processing unit, An object determining unit configured to determine whether or not a predetermined object exists in the image represented by the processed image data, and the imaging processing unit is determined by the object determining unit that the object exists; If Switches the processing mode from the first processing mode in the second processing mode. According to this aspect, since it is switched from the first processing mode to the second processing mode when it is determined that the object exists, the data processing amount can be appropriately adjusted depending on whether the object exists or not. .

(2) The head-mounted display device according to the above aspect, further detecting the presence of an object that is a candidate for the target in a certain target detection area in the imaging area of the outside scene imaging camera. The imaging processing unit may execute the first processing mode when the object detecting unit detects the presence of the object. According to this aspect, the first processing mode is executed when the object detection unit detects the presence of an object, thereby reducing the possibility of unnecessary imaging of the outside scene by the outside scene imaging camera. . Thereby, the data processing load by the imaging processing unit can be reduced.

(3) In the head-mounted display device according to the above aspect, the certain object detection area is within a range where the user can visually recognize the object, and the user is level in the depth direction. It may be within the reach of the user's hand when reaching out. According to this aspect, the object can be easily confirmed by the user because the object detection area is within a range where the object can be visually recognized and within a reachable range.

(4) In the head-mounted display device of the above aspect, the range in which the object can be visually recognized is a range in which one side is 60 degrees or less with respect to the user's line-of-sight direction in the horizontal direction. The side may be a range of 60 degrees or less, and in the vertical direction, the upper side of the line-of-sight direction may be a range of 30 degrees or less, and the lower side of the line-of-sight direction may be a range of 40 degrees or less. According to this form, the user can confirm the object more easily.

(5) In the head-mounted display device according to the above aspect, the movement of the user's head is further detected, and the movement of the user's head is detected based on the movement of the head detected over time. A head movement detection unit that calculates a movement amount; a gaze direction detection unit that detects a gaze direction of the user and calculates a change amount of the gaze direction of the user based on the gaze direction detected over time; When the processing mode is the first processing mode, the imaging processing unit, when the amount of motion exceeds a first threshold and the amount of change exceeds a second threshold, The processing mode may be maintained in the first processing mode regardless of the determination result of the object determination unit. According to this aspect, the first processing mode can be maintained when the amount of motion exceeds the first threshold and the amount of change exceeds the second threshold.

(6) In the head-mounted display device according to the above aspect, the imaging processing unit configures the first frame in at least one of the first processing mode and the second processing mode. A fixed pixel region including a pixel region where the object is located in the processed image data is set, and the fixed pixel is set with respect to the captured image data constituting the second frame imaged next to the first frame. Data in a range corresponding to the region may be extracted, and image processing may be performed on the extracted data. According to this aspect, since it is not necessary to perform image processing on data in a region that does not include an object, the load of data processing by the imaging processing unit can be further reduced.

(7) In the head-mounted display device according to the above aspect, the imaging processing unit may further include the object based on the captured image data of a plurality of frames captured over time in the second processing mode. An object motion detector that detects the movement of the object, and the image display controller may change the display content of the virtual image based on the motion of the object detected by the object motion detector. According to this aspect, when detecting the movement of the object, since the second processing mode with a large amount of data processing per unit time is executed, the movement of the object can be accurately detected, and the object is detected. The display content of the virtual image based on the movement of the image can be accurately changed.

(8) In the head mounted display device according to the above aspect, the object may be a user's hand. According to this form, the display content of the virtual image can be changed based on the movement of the user's hand.

(9) In the head mounted display device according to the above aspect, the first processing mode includes a first imaging mode in which the outside scene is captured at a first frame rate by the outside scene imaging camera, and the second processing is performed. The mode may include a second imaging mode in which the outside scene is imaged at a second frame rate higher than the first frame rate by the outside scene imaging camera. According to this aspect, it is possible to set one of the first processing mode and the second processing mode by changing the frame rate of the outside scene imaging camera.

(10) In the head-mounted display device according to the above aspect, the first processing mode includes a first resolution mode in which the outside scene is captured at a first resolution by the outside scene imaging camera, and the second processing mode May include a second resolution mode in which the outside scene is imaged at a second resolution higher than the first resolution by the outside scene imaging camera. According to this aspect, it is possible to set one of the first processing mode and the second processing mode by changing the resolution of the outside scene imaging camera.

(11) In the head-mounted display device according to the above aspect, in the first processing mode, a part of the plurality of frames constituting the captured image data captured by the outside scene imaging camera is thinned out. The second processing mode including a mode for performing image processing on the captured image data after thinning out may include a mode for performing image processing on the captured image data configured with more frames than in the first processing mode. . According to this aspect, it is possible to set one of the first processing mode and the second processing mode by changing the number of frames of captured image data to be subjected to image processing.

(12) According to another aspect of the present invention, a head-mounted display device is provided. The head-mounted display device includes an image display unit that allows a user to visually recognize a virtual image, an image display control unit that controls an operation of the image display unit, and an outside scene imaging unit that captures an outside scene and generates captured image data. A camera, an imaging processing unit that controls the operation of the outside scene imaging camera, an object detection unit for detecting the presence of an object in a certain target detection area among the imaging areas of the outside scene imaging camera, The imaging processing unit may start imaging the outside scene by the outside scene imaging camera when the object detecting unit detects that the object is present. According to this aspect, when the presence of an object is detected, the outside scene imaging camera starts imaging of the outside scene, so that the data processing amount depends on whether the object exists in the target object detection area or not. Can be adjusted appropriately.

(13) In the head-mounted display device according to the above aspect, the certain target object detection area is within a range where the user can visually recognize the object, and the user is level in the depth direction. It may be within the reach of the user's hand when reaching out. According to this aspect, the object can be easily confirmed by the user because the object detection area is within a range where the object can be visually recognized and within a reachable range.

(14) In the head-mounted display device according to the above aspect, the range in which the object can be visually recognized is a range in which one side is 60 degrees or less with respect to the user's line-of-sight direction in the horizontal direction. The side may be a range of 60 degrees or less, and in the vertical direction, the upper side of the line-of-sight direction may be a range of 30 degrees or less, and the lower side of the line-of-sight direction may be a range of 40 degrees or less. According to this form, the user can confirm the object more easily.

(15) In the head-mounted display device according to the above aspect, the imaging processing unit is configured to perform imaging based on a plurality of frames of captured image data captured over time when the outside scene is captured. An object motion detection unit for detecting the motion of the object, wherein the image display control unit may change the display content of the virtual image based on the motion of the object detected by the object motion detection unit. Good. According to this form, the display content of the virtual image based on the movement of the object can be changed with high accuracy.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with a new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  The present invention can be realized in various modes. For example, a head-mounted display device, a head-mounted display device control method, an image display system including a head-mounted display device, and the like. The present invention can be realized in the form of a computer program for realizing the functions of the apparatus, method, and system, a server apparatus for distributing the computer program, a storage medium storing the computer program, and the like.

It is explanatory drawing which shows schematic structure of the head mounted display apparatus in 1st Embodiment. It is a figure which shows the principal part structure of the image display part seen from the user. It is a block diagram which shows the structure of HMD functionally. It is a block diagram which shows the structure of CPU functionally. It is a 1st figure for demonstrating a target object detection area | region. It is a 2nd figure for demonstrating a target object detection area | region. It is a figure for demonstrating a memory | storage part. It is a figure for demonstrating a content corresponding table. It is a 1st figure which shows an example of the visual field visually recognized by the user. It is a 2nd figure which shows an example of the visual field visually recognized by the user. It is the processing flow which CPU performs. It is another example of the processing flow which CPU performs. It is explanatory drawing which shows schematic structure of a display system provided with HMD. It is a figure for demonstrating one scene where a display system is used. It is a 1st figure which shows an example of the visual field visually recognized by the user. It is a 2nd figure which shows an example of the visual field visually recognized by the user. It is a figure which shows one scene in which another processing flow is performed. It is another processing flow which CPU performs. It is a figure for demonstrating a cutting-out process. It is explanatory drawing which shows the structure of the external appearance of HMD in a 3rd modification. It is explanatory drawing which shows the structure of the external appearance of HMD in a 4th modification. It is a block diagram which shows functionally the structure of HMD in a 6th modification.

A. First embodiment:
FIG. 1 is an explanatory diagram showing a schematic configuration of a head-mounted display device (HMD) 100 according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating a main configuration of the image display unit 20 as viewed from the user Ps. FIG. 3 is a block diagram functionally showing the configuration of the HMD 100. FIG. 4 is a block diagram functionally showing the configuration of the CPU 140.

  HMD100 (FIG. 1) is provided with the image display part 20 which makes the user Ps visually recognize a virtual image in the state with which the user Ps was mounted | worn, and the control part (controller) 10 which controls the image display part 20. Yes.

  The image display unit 20 is a mounting body that is worn on the head of the user Ps, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, and a left optical image display unit 28. It is out. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user Ps when the user Ps wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user Ps when the user Ps wears the image display unit 20.

  The right holding unit 21 extends from the end ER, which is the other end of the right optical image display unit 26, to a position corresponding to the temporal region of the user Ps when the user Ps wears the image display unit 20. It is a member provided. Similarly, the left holding unit 23 extends from the end EL that is the other end of the left optical image display unit 28 to a position corresponding to the temporal region of the user Ps when the user Ps wears the image display unit 20. It is a member provided by being stretched. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the head of the user Ps like a temple of glasses.

  The right display driving unit 22 is disposed inside the right holding unit 21, in other words, on the side facing the user Ps' head when the user Ps wears the image display unit 20. Further, the left display driving unit 24 is disposed inside the left holding unit 23. Hereinafter, the right holding unit 21 and the left holding unit 23 will be described as “holding units” without being distinguished from each other. Similarly, the right display drive unit 22 and the left display drive unit 24 are described as “display drive units” without being distinguished from each other, and the right optical image display unit 26 and the left optical image display unit 28 are not distinguished from each other as “optical image display units”. Will be described.

  The display driving unit includes a liquid crystal display (hereinafter referred to as “LCD”) 241, 242, projection optical systems 251, 252, and the like (see FIG. 3). Details of the configuration of the display driving unit will be described later. The optical image display unit as an optical member includes light guide plates 261 and 262 (see FIG. 3) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving unit to the eyes of the user Ps. The light control plate is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20 (the side opposite to the eye side of the user Ps). The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. In addition, by adjusting the light transmittance of the light control plate, it is possible to adjust the external light quantity that enters the eyes of the user Ps and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The image display unit 20 (FIG. 1) further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48 connected to the control unit 10, a right cord 42 and a left cord 44 in which the main body cord 48 branches into two, and a connecting member 46 provided at the branch point. . The right cord 42 is connected to the right display driving unit 22, and the left cord 44 is connected to the left display driving unit 24. The connecting member 46 is provided with a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that fits to each other is provided on each end of the main body cord 48 on the opposite side of the connecting member 46 and the control section 10. The connector of the main body cord 48 and the connector of the control section 10 are provided. The control unit 10 and the image display unit 20 are connected to or disconnected from each other by the fitting / releasing. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the HMD 100. The control unit 10 includes a lighting unit 12, a touch pad 14, a cross key 16, and a power switch 18. The lighting unit 12 notifies the operation state of the HMD 100 (for example, ON / OFF of the power supply) by its light emission mode. For example, an LED (Light Emitting Diode) can be used as the lighting unit 12. The touch pad 14 detects a contact operation on the operation surface of the touch pad 14 and outputs a signal corresponding to the detected content. As the touch pad 14, various touch pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The cross key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power state of the HMD 100 by detecting a slide operation of the switch.

  The control unit 10 (FIG. 3) includes an input information acquisition unit 110, a storage unit 120, a power source 130, a wireless communication unit 132, a GPS module 134, a CPU 140, an interface 180, a transmission unit (Tx) 51, 52, and each part is connected to each other by a bus (not shown).

  The input information acquisition unit 110 acquires a signal corresponding to an operation input to the touch pad 14, the cross key 16, the power switch 18, and the like, for example. The storage unit 120 includes a ROM, a RAM, a DRAM, a hard disk, and the like.

  The power supply 130 supplies power to each part of the HMD 100. As the power source 130, for example, a secondary battery such as a lithium polymer battery or a lithium ion battery can be used. Further, instead of the secondary battery, a primary battery or a fuel cell may be used, or the wireless battery may be operated by receiving wireless power feeding. Furthermore, you may make it receive electric power supply from a solar cell and a capacitor. The wireless communication unit 132 performs wireless communication with other devices in accordance with a predetermined wireless communication standard such as a wireless LAN, Bluetooth (registered trademark), or iBeacon (registered trademark). The GPS module 134 detects its current position by receiving a signal from a GPS satellite.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer PC, a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 26 as a right optical image display unit 26, a left light guide plate 28 as a left optical image display unit 28, and an outside scene imaging. Camera 61 and 9-axis sensor 66. The image display unit 20 includes a first inner camera 62, a second inner camera 64, and a distance detection sensor 68.

  The outside scene imaging camera 61 (FIG. 1) is disposed at a position corresponding to the eyebrow of the user Ps when the user Ps wears the image display unit 20. Therefore, the outside scene imaging camera 61 captures an outside scene that is an external scene in the direction in which the user Ps is facing in a state where the user Ps wears the image display unit 20 on the head. The outside scene imaging camera 61 is a monocular camera. The outside scene imaging camera 61 is a digital camera including an imaging element such as a CCD or CMOS, an imaging lens, and the like. In the present embodiment, the outside scene imaging camera 61 has a telephoto function and can perform telephoto shooting. The telephoto function may be based on either optical zoom or digital zoom. Furthermore, the external scene imaging camera 61 may be an external camera such as a USB connection.

  The first inner camera 62 (FIG. 2) and the second inner camera 64 (FIG. 2) are digital cameras including an imaging element such as a CCD or CMOS, an imaging lens, and the like, similar to the outside scene imaging camera 61. The first inner camera 62 and the second inner camera 64 image the inner direction of the HMD 100, in other words, the direction facing the user Ps when the image display unit 20 is worn. The first inner camera 62 is used to image the right eye RE of the user Ps, and the second inner camera 64 is used to image the left eye LE of the user Ps. The width of the angle of view of the first inner camera 62 is preferably set within a range where the entire right eye RE of the user Ps can be imaged. Moreover, it is preferable that the angle of view of the second inner camera 64 is set to a range in which the entire left eye LE of the user Ps can be imaged. The first inner camera 62 and the second inner camera 64 execute imaging in accordance with the control of the control unit 10 (FIG. 3), and output the obtained captured image data to the control unit 10, as with the outside scene imaging camera 61. .

  In the state shown in FIG. 2, the back sides of the right light guide plate 26 and the left light guide plate 28 can be visually recognized, and the half mirror 261 for irradiating the right eye RE with the image light and the left eye LE for irradiating the image light. The half mirror 281 can be visually recognized as a substantially rectangular area. The user Ps views the outside scene through the entire left and right light guide plates 26 and 28 including the half mirrors 261 and 281, and visually recognizes a rectangular display image at the position of the half mirrors 261 and 281.

  Two distance detection sensors 68 (FIG. 1) are provided near the center of the front frame 27. The distance detection sensor 68 is used to detect a distance to a measurement object located in a preset measurement region. The measurement area can be a front area of the HMD 100 (an area overlapping with the imaging area of the outside scene imaging camera 61). The distance detection sensor 68 is, for example, a laser distance sensor, and includes a light emitting unit that emits light in the infrared region (infrared rays), and a light receiving unit that receives reflected light that is reflected from the light to be measured by the light source. Can be configured. In this case, the distance is obtained by triangular distance measurement processing or distance measurement processing based on a time difference. The distance detection sensor 68 is, for example, an ultrasonic sensor, and may be configured by a transmission unit that emits ultrasonic waves and a reception unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance is obtained by distance measurement processing based on the time difference. The distance detection sensor 68 outputs the detection result to the control unit 10.

  The 9-axis sensor 66 (FIG. 3) is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). In this embodiment, the position corresponds to the temple on the right side of the user Ps. Is arranged. Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is attached to the head of the user Ps, the CPU 140 detects the movement of the head of the user Ps. Used. The direction of the image display unit 20 is specified from the detected movement of the head of the user Ps.

  The right display driving unit 22 (FIG. 3) includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, and a right LCD control that functions as a display element. The unit 211 and the right LCD 241 and the right projection optical system 251 are included. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are also collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal, the vertical synchronization signal, the horizontal synchronization signal, and the right eye image data input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby converting the illumination light emitted from the right backlight 221 into an image. Modulate into effective image light to represent.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 26 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user Ps while reflecting the image light along a predetermined optical path. The optical image display unit can use any method as long as a virtual image is formed in front of the user Ps using image light. For example, a diffraction grating may be used, or a transflective film may be used. Also good. Note that the emission of image light by the HMD 100 is also referred to as “displaying an image (or virtual image)” in this specification.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. That is, the left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, and a left LCD control unit 212 that functions as a display element. And a left LCD 242 and a left projection optical system 252. Similar to the right LCD 241, the left LCD 242 is irradiated from the left backlight 222 by changing the transmittance of light transmitted through the left LCD 242 by driving the liquid crystal at each pixel position arranged in a matrix. The illumination light is modulated into effective image light representing the image. In this embodiment, the backlight method is adopted, but image light may be emitted using a front light method or a reflection method.

  The CPU 140 (FIG. 4) reads out and executes the computer program stored in the storage unit 120, thereby operating the operating system (OS) 150, the image processing unit 160, the image display control unit 162, the imaging processing unit 164, and the audio processing. Unit 170, object detection unit 172, head movement detection unit 173, and line-of-sight direction detection unit 176.

  The image processing unit 160 generates a signal based on content (video) input via the interface 180 or the wireless communication unit 132. Then, the image processing unit 160 controls the image display unit 20 by supplying the generated signal to the image display unit 20 via the connection unit 40. The signal supplied to the image display unit 20 differs between the analog format and the digital format. In the case of the analog format, the image processing unit 160 generates and transmits a clock signal PCLK, a vertical synchronization signal VSync, a horizontal synchronization signal HSync, and image data Data. Specifically, the image processing unit 160 acquires an image signal included in the content. For example, in the case of a moving image, the acquired image signal is an analog signal generally composed of 30 frame images per second. The image processing unit 160 separates a synchronization signal such as a vertical synchronization signal VSync and a horizontal synchronization signal HSync from the acquired image signal, and generates a clock signal PCLK by a PLL circuit or the like according to the period. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like. The image processing unit 160 stores the converted digital image signal as image data Data of RGB data in the DRAM in the storage unit 120 for each frame.

  On the other hand, in the case of the digital format, the image processing unit 160 generates and transmits the clock signal PCLK and the image data Data. Specifically, when the content is in digital format, the clock signal PCLK is output in synchronization with the image signal, so that the generation of the vertical synchronization signal VSync and the horizontal synchronization signal HSync and the A / D conversion of the analog image signal are performed. It becomes unnecessary. The image processing unit 160 performs image processing such as resolution conversion processing, various tone correction processing such as adjustment of luminance and saturation, and keystone correction processing on the image data Data stored in the storage unit 120. May be executed.

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data Data1”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data Data2”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  The image display control unit 162 controls the operation of the image display unit 20. The image display control unit 162 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the image display control unit 162 uses the control signal to drive ON / OFF of the right LCD 241 by the right LCD control unit 211, drive ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD. By individually controlling the ON / OFF of the left LCD 242 by the control unit 212 and the ON / OFF of the left backlight 222 by the left backlight control unit 202, the right display driving unit 22 and the left display driving unit 24 are controlled. Each controls the generation and emission of image light. The image display control unit 162 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. Similarly, the image display control unit 162 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  Further, the image display control unit 162 changes the display content of the virtual image (image) based on a predetermined movement of the object in the processed image data generated by the imaging processing unit 164 described later. The target object may be an object serving as a trigger for changing the display content, and examples thereof include the hand of the user Ps and a pen. A specific example of the display content change executed by the image display control unit 162 will be described later.

  The imaging processing unit 164 performs image processing on the captured image data captured by the outside scene imaging camera 61 to generate processed image data. Specifically, the imaging processing unit 164 includes display image processing for displaying captured image data as a virtual image, and object extraction processing for extracting an object from the captured image data. Examples of display image processing include gamma correction, luminance correction, contrast correction, noise removal, and contour enhancement. Examples of the object extraction process include a process for removing the background of the object and a process for extracting the outline of the object.

  The imaging processing unit 164 has a first processing mode M1 for processing captured image data and a second processing mode M2 for processing captured image data, and the first processing mode M1 has a data capacity to process. It can be set to any one of different second processing modes M2. The second processing mode M2 has a larger data processing amount per unit time than the first processing mode M1. In the present embodiment, the first processing mode M1 is a mode (first imaging mode) in which an outside scene is captured by the outside scene imaging camera 61 at the first frame rate. In the present embodiment, the second processing mode M2 is a mode (second imaging mode) in which the outside scene imaging camera 61 images the outside scene at a second frame rate higher than the first frame rate. The first frame rate is a frame rate that allows an object determination unit 166 described later to determine that data representing an object present in the processed image data is object data representing a predetermined object. Good. For example, the first frame rate is set to any value in the range of 1 fps to 10 fps. The second frame rate may be a frame rate that can track the movement of the object based on the processed image data for each of a plurality of frames arranged with time. For example, the second frame rate is set to a value of 30 fps or more. The second frame rate may be 120 fps or less in order to reduce the possibility that the load on the CPU 140 becomes excessive.

  The imaging processing unit 164 includes an object determination unit 166. The object determination unit 166 determines whether or not a predetermined object exists in the image represented by the processed image data. The object discriminating unit 166 compares the processed image data on which the object extraction process has been performed with the object data 129 described later by pattern matching or a statistical identification method, so that the image represented by the processed image data is included. It is determined whether or not an object exists. That is, when the target object is present in the image represented by the processed image data, it is determined that the target object is present in the imaging region captured by the outside scene imaging camera 61, and the target object is not present in the processed image data. Is determined that there is no object in the imaging area imaged by the outside scene imaging camera 61. When the imaging processing unit 164 is executing the first processing mode M1, when the target determination unit 166 determines that the target is present in the imaging region, the imaging processing unit 164 changes the processing mode from the first processing mode M1. The operation is switched to the second processing mode M2.

  The imaging processing unit 164 includes an object motion detection unit 178. The target object motion detection unit 178 detects the motion of the target object based on the captured image data of a plurality of frames captured over time by the outside scene image capturing camera 61. Specifically, processed image data obtained by extracting an object from captured image data of a plurality of frames is generated, and the movement of the object is detected by a change in the position of the object in the processed image data.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and transmits the acquired audio signal to a speaker (not shown) in the right earphone 32 and a speaker (not shown) in the left earphone 34 connected to the connecting member 46. To supply. For example, when a Dolby (registered trademark) system is adopted, processing is performed on an audio signal, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, respectively.

  The object detection unit 172 obtains detection data (distance information) from the distance detection sensor 68, and based on the detection result, the object detection unit 172 detects object candidates in a certain object detection area in the imaging area of the outside scene imaging camera 61. The presence of an object is detected. Specifically, when the distance information of the object is located within the object detection area, it is detected that the object is located in the object detection area. The object detection area may be the same area as the imaging area, or may be an area inside the imaging area and narrower than the imaging area. The object detection area is preferably set in a range in which the color can be discriminated in the actual visual field that the user Ps visually recognizes through the right light guide plate 26 and the left light guide plate 28. Details of the object detection area will be described later.

  The head movement detection unit 173 detects the movement of the head of the user Ps. Specifically, the movement of the head of the user Ps is detected based on the detection data acquired from the 9-axis sensor 66. Further, the head movement detection unit 173 moves the amount of movement of the head of the user Ps based on the movement of the head of the user Ps detected over time (for example, the amount of movement of the moving unit that has moved for a certain time). Is calculated. The amount of movement may be, for example, the sum of the rotation angles that have changed over a certain period of time, the amount of movement in the horizontal direction and the vertical direction that has changed over a period of time, or the sum of the rotation angles and the amount of movement. May be used in combination.

  The gaze direction detection unit 176 detects the gaze direction of the user Ps. Specifically, from the image of the right eye RE of the user Ps imaged by the first inner camera 62 and the image of the left eye LE of the user Ps imaged by the second inner camera 64, both eyes RE, LE Detect the direction of black eyes. Then, the gaze direction detection unit 176 estimates the gaze direction of the user Ps from the detected direction of the black eye. It can be said that the line-of-sight direction is a direction from an intermediate point between the right eye and the left eye toward the gazing point that the user Ps is gazing at. Further, the line-of-sight direction detection unit 176 calculates the amount of change in the line-of-sight direction of the user Ps based on the line-of-sight direction estimated over time (for example, the amount of change in the line-of-sight direction that has changed over time). The amount of change may be, for example, the total amount of angle change in the line-of-sight direction over a fixed time.

FIG. 5 is a first diagram for explaining the object detection area. FIG. 6 is a second diagram for explaining the object detection area. A preferable range of the object detection area will be described with reference to FIGS. 5 and 6. The target object detection area is preferably set in the ranges Rx and Rz where the user Ps can visually recognize the object. The ranges Rx and Rz in which the object can be visually recognized may be set in any of the following ranges (a) to (d).
(A) Range in which color can be distinguished:
Ranges Rxa and Rza that can discriminate colors when the color and brightness change.
(B) Guidance field range:
Ranges Rxb and Rzb that cause a coordinate system induction effect by visual information and cause realism.
(C) Range of stable focus field:
Ranges Rxc and Rzc that allow the user to gaze at the object comfortably with eyeball and head movements and to receive information effectively.
(D) Effective visual field range:
Ranges Rxd, Rzd where information can be received instantly only by eye movement.

  The horizontal range Rxa among the ranges in which the colors can be distinguished is a range of 60 degrees or less on one side with respect to the line-of-sight direction ED (straight line-of-sight direction ED) and the line-of-sight direction ED ( The other side is in a range of 60 degrees or less with respect to the straight line-of-sight direction (ED). That is, the target object detection area may have a horizontal angle of 120 degrees or less. Further, as shown in FIG. 6, the range Rza in the vertical direction among the ranges in which the color can be distinguished is a range of 30 degrees or less above the line-of-sight direction ED and 40 degrees or less below the line-of-sight direction ED. Range.

  The range Rxb in the horizontal direction in the range of the guidance visual field is a range of 50 degrees or less on one side with respect to the line-of-sight direction ED (straight line-of-sight direction ED), and the line-of-sight direction ED (straight) The other side is within a range of 50 degrees or less with respect to the line-of-sight direction ED). That is, the object detection area may be in a range where the horizontal angle is 100 degrees or less. Further, as shown in FIG. 6, the vertical range Rzb of the guidance visual field range is a range of 35 degrees or less above the line-of-sight direction ED and 50 degrees or less below the line-of-sight direction ED. It is a range.

  The horizontal range Rxc of the stable gazing range is a range of 45 degrees or less on one side with respect to the line-of-sight direction ED (straight line-of-sight direction ED), and the line-of-sight direction ED ( The other side is in a range of 45 degrees or less with respect to the straight line-of-sight direction (ED). That is, the object detection area may be in a range where the horizontal angle is 90 degrees or less. Further, as shown in FIG. 6, the range Rzc in the vertical direction of the range of the stable gaze field is a range of 30 degrees or less above the line-of-sight direction ED and 40 degrees or less below the line-of-sight direction ED. Range.

  Of the effective visual field range, the horizontal range Rxd is a range of 15 degrees or less on one side with respect to the line-of-sight direction ED (straight line-of-sight direction ED), and the line-of-sight direction ED (straight) The other side with respect to the line-of-sight direction ED) is within a range of 15 degrees or less. That is, the target object detection area may be in a range where the horizontal angle is 30 degrees or less. Further, as shown in FIG. 6, the vertical range Rzd of the effective visual field range is a range of 8 degrees or less above the line-of-sight direction ED and 12 degrees or less below the line-of-sight direction ED. It is a range.

  Further, the object detection area is a region from the intermediate point CE between the right eye RE and the left eye LE to the tip of the user Ps hand when the user Ps extends his hand horizontally in the depth direction. It is preferable to set the range Ry. The range Ry is within 1 meter, for example. That is, the target object detection area is within the range of angles (angles Rx, Rz) at which the user can visually recognize the object, and when the user Ps extends his hand horizontally in the depth direction, It is preferable that the value is within a range that can be reached. The user Ps can easily confirm the target object because the target object detection region is within a range where the user can visually recognize the object and within a range where the hand can reach. In particular, when the object is the hand of the user Ps, the depth direction of the object detection area is within the reach of the hand, so that the situation where objects other than the hand are located in the object detection area can be reduced. Therefore, the detection accuracy of the object can be improved.

  FIG. 7 is a diagram for explaining the storage unit 120. The storage unit 120 stores various data used for processing in the CPU 140. For example, the storage unit 120 stores content data 122, a content correspondence table 124, threshold data 126, and object data 129.

  The content data 122 includes content data (image data, video data, audio data, etc.) including images and videos displayed by the image display unit 20 under the control of the control unit 10. The content data 122 may include interactive content data. The interactive content is the operation of the user Ps acquired by the control unit 10, the control unit 10 executes processing according to the acquired operation content, and the content according to the processing content is stored in the image display unit 20. Means the type of content to display. In this case, the content data may include image data of a menu screen for acquiring the operation of the user Ps, data for defining a process corresponding to an item included in the menu screen, and the like.

  In the content correspondence table 124, the movement of the predetermined object is associated with the content displayed on the image display unit 20 in accordance with the movement of the predetermined object. In the present embodiment, the predetermined movement of the object is the movement of the hand of the user Ps, and specifically, a gesture change (change in the shape of the hand). Note that the predetermined movement of the object is not limited to the above, and any movement that serves as a trigger for changing the display content of the image display unit 20 may be used. For example, the movement of a predetermined object may be a movement without changing the shape of the hand of the user Ps (such as moving the hand from right to left) or a pen movement (pen speed or pen movement). Direction). The content correspondence table 124 and changes in the display content of the image display unit 20 executed with reference to the content correspondence table 124 will be described later.

  The threshold data 126 includes a first threshold 127 and a second threshold 128 for determining whether to change the mode of the imaging processing unit 164.

  The first threshold 127 is a reference value of the amount of movement of the head of the user Ps detected by the head movement detection unit 173. The first threshold value 127 is a reference value for determining whether or not the user Ps is intentionally moving the head. When the reference value is exceeded, the user Ps intentionally moves the head. It is preferable to set to a value that can be estimated. For example, the first threshold 127 may be set to 20 degrees or more when the rotation angle in a certain time (for example, 0.5 seconds) is used as an index. The first threshold 127 may be set to any value between 10 cm and 30 cm when the amount of movement of the head in a certain time (for example, 0.5 seconds) is used as an index.

  The second threshold 128 is a reference value for the amount of change in angle detected by the line-of-sight direction detection unit 176. The second threshold value 128 is a reference value for determining whether or not the line of sight has been moved from the target object that the user Ps is paying attention to. The second threshold value 128 is intended by the user Ps when the reference value is exceeded. It is preferable to set the value so that it is possible to estimate that the line-of-sight direction is changed. The second threshold 128 is the amount of change in the viewing direction angle for a fixed time (for example, 0.5 seconds). For example, the angle change amount may be set to a value of 20 degrees or more.

  The object data 129 is object data for discrimination by the object discrimination unit 166 (FIG. 4). The object data 129 may be, for example, data representing a human hand or data representing a pen. Further, the object data 129 may be appropriately changed depending on the user Ps or the like.

  FIG. 8 is a diagram for explaining the content correspondence table 124. FIG. 9 is a first diagram illustrating an example of a visual field VR visually recognized by the user Ps. FIG. 10 is a second diagram illustrating an example of the visual field VR visually recognized by the user Ps.

  The content correspondence table 124 (FIG. 8) defines a predetermined movement of the object and content to be displayed on the image display unit 20. That is, the content correspondence table 124 defines a predetermined movement of the object and display contents of the image display unit 20 that are changed according to the movement of the object. When the movement of the object is the first movement, a browser is displayed on the image display unit 20. The first movement is, for example, a change in the gesture of the user Ps, and is a movement that changes the hand from a goo state to a par state. When the movement of the object is the second movement, the telephone content (for example, an image representing the registrant name and the telephone number of the registrant name) is displayed on the image display unit 20. The second movement is, for example, a change in the gesture of the user Ps, and is a movement that changes the hand from a par state to a choke state. When the movement of the object is the third movement, the mail content is displayed on the image display unit 20. The third movement is, for example, a change in the gesture of the user Ps, and is a movement that changes the hand from a choke state to a par state. When the movement of the object is the fourth movement, the content displayed on the image display unit 20 is not displayed.

  As illustrated in FIG. 9, when the user Ps activates the HMD 100 and operates the touch pad 14 or the like so that the HMD 100 enters a specific mode (acceptance mode), the user Ps displays the optical image display units 26 and 28. And the menu image VI displayed on the image display unit 20 can be visually recognized. The menu image VI includes various selection images VIa to VId. When the browser selection image VIa is selected, the browser screen is displayed on the image display unit 20. When the telephone selection image VIb is selected, an image for making a telephone call is displayed on the image display unit 20. Further, when the mail selection image VIc is selected, a mail content image VIB (FIG. 10) is displayed on the image display unit 20. The end selection image VId is selected to hide the content image displayed on the image display unit 20. That is, the active content ends when the end selection image VId is selected. The user Ps may select various selection images VIa to VId by using the touch pad 14 or the cross key 16, or the various selection images VIa to VId by causing the imaging processing unit 164 to detect a change in gesture. The content corresponding to may be executed. For example, the user Ps moves his / her hand as defined in the content correspondence table 124 by placing his / her hand in the object detection area within the imaging area of the outside scene imaging camera 61.

  FIG. 11 is a processing flow executed by the CPU 140. The processing flow shown in FIG. 11 is triggered by the fact that the HMD 100 has entered a specific mode (acceptance mode). First, the object detection unit 172 detects whether or not an object exists in the target object detection area (step S12). If there is no object (step S12: No), step S12 is repeatedly executed. When it is detected that an object is present (step S12: Yes), the imaging processing unit 164 executes the first processing mode M1 (step S14). That is, the imaging processing unit 164 causes the outside scene imaging camera 61 to capture an outside scene at the first frame rate. The imaging processing unit 164 generates processed image data by performing image processing on the captured image data of each frame imaged at the first frame rate. Next, the object determination unit 166 determines whether or not the object represented by the predetermined object data stored in the storage unit 120 exists in the image represented by the processed image data (step S16). When it is determined that the object does not exist (step S16: No), step S16 is repeatedly executed on the generated processed image data.

  When it is determined that an object exists (step S16: Yes), threshold determination is performed (step S18). In the threshold determination, the imaging processing unit 164 determines whether the amount of head movement exceeds the first threshold 127 and whether the amount of change in the line-of-sight direction exceeds the second threshold 128. The end point of the fixed time used for this determination may be, for example, the time (shooting time) when the captured image data that is the basis of the processed image data determined to be present in step 16 is generated. Note that step S18 may be omitted.

  When at least one of the head movement amount exceeding the first threshold 127 and the line-of-sight direction change amount exceeding the second threshold 128 is not satisfied (step S18: No), imaging is performed. The processing unit 164 switches the processing mode from the first processing mode M1 to the second processing mode M2 (step S20). Thereby, the imaging processing unit 164 executes the second processing mode M2. That is, the imaging processing unit 164 causes the outside scene imaging camera 61 to capture an outside scene at the second frame rate. The imaging processing unit 164 generates processed image data by performing image processing on the captured image data of each frame captured at the second frame rate. On the other hand, if the amount of head movement exceeds the first threshold 127 and the amount of change in the line-of-sight direction exceeds the second threshold 128, the process returns to step S14. That is, when the processing mode is the first processing mode M1, if the amount of motion exceeds the first threshold 127 and the amount of change exceeds the second threshold 128, is it detected that an object is present? Regardless of whether or not, the processing mode is maintained at the first processing mode M1. Note that step S18 may be omitted.

  Next, the object motion detection unit 178 detects the motion of the object based on the captured image data captured at the second frame rate (step S22). Specifically, the imaging processing unit 164 generates processed image data by performing image processing on the captured image data of each frame imaged at the second frame rate in order to detect the motion of the object. Then, the object motion detection unit 178 detects the motion of the object based on the generated processed image data.

  Next, the imaging processing unit 164 determines whether there is content corresponding to the detected movement of the object (step S24). Specifically, the imaging processing unit 164 refers to the content correspondence table 124 (FIG. 8) stored in the storage unit 120 and determines whether there is content corresponding to the detected movement of the target object. If there is corresponding content, the content corresponding to the movement of the object (for example, mail content) is executed, and the image display control unit 162 displays the content screen on the image display unit 20 (step S25). . That is, the image display control unit 162 changes the display content of the virtual image based on the motion of the object detected by the object motion detection unit 178.

  According to the first embodiment, the imaging processing unit 164 switches from the first processing mode M1 to the second processing mode M2 when it is determined that an object is present (steps S14 to S20 in FIG. 11). The amount of data processing can be adjusted appropriately depending on whether or not the object is present. That is, the second processing mode M2 with a large amount of data processing is executed when detecting the movement of the object, and the first processing mode M1 with a small amount of data processing is executed when determining the presence of the object. By doing so, the data processing load by the imaging processing unit 164 can be reduced, and the power consumption of the HMD 100 can be reduced. Further, according to the first embodiment, it is possible to set one of the first processing mode M1 and the second processing mode M2 by changing the frame rate of the outside scene imaging camera 61.

  Further, according to the first embodiment, the first processing mode M1 is executed when the object detection unit 172 detects the presence of an object (steps S12 and S14 in FIG. 11). Accordingly, it is possible to reduce the possibility of unnecessarily capturing the outside scene with the outside scene imaging camera 61. Thereby, the data processing load by the imaging processing unit 164 can be further reduced.

  Further, according to the first embodiment, the imaging processing unit 164 determines the determination result of the object determination unit 166 when the amount of motion exceeds the first threshold 127 and the amount of change exceeds the second threshold 128. Regardless, the first processing mode is maintained (step S18 in FIG. 11: Yes). When the amount of movement exceeds the first threshold 127 and the amount of change exceeds the second threshold 128, it is assumed that the user Ps is not paying attention to the movement of the object. Therefore, in this case, it is not necessary to set the second processing mode M2, which is a processing mode for analyzing the motion of the object. Thereby, the data processing load by the imaging processing unit 164 can be further reduced.

  Further, according to the first embodiment, the object motion detection unit 178 detects the movement of the object based on the captured image data of a plurality of frames captured in the second processing mode M2, and the image display control unit 162 The display content of the virtual image is changed based on the detected movement of the object (step S20 to step S25 in FIG. 11). When detecting the movement of the object, the second processing mode M2 with a large amount of data processing per unit time is executed, so that the movement of the object can be detected with high accuracy and based on the movement of the object. The display content of the virtual image can be changed with high accuracy. Moreover, according to the said 1st Embodiment, the display content of a virtual image can be changed based on a motion of the user's Ps hand because a target object is the user's Ps hand. Thereby, the user Ps can operate the HMD 100 more intuitively.

  FIG. 12 is another example of the processing flow executed by the CPU 140. The CPU 140 may execute the processing flow shown in FIG. 12 instead of the processing flow shown in FIG. Here, steps having the same contents as those in FIG. 11 are denoted by the same reference numerals as those in FIG. The main difference between the processing flow shown in FIG. 12 and the processing flow shown in FIG. 11 is that the imaging processing unit 164 sets the second processing mode M2 when it is detected that an object is present in the object detection region. The point to be executed and the point that step S18 is omitted.

  First, the object detection unit 172 detects whether or not an object exists in the target object detection area (step S12). When there is no object (step S12: No), step S12 is repeatedly executed. When it is detected that an object is present (step S12: Yes), the imaging processing unit 164 executes the second processing mode (step S20a). That is, the imaging processing unit 164 causes the outside scene imaging camera 61 to capture an outside scene at the second frame rate. The imaging processing unit 164 generates processed image data by performing image processing on the captured image data of each frame captured at the second frame rate. As described above, step S20a is a process for starting imaging by activating the outside scene imaging camera 61 when an object is present in the object detection region. Therefore, when this processing flow is used, the imaging processing unit 164 may not have the first processing mode M1 (FIG. 4).

  Next, the object determination unit 166 determines whether or not the object represented by the predetermined object data stored in the storage unit 120 exists in the image represented by the processed image data (step S21). If it is determined that the object does not exist (step S21: No), step S21 is repeatedly executed on the generated processed image data.

  If it is determined that the object exists (step S21: Yes), the movement of the object is detected based on the captured image data of each frame imaged at the second frame rate (step S22). Next, the imaging processing unit 164 executes the processing from step S24 onward in the same manner as the processing flow shown in FIG.

  According to this processing flow, when the object detection unit 172 detects the presence of an object, the imaging processing unit 164 causes the outside scene imaging camera 61 to start imaging the outside scene. By doing so, the amount of data processing can be adjusted appropriately depending on whether or not an object is present in the object detection area. In other words, in a situation where there is no candidate object in the target detection area, there is a possibility that the data processing amount of the imaging processing unit 164 becomes unnecessarily excessive by not imaging the outside scene by the outside scene imaging camera 61. Can be reduced.

B. Second embodiment:
FIG. 13 is an explanatory diagram illustrating a schematic configuration of the display system 1 including the HMD 100. In the second embodiment, a usage example of the HMD 100 will be described using the display system 1 including the HMD 100 of the first embodiment and a personal computer (PC) 250 that performs communication of the HMD 100 via the server 60 as an example. Specifically, the display system 100 according to the second embodiment can be used, for example, as a remote support system that remotely instructs the user Ps wearing the HMD 100 in a factory or the like, such as work contents.

  The HMD 100 is connected to the Internet INT by wireless communication via a communication carrier. The PC 250 is connected to the Internet INT by wireless communication via the communication carrier BS. The server 60 is connected to the Internet INT by wired communication. As a result, the server 60, the HMD 100, and the PC 250 can perform data communication with each other via the Internet INT. The communication carrier BS includes a transmission / reception antenna, a radio base station, and an exchange station.

  FIG. 14 is a diagram for explaining one scene in which the display system 1 is used. The scene shown in FIG. 14 is a scene in which the user Ps of the HMD 100 is producing the product 602 in the factory. The product 602 is disposed on the work table 601. The user Ps performs assembly of the product 602, inspection of defective products, and the like.

  FIG. 15 is a first diagram illustrating an example of a visual field VR visually recognized by the user Ps. FIG. 16 is a second diagram illustrating an example of the visual field VR visually recognized by the user Ps. When the HMD 100 is activated and the HMD 100 enters a specific mode (acceptance mode) by operating the touch pad 14 or the like, the user Ps transmits the outside scene SC that has passed through the optical image display units 26 and 28, and the image display unit. The menu image VI1 displayed at 20 can be visually recognized (FIG. 15).

  Menu image VI1 includes help selection image VI2, call selection image VI3, and end selection image VI4. When the help selection image VI2 is selected, procedure contents such as assembly procedure contents and inspection procedure contents of the product 602 are displayed as an image VI5 (FIG. 16). The call selection image VI3 can be selected to make a call with a predetermined partner (in this embodiment, a remote work supporter who operates the PC 250), and an image related to the call (such as a call time image) is displayed as an image. Displayed on the unit 20. When the end selection image VI4 is selected, the active content (for example, the image VI4 of the procedure details) ends. The user Ps may select various selection images VI2 to VI4 by using the touch pad 14 or the cross key 16, or the various selection images VI2 to VI4 by causing the imaging processing unit 164 to detect a change in gesture. The content corresponding to may be executed. For example, the user Ps moves his / her hand as defined in the content correspondence table 124 (FIG. 8) by placing his / her hand in the object detection area within the imaging area of the outside scene imaging camera 61. Here, the content correspondence table 124 is changed to the content corresponding to the scene of the second embodiment. For this change, a plurality of content correspondence tables 124 corresponding to each scene may be stored in the storage unit 120, and the user Ps may select the content correspondence table 124 corresponding to the scene. In the case of the second embodiment, the content corresponding to the help selection image VI2 is executed by the first movement (image VI4 is displayed), the content corresponding to the call selection image VI3 is executed by the second movement, and the third The content corresponding to the end selection image VI4 is executed by the movement of.

  The processing flow executed by the CPU 140 can employ a processing flow similar to the processing flow shown in FIG. 11 of the first embodiment and the processing flow shown in FIG. 12 which is a modification of the first embodiment. Here, after the outside scene imaging camera 61 is activated, after step S14 in FIG. 11 and after step S20a in FIG. 12, the captured image data captured by the outside scene imaging camera 61 is subjected to image processing, and is sent to the PC 250. You may send it. In this way, the remote operation supporter on the PC 250 side can check the image captured by the outside scene imaging camera 61 via the display unit 258 of the PC 250. Further, when the user Ps moves a predetermined object in the object detection area, it is estimated that there is a high demand for receiving an instruction from the remote operation supporter for the product 602. In the processing flow shown in FIGS. 11 and 12, in the step 22 for detecting the movement of the object, the outside scene SC is imaged by the outside scene imaging camera 61 at the second frame rate having a high frame rate. Therefore, since the remote work support person can visually recognize the moving image captured at the second frame rate on the display unit 258, the remote work supporter can grasp the product 602 and the surrounding situation in more detail. Thereby, the remote operation supporter can give a more appropriate instruction to the user Ps.

  According to the said 2nd Embodiment, as long as it has the structure similar to the said 1st Embodiment, there exists an effect similar to 1st Embodiment. For example, the imaging processing unit 164 switches from the first processing mode M1 to the second processing mode M2 when it is determined that the object exists (step S14 to step S20 in FIG. 11), and thus the object exists. The amount of data processing can be adjusted appropriately depending on whether or not it exists. That is, the second processing mode M2 with a large amount of data processing is executed when detecting the movement of the object, and the first processing mode M1 with a small amount of data processing is executed when determining the presence of the object. By doing so, the data processing load by the imaging processing unit 164 can be reduced, and the power consumption of the HMD 100 can be reduced.

C. Variations of the second embodiment:
In the second embodiment, a processing flow similar to that in FIGS. 11 and 12 of the first embodiment can be employed, but other processing flows may be employed. FIG. 17 is a diagram showing one scene in which another processing flow is executed. The scene shown in FIG. 17 is a scene in which the user Ps repairs a product in which a problem has occurred. This is a scene in which an instruction is given to the user Ps. In FIG. 17, it is assumed that a defect occurs in the product 605 among the products 604 and 605. The user Ps or the remote operation supporter stores in advance information for identifying the product 605 (for example, shape data of the product 605) in the object data 129 of the storage unit 120.

  FIG. 18 is another processing flow executed by the CPU 140. The processing flow shown in FIG. 18 is executed in the scene shown in FIG. 17, for example. Like the other processing flows described below, in the processing flow shown in FIG. 11 and FIG. 12, the object detection unit 172 has a step of detecting whether or not an object exists in the target object detection area. (Step S12) may be omitted.

  The processing flow shown in FIG. 18 is triggered by the fact that the HMD 100 has entered a specific mode (acceptance mode). First, the imaging processing unit 164 activates the outside scene imaging camera 61 and executes the first processing mode M1 (step S40). The first processing mode M1 has the same contents as in the first embodiment, and is a mode in which an outside scene is imaged at the first frame rate by the outside scene imaging camera 61. The imaging processing unit 164 generates processed image data by performing image processing on the captured image data of each frame imaged at the first frame rate. Each frame imaged at the first frame rate is subjected to image processing and transmitted to the PC 250 (FIG. 13).

  Next, the object discriminating unit 166 discriminates whether or not the object (product 605) represented by the predetermined object data stored in the storage unit 120 exists in the image represented by the processed image data (Step 605). S42). If it is determined that the object does not exist (step S42: No), step S42 is repeatedly executed on the generated processed image data.

  When it is determined that the object exists (step S42: Yes), the imaging processing unit 164 switches the processing mode from the first processing mode M1 to the second processing mode M2 (step S44). Thereby, the imaging processing unit 164 executes the second processing mode M2. That is, the imaging processing unit 164 causes the outside scene imaging camera 61 to capture an outside scene at the second frame rate. The imaging processing unit 164 generates processed image data by performing image processing on the captured image data of each frame captured at the second frame rate. Each frame imaged at the second frame rate is subjected to image processing and transmitted to the PC 250 (FIG. 13). Note that step S18 shown in FIG. 11 may be executed between step S42 and step S44.

  According to the above change mode, when it is determined that the object is present, the first processing mode M1 is switched to the second processing mode M2, so that the data processing amount is appropriately set depending on whether the object exists or not. Can be adjusted. In addition, when there is an object, the captured image data captured at the second frame rate higher than the first frame rate is processed and transmitted to the PC 250 side. Can be observed with higher accuracy. Therefore, the remote operation supporter can give a more appropriate instruction to the user Ps.

D. Variations in processing modes:
In each of the above embodiments, the first processing mode M1 is a first imaging mode in which the outside scene imaging camera 61 captures images at the first frame rate, and the second processing mode M2 is performed by the outside scene imaging camera 61 more than the first frame rate. This was the second imaging mode in which an outside scene was imaged at a high second frame rate. However, if the data processing amount per unit time in the second processing mode M2 is larger than the data processing amount per unit time in the first processing mode M1, the contents for realizing each processing mode are limited to the above. It is not a thing. Hereinafter, modifications of the first processing mode M1 and the second processing mode M2 will be described.

  The first processing mode M1 may be a first resolution mode in which an outside scene is captured by the outside scene imaging camera 61 at a first resolution, and the second processing mode M2 is higher than the first resolution by the outside scene imaging camera 61. A second resolution mode may be used in which an outside scene is captured at the second resolution. The first resolution may be a resolution that allows the object determination unit 166 to determine whether data representing an object existing in the processed image data is object data representing a predetermined object.

  Further, in the first processing mode M1, after thinning out some frames of the captured image data of a plurality of frames captured by the outside scene imaging camera 61 at a predetermined frame rate (for example, the second frame rate), A mode (thinning-out mode) for image processing of the captured image data may be used. Further, the second processing mode M2 may be a mode (normal mode) in which captured image data composed of more frames than the first processing mode M1 is subjected to image processing. The number of frames for image processing per unit time is greater in the second processing mode M2 than in the first processing mode M1. In the second processing mode M2, for example, all the captured image data of a plurality of frames captured by the outside scene imaging camera 61 at a predetermined frame rate (for example, the second frame rate) is subjected to image processing. The first processing mode M1 may be executed by combining two or more of the first imaging mode, the first resolution mode, and the thinning mode. The second processing mode M2 may be executed by combining two or more of the second imaging mode, the second resolution mode, and the normal mode.

  In at least one of the first processing mode M1 and the second processing mode M2, the imaging processing unit 164 may perform a clipping process described below. FIG. 19 is a diagram for explaining the extraction process. FIG. 19 shows an example of the processed image data IMGD. When the object determination unit 166 determines the object hd (in this example, the hand hd) based on the processed image data IMGD constituting the first frame, the imaging processing unit 164 captures the pixel region IMGR2 where the object hd is located. A constant pixel region IMGR1 including the same is set. In the example illustrated in FIG. 19, the imaging processing unit 164 sets a constant pixel area IMGR1 that is larger than the pixel area IMGR2 where the object hd is located. In the present embodiment, the captured image data and the processed image data have the same number of pixels and the same fixed pixel region IMGR1. The imaging processing unit 164 extracts data in a range corresponding to the certain pixel region IMGR1 for the captured image data constituting the second frame imaged by the outside scene imaging camera 61 next to the first frame, Image processing is performed on the extracted data. In this example, the range corresponding to the fixed pixel region IMGR1 is the same region as the fixed pixel region IMGR1, and can be specified by the coordinates of the pixel. You may perform image processing with respect to the extracted data, for example, in order to detect the motion of the target object by the target object motion detection part 178, for example. Alternatively, the extracted data may be subjected to image processing and replaced with pixel data of the fixed pixel region IMGR1 in the processed image data IMGD one frame before. By performing image processing on the extracted data, it is not necessary to perform image processing on data in a region that does not include the object. Thereby, the data processing load by the imaging processing unit 164 can be further reduced.

E. Variations:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

E-1. First modification:
In the above embodiments, the object may be a mark image that is an image of a specific mark. Examples of the stamp image include a QR code (registered trademark) and a barcode. The imaging processing unit 164 may read the mark image and display a virtual image corresponding to the mark image on the image display unit 20. For example, it may be determined in the first processing mode M1 that the object is a mark image, and then the mark image may be read in the second processing mode M2.

E-2. Second modification:
In each of the above embodiments, the outside scene imaging camera 61 is a monocular camera, but may be a stereo camera. Further, the detection of the line-of-sight direction is performed based on the captured images of the first inner camera 62 and the second inner camera 64. However, the present invention is not limited to this, and may be performed using a myoelectric sensor, for example. Good.

E-3. Third modification:
FIG. 20 is an explanatory diagram showing an external configuration of the HMD in the third modification. In the example shown in FIG. 20, the image display unit 20 x includes a right optical image display unit 26 x instead of the right optical image display unit 26, and includes a left optical image display unit 28 x instead of the left optical image display unit 28. Yes. The right optical image display unit 26x and the left optical image display unit 28x are formed smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely above the right eye and the left eye of the user Ps when the HMD is worn. Yes. Although not shown in the figure, the image display unit 20x is similar to the first embodiment described above in the distance detection sensor 68, the outside scene imaging camera 61, the 9-axis sensor 66, the first inner camera 62, and the second inner camera 64. A configuration for implementing each of the above embodiments is provided.

E-4. Fourth modification:
FIG. 21 is an explanatory diagram showing the configuration of the appearance of the HMD in the fourth modification. In the case of the example shown in FIG. 21, the image display unit 20 y includes a right optical image display unit 26 y instead of the right optical image display unit 26, and includes a left optical image display unit 28 y instead of the left optical image display unit 28. Yes. The right optical image display unit 26y and the left optical image display unit 28y are formed to be smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely below the right eye and the left eye of the user Ps when the HMD is worn. Yes. Although not shown, the image display unit 20y has a distance detection sensor 68, an outside scene imaging camera 61, a nine-axis sensor 66, a first inner camera 62, and a second inner camera 64, as in the first embodiment. A configuration for implementing each of the above embodiments is provided. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the eyes of the user Ps. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the eyes of the user Ps, in other words, the optical image display unit covers the eyes of the user Ps. It can also be realized as an HMD that is not completely covered.

E-5. Fifth modification:
In the above embodiment, the so-called transmission type HMD 100 in which the right light guide plate 26 and the left light guide plate 28 transmit external light has been described. However, the present invention can also be applied to, for example, a so-called non-transmissive HMD 100 that displays an image in a state where an outside scene cannot be visually recognized. In addition, in these HMDs 100, in addition to the AR display that displays an image superimposed on the real space described in the above embodiment, an MR (Mixed Reality) display that displays a combination of the captured real space image and a virtual image, Or VR (Virtual Reality) display which displays virtual space can also be performed. The present invention can also be applied to an apparatus that does not perform any of AR, MR, and VR display.

E-6. Sixth modification:
The functional block diagram of the HMD 100 in each of the above embodiments has the configuration shown in FIG. 3, but is not limited to the above, and may be another functional block diagram. Hereinafter, the HMD 100a having functional blocks different from the respective embodiments will be described. FIG. 22 is a block diagram functionally showing the configuration of the HMD 100a. The same components as those of the HMD 100 of the above embodiment are denoted by the same reference numerals and description thereof is omitted as appropriate.

  The control device 10a includes a main processor 140a that executes a program to control the HMD 100a, a storage unit, an input / output unit, sensors, an interface, and a power supply unit 130a. The storage unit, input / output unit, sensors, interface, and power supply unit 130a are connected to the main processor 140a. The main processor 140a is mounted on a controller board 120a built in the control device 10a. The main processor 140a can execute the same processing as the CPU 140 of each of the above embodiments. For example, the main processor 140a includes at least the functional unit illustrated in FIG.

  The storage unit includes a memory 118 and a nonvolatile storage unit 121. The memory 118 constitutes a work area that temporarily stores a computer program executed by the main processor 140a and data to be processed. The nonvolatile storage unit 121 is configured by a flash memory or an eMMC (embedded Multi Media Card). The non-volatile storage unit 121 stores a computer program executed by the main processor 140a and various data processed by the main processor 140a. The nonvolatile storage unit 121 stores at least data stored in the storage unit 120 illustrated in FIG. In the present embodiment, these storage units are mounted on the controller board 120a.

  The input / output unit includes a track pad 14a and an operation unit 110a. The operation unit 110a includes, for example, a button 11, an LED indicator 12a, and a cross key 16. The main processor 140a controls these input / output units and acquires signals output from the input / output units.

  The sensors include a six-axis sensor 111, a magnetic sensor 113, and a GPS (Global Positioning System) 115. The 6-axis sensor 111 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 111 may employ an IMU (Inertial Measurement Unit) in which these sensors are modularized. The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor. The GPS receiving unit 115 includes a GPS antenna (not shown), receives a radio signal transmitted from a GPS satellite, and detects the coordinates of the current position of the control device 10a. These sensors (six-axis sensor 111, magnetic sensor 113, GPS 115) output detection values to the main processor 140a according to a sampling frequency designated in advance. The timing at which each sensor outputs a detection value may be in accordance with an instruction from the main processor 140a.

  The interface includes a wireless communication unit 117, an audio codec 181, an external connector 184, an external memory interface 186, a USB (Universal Serial Bus) connector 188, a sensor hub 192, an FPGA 194, and an interface 196. ing. These function as an interface with the outside. The wireless communication unit 117 performs wireless communication between the HMD 100a and an external device. The wireless communication unit 117 includes an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like (not shown), or is configured as a device in which these are integrated. The wireless communication unit 117 performs wireless communication based on standards such as a wireless LAN including Bluetooth (registered trademark) and Wi-Fi, for example.

  The audio codec 181 is connected to the audio interface 182, and encodes / decodes an audio signal input / output via the audio interface 182. The audio codec 181 may have a function of processing audio input / output via a headset including a microphone, the right earphone 32, and the left earphone 34. Furthermore, the audio codec 181 is an A / D converter that converts analog audio signals to digital audio data for audio signals that are input / output via the built-in headset, and D that performs the reverse conversion. / A converter may be provided. The A / D converter converts an analog audio signal input via a microphone (not shown) into digital audio data and outputs the digital audio data to the main processor 140a. The D / A converter converts the digital audio data output from the main processor 140a into an analog audio signal, and outputs the analog audio signal to the right earphone 32 and the left earphone 34 (FIG. 1). The audio interface 182 is an interface for inputting / outputting audio signals to / from the headset.

  The external connector 184 is a connector for connecting an external device (for example, a personal computer, a smartphone, a game machine, etc.) that communicates with the main processor 140a to the main processor 140a. The external device connected to the external connector 184 can be a content supply source, and can be used for debugging a computer program executed by the main processor 140a and collecting an operation log of the HMD 100a. The external connector 184 can employ various modes. As the external connector 184, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, a memory card interface, or an interface corresponding to a wireless connection such as a wireless LAN interface or a Bluetooth interface can be adopted.

  The external memory interface 186 is an interface to which a portable memory device can be connected. The external memory interface 186 includes, for example, a memory card slot in which a card-type recording medium is mounted and data is read and written, and an interface circuit. The size, shape, standard, etc. of the card type recording medium can be selected as appropriate. The USB connector 188 is an interface that can connect a memory device, a smartphone, a personal computer, or the like that conforms to the USB standard. The USB connector 188 includes, for example, a connector conforming to the USB standard and an interface circuit. The size, shape, USB standard version, etc. of the USB connector 188 can be selected as appropriate.

  In addition, the HMD 100 a includes a vibrator 19. The vibrator 19 includes a motor (not shown), an eccentric rotor, and the like, and utters vibrations under the control of the main processor 140a. The HMD 100a causes the vibrator 19 to generate vibration with a predetermined vibration pattern when, for example, an operation on the operation unit 110a is detected or when the power of the HMD 100a is turned on / off.

  The sensor hub 192 and the FPGA 194 are connected to the image display unit 20a via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20a and outputs them to the main processor 140a. The FPGA 194 executes processing of data transmitted and received between the main processor 140a and each unit of the image display unit 20a and transmission via the interface 196. The interface 196 is connected to each of the right display unit 22a and the left display unit 24a of the image display unit 20a. In the example of this embodiment, the connection cable 40 is connected to the left holding unit 23, the wiring connected to the connection cable 40 is laid inside the image display unit 20a, and each of the right display unit 22a and the left display unit 24a is It is connected to the interface 196 of the control device 10a.

  The power supply unit 130a supplies power for operating the control device 10a. The battery 136 is a rechargeable battery. The power control circuit 135 detects the remaining capacity of the battery 136 and controls the charging of the OS 143. The power supply control circuit 135 is connected to the main processor 140a, and outputs the detected value of the remaining capacity of the battery 136 and the detected value of the voltage of the battery 136 to the main processor 140a. Note that power may be supplied from the control device 10a to the image display unit 20a based on the power supplied by the power supply unit 130a. The power supply state from the power supply unit 130a to each unit of the control device 10a and the image display unit 20a may be configured to be controllable by the main processor 140a.

  The right display unit 22a includes a display unit substrate 210a, an OLED unit 221a, an outside scene imaging camera 61, a first inner camera 62, an illuminance sensor 65a, an LED indicator 67a, a temperature sensor 217a, and a distance detection sensor 68. And comprising. An interface (I / F) 211a connected to the interface 196, a receiving unit (Rx) 213a, and an EEPROM (Electrically Erasable Programmable Read-Only Memory) 215a are mounted on the display unit substrate 210a. The receiving unit 213a receives data input from the control device 10a via the interface 211a. When receiving the image data of the image displayed by the OLED unit 221a, the receiving unit 213a outputs the received image data to the OLED drive circuit.

  The EEPROM 215a stores various data in a manner that can be read by the main processor 140a. The EEPROM 215a stores, for example, data on the light emission characteristics and display characteristics of the OLED units 221a and 241a of the image display unit 20a, data on the sensor characteristics of the right display unit 22a or the left display unit 24a, and the like. Specifically, for example, parameters relating to gamma correction of the OLED units 221a and 241a, data for compensating detection values of temperature sensors 217a and 239a described later, and the like are stored. These data are generated by factory inspection of the HMD 100a and written in the EEPROM 215a. After shipment, the main processor 140a reads the data in the EEPROM 215a and uses it for various processes.

  The outside scene imaging camera 61 executes imaging in accordance with a signal input via the interface 211a, and outputs captured image data or a signal representing an imaging result to the control device 10a. The illuminance sensor 65a is disposed so as to receive external light from the front of the user wearing the image display unit 20a. The illuminance sensor 65a outputs a detection value corresponding to the amount of received light (received light intensity). The LED indicator 67a is disposed in the vicinity of the outside scene imaging camera 61. The LED indicator 67a lights during execution of imaging by the outside scene imaging camera 61 to notify that imaging is being performed.

  The temperature sensor 217a detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 217a is mounted on the back side of an OLED panel (not shown). For example, the temperature sensor 217a may be mounted on the same substrate as the OLED drive circuit. With this configuration, the temperature sensor 217a mainly detects the temperature of the OLED panel. The temperature sensor 217a may be built in the OLED panel or the OLED drive circuit. For example, when the OLED panel is mounted as an integrated circuit on an integrated semiconductor chip together with an OLED drive circuit as a Si-OLED, the temperature sensor 217a may be mounted on the semiconductor chip.

  The left display unit 24a includes a display unit substrate 230a, an OLED unit 241a, a temperature sensor 239a, and a second inner camera 64. An interface (I / F) 231a connected to the interface 196, a receiving unit (Rx) 233a, a 6-axis sensor 235a, and a magnetic sensor 237a are mounted on the display unit substrate 230a. The receiving unit 233a receives data input from the control device 10a via the interface 231a. When receiving the image data of the image displayed by the OLED unit 241a, the receiving unit 233a outputs the received image data to an OLED drive circuit (not shown).

  The 6-axis sensor 235a is a motion sensor (inertia sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235a may employ an IMU in which the above sensors are modularized. The magnetic sensor 237a is a triaxial geomagnetic sensor, for example. The temperature sensor 239a detects a temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 239a is mounted on the back side of an OLED panel (not shown). For example, the temperature sensor 239a may be mounted on the same substrate as the OLED drive circuit. With this configuration, the temperature sensor 239a mainly detects the temperature of the OLED panel. The temperature sensor 239a may be incorporated in the OLED panel or the OLED drive circuit. The details are the same as those of the temperature sensor 217a.

  The external view imaging camera 61, the illuminance sensor 65a, the temperature sensor 217a of the right display unit 22a, and the 6-axis sensor 235a, the magnetic sensor 237a, and the temperature sensor 239a of the left display unit 24a are connected to the sensor hub 192 of the control device 10a. . The sensor hub 192 sets and initializes the sampling period of each sensor according to the control of the main processor 140a. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling period of each sensor. The sensor hub 192 outputs detection values of the sensors included in the right display unit 22a and the left display unit 24a to the main processor 140a at a preset timing. The FPGA 194 may have a cache function that temporarily holds the detection value of each sensor. The sensor hub 192 may have a conversion function (for example, a conversion function to a unified format) of a signal format or a data format of a detection value of each sensor. The FPGA 194 turns on or off the LED indicator 67a by starting and stopping energization of the LED indicator 67a according to the control of the main processor 140a.

E-7. Seventh modification:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

    DESCRIPTION OF SYMBOLS 1 ... Display system, 10 ... Control part, 10a ... Control apparatus, 11 ... Button, 12 ... Lighting part, 12a ... LED indicator, 14 ... Touch pad, 14a ... Track pad, 16 ... Cross key, 18 ... Power switch, 19 ... Vibrators 20, 20a, 20x, 20y ... Image display part, 21 ... Right holding part, 22 ... Right display driving part, 22a ... Right display unit, 23 ... Left holding part, 24 ... Left display driving part, 24a ... Left Display unit, 26, 26x, 26y ... right optical image display unit, 27 ... front frame, 28, 28x, 28y ... left optical image display unit, 30 ... earphone plug, 32 ... right earphone, 34 ... left earphone, 40 ... Connection part (connection cable), 42 ... right cord, 44 ... left cord, 46 ... connecting member, 48 ... main body cord, 51 ... transmission part, 52 ... transmission part, 53 ... reception part, 6 DESCRIPTION OF SYMBOLS ... Server, 61 ... Camera for outside scene imaging, 62 ... First inner camera, 64 ... Second inner camera, 65a ... Illuminance sensor, 66 ... Nine-axis sensor, 67a ... LED indicator, 68 ... Distance detection sensor, 100 ... Display system 110 ... input information acquisition unit, 110a ... operation unit, 111 ... 6-axis sensor, 113 ... magnetic sensor, 115 ... GPS reception unit, 117 ... wireless communication unit, 118 ... memory, 120 ... storage unit, 120a ... controller board, 121 ... Non-volatile storage unit, 122 ... Content data, 124 ... Content correspondence table, 126 ... Threshold value data, 127 ... First threshold value, 128 ... Second threshold value, 129 ... Target data, 130 ... Power source, 130a ... Power source unit, 132: wireless communication unit, 134: GPS module, 135: power control circuit, 136: battery 140, CPU, 140a, main processor, 160, image processing unit, 162, image display control unit, 164, imaging processing unit, 166, object discrimination unit, 170, audio processing unit, 172, object detection unit, 173 ... Head movement detection unit, 176 ... Gaze direction detection unit, 178 ... Object movement detection unit, 180 ... Interface, 181 ... Audio codec, 182 ... Audio interface, 184 ... External connector, 186 ... External memory interface, 188 ... USB Connector, 192 ... Sensor hub, 196 ... Interface, 201 ... Right backlight controller, 202 ... Left backlight controller, 211 ... Right LCD controller, 212 ... Left LCD controller, 213a ... Receiver, 215a ... EEPROM, 217a ... temperature sensor, 221 ... right backlight, 2 21a ... OLED unit, 222 ... left backlight, 230a ... display unit board, 231a ... interface, 233a ... receiving unit, 235a ... 6-axis sensor, 237a ... magnetic sensor, 239a ... temperature sensor, 241a ... OLED unit, 250 ... PC 251 ... Right projection optical system, 252 ... Left projection optical system, 258 ... Display unit, 261,281 ... Half mirror, 601 ... Workbench, 602, 604,605 ... Product, ED ... Gaze direction, EL ... End, ER ... End, IMGD ... Processed image data, IMGR1 ... Constant pixel area, IMGR2 ... Pixel area, INT ... Internet, LE ... Left eye, Ps ... User, RE ... Right eye, SC ... Outside view, VR ... Field of view, hd ... hand (object)

Claims (17)

A head-mounted display device,
An image display unit for allowing a user to visually recognize a virtual image;
An image display control unit for controlling the operation of the image display unit;
An outside scene imaging camera for imaging the outside scene;
An image processing unit that performs image processing on captured image data captured by the outside scene image capturing camera to generate processed image data, the first processing mode for processing the captured image data, and the first processing mode A second processing mode with a large amount of data processing per unit time, the imaging processing unit being settable to any one of the second processing mode for processing the captured image data,
The imaging processing unit includes an object determination unit that determines whether or not a predetermined object exists in an image represented by the processed image data,
The head-mounted display device that switches the processing mode from the first processing mode to the second processing mode when the imaging processing unit determines that the target is present by the target determination unit. The head-mounted display device according to claim 1, further comprising:
An object detection unit for detecting the presence of an object that is a candidate for the object in a certain object detection area in the imaging area of the camera for capturing an outside scene;
The imaging processing unit is a head-mounted display device that executes the first processing mode when the object detection unit detects the presence of the object. The head-mounted display device according to claim 2,
The certain object detection area is within a range where the user can visually recognize the object, and within a range where the user can reach when the user extends his hand horizontally in the depth direction. A head-mounted display device. The head-mounted display device according to claim 3,
The range in which the object can be viewed is
In the horizontal direction, one side is a range of 60 degrees or less with respect to the user's line-of-sight direction, and the other side is a range of 60 degrees or less,
A head-mounted display device in which, in the vertical direction, the upper side of the line-of-sight direction is a range of 30 degrees or less and the lower side of the line-of-sight direction is a range of 40 degrees or less. The head-mounted display device according to any one of claims 1 to 4, further comprising:
A head movement detection unit that detects movement of the user's head and calculates a movement amount of the user's head based on the movement of the head detected over time;
A gaze direction detection unit that detects the gaze direction of the user and calculates a change amount of the gaze direction of the user based on the gaze direction detected over time,
The imaging processing unit
When the processing mode is the first processing mode, if the amount of motion exceeds the first threshold and the amount of change exceeds the second threshold, regardless of the determination result of the object determination unit A head-mounted display device that maintains the processing mode in the first processing mode. The head-mounted display device according to any one of claims 1 to 5,
The imaging processing unit includes a pixel region in which at least one of the first processing mode and the second processing mode includes a pixel region in which the target object in the processed image data constituting the first frame is located. A pixel area is set, data in a range corresponding to the certain pixel area is extracted from the captured image data constituting the second frame imaged next to the first frame, and the extracted data is A head-mounted display device that performs image processing. The head-mounted display device according to any one of claims 1 to 6,
The imaging processing unit further includes:
In the second processing mode, an object motion detection unit that detects the motion of the object based on the captured image data of a plurality of frames captured over time,
The image display control unit is a head-mounted display device that changes display contents of the virtual image based on the movement of the object detected by the object movement detection unit. The head-mounted display device according to claim 7,
The head mounted display device, wherein the object is a user's hand. The head-mounted display device according to any one of claims 1 to 8,
The first processing mode includes a first imaging mode in which the outside scene is imaged at a first frame rate by the outside scene imaging camera;
The head mounted display device, wherein the second processing mode includes a second imaging mode in which the outside scene is imaged at a second frame rate higher than the first frame rate by the outside scene imaging camera. The head-mounted display device according to any one of claims 1 to 9,
The first processing mode includes a first resolution mode in which the outside scene is imaged at a first resolution by the outside scene imaging camera;
The head-mounted display device, wherein the second processing mode includes a second resolution mode in which the outside scene is imaged at a second resolution higher than the first resolution by the outside scene imaging camera. The head-mounted display device according to any one of claims 1 to 10,
The first processing mode includes a mode in which a part of a plurality of frames constituting the captured image data captured by the outside scene imaging camera is thinned, and the captured image data after the thinning is subjected to image processing. The head-mounted display device, wherein the second processing mode includes a mode for performing image processing on the captured image data composed of more frames than in the first processing mode. A head-mounted display device,
An image display unit for allowing a user to visually recognize a virtual image;
An image display control unit for controlling the operation of the image display unit;
An outside scene imaging camera that images the outside scene and generates captured image data;
An imaging processor for controlling the operation of the outside scene imaging camera;
An object detection unit for detecting the presence of an object in a certain target detection region in the imaging region of the outside scene imaging camera;
The imaging processing unit
A head-mounted display device that, when the object detection unit detects the presence of the object, starts imaging the outside scene with the outside scene imaging camera. The head-mounted display device according to claim 12,
The certain object detection area is within a range where the user can visually recognize the object, and within a range where the user can reach when the user extends his hand horizontally in the depth direction. A head-mounted display device. The head-mounted display device according to claim 13,
The range in which the object can be viewed is
In the horizontal direction, one side is a range of 60 degrees or less with respect to the user's line-of-sight direction, and the other side is a range of 60 degrees or less,
A head-mounted display device in which, in the vertical direction, the upper side of the line-of-sight direction is a range of 30 degrees or less and the lower side of the line-of-sight direction is a range of 40 degrees or less. The head-mounted display device according to any one of claims 12 to 14,
The imaging processing unit includes an object motion detection unit that detects a motion of the object based on captured image data of a plurality of frames captured over time when imaging of the outside scene is performed,
The image display control unit is a head-mounted display device that changes display contents of the virtual image based on the movement of the object detected by the object movement detection unit. A method for controlling a head-mounted display device including an outside scene imaging camera that images an outside scene,
(A) capturing the outside scene by the outside scene imaging camera and obtaining captured image data;
(B) image-processing the captured image data to generate processed image data;
(C) determining whether or not a predetermined object exists in the image represented by the processed image data;
(D) including a step of switching the processing mode from the first processing mode to the second processing mode when it is determined in step (c) that the object exists.
The first processing mode is a processing mode for processing the captured image data,
The control method, wherein the second processing mode is a processing mode having a larger data processing amount per unit time than the first processing mode, and is a processing mode for processing the captured image data. A method for controlling a head-mounted display device including an outside scene imaging camera that images an outside scene,
(A) detecting the presence of an object in a certain object detection region in the imaging region of the outside scene imaging camera;
(B) a step of starting imaging of the outside scene by the outside scene imaging camera when the presence of the object is detected in the step (a).

Images