WO2017094557A1

WO2017094557A1 - Electronic device and head-mounted display

Info

Publication number: WO2017094557A1
Application number: PCT/JP2016/084546
Authority: WO
Inventors: 軌行石井
Original assignee: コニカミノルタ株式会社
Priority date: 2015-12-02
Filing date: 2016-11-22
Publication date: 2017-06-08

Abstract

The present invention provides an electronic device and a head-mounted display that are capable of appropriately adjusting the brightness of an image displayed on a screen while providing non-contact detection of movement and/or changes in shape of, e.g., an indicator for inputting gesture operations. In the electronic device, when an indicator detection device has detected that the indicator is within a detection area, a brightness control device interrupts execution of the normal brightness adjustment mode and executes a special brightness adjustment mode in accordance with movement and/or changes in shape of the indicator, the special brightness adjustment mode being different from the brightness adjustment mode.

Description

Electronic equipment and head mounted display

The present invention relates to an electronic device provided with an image display device and a head mounted display.

In recent years, mobile terminals such as smartphones that have been rapidly developed are often used for business and home work assistance. A typical mobile terminal has a touch panel screen that serves both as an image display and a user interface. By touching this screen, the user can make necessary inputs to display a desired image or input information. Can be performed. By the way, the visibility of the image displayed on the screen changes according to the ambient brightness. That is, even images with the same brightness are difficult to see when the surroundings are relatively bright, but are easy to see when the surroundings are relatively dark. Therefore, in general mobile terminals, display control is performed in which ambient brightness is detected using an illuminance sensor or the like, and the brightness of an image displayed on the screen is changed according to the detected ambient brightness. Yes.

On the other hand, when the user's hand is wet or dirty, there are cases where it is desired to operate the mobile terminal without touching the screen. In response to such a request, the light emitted from the mobile terminal is reflected on the user's hand, and the reflected light is detected by the infrared proximity sensor to detect the movement of the hand, and by the movement (gesture) of the hand. A technology capable of performing an input operation desired by a user such as swipe without contact has been developed. If such a technique is used, even if the user's hand is dirty, there is no possibility of contaminating the screen of the portable terminal.

However, when performing an input operation with a gesture in this way, there is a possibility that a shadow will be created when the hand is close to the illuminance sensor, and the state of this shadow may be erroneously detected as ambient brightness. In such a case, the brightness of the image displayed on the screen may change every time the hand is moved closer / away to perform a gesture operation, and the user may feel uncomfortable, and power consumption for brightness control May also increase.

On the other hand, in Patent Document 1, when the brightness measured by the optical sensor changes from light to dark, when there is a touch at a predetermined position on the touch panel, the timing of reducing the brightness of the image displayed on the screen. A mobile terminal that delays the delay is disclosed. On the other hand, Patent Document 2 discloses a technique for switching between enabling / disabling brightness adjustment of a display unit by using an inclination sensor and a proximity sensor together. If it is possible to combine such technologies, when a hand is approached to perform a gesture operation, it becomes possible for the proximity sensor to detect this and invalidate the normal brightness adjustment. It can be said that it is possible to avoid the problem that the brightness of the image displayed on the screen changes every time the operation is performed.

JP 2012-137859 A JP 2014-107153 A

However, even if these known techniques can be combined, the user may not be able to effectively resolve the uncomfortable feeling. More specifically, for example, a gesture operation is different from a tapping operation, and involves movement of an indicator such as a hand and a wide range of shape changes. Brightness adjustment is required.

The present invention has been made in view of the above circumstances. For example, an image displayed on a screen is detected while detecting movement and / or shape change of a pointer in a non-contact manner for input of a gesture operation. An object of the present invention is to provide an electronic device and a head mounted display capable of appropriately adjusting brightness.

In order to achieve at least one of the above objects, an electronic device reflecting one aspect of the present invention is provided.
A pointer detection apparatus that has a detection region and detects whether or not a pointer moved by a user exists in the detection region, and movement and / or shape change of the pointer in the detection region in a non-contact manner; ,
A brightness detection device for detecting ambient brightness including the detection area;
An image display device for displaying an image;
A brightness control device that executes a normal brightness adjustment mode that adjusts the brightness of an image displayed on the image display device according to the ambient brightness detected by the brightness detection device;
When the indicator detection device detects that the indicator is present in the detection area, the luminance control device interrupts execution of the normal luminance adjustment mode, and the movement and / or shape of the indicator A special brightness adjustment mode different from the brightness adjustment mode is executed according to the change.

According to the present invention, for example, an electronic device and a head mount that can appropriately adjust the brightness of an image displayed on a screen while detecting movement and / or shape change of an indicator without contact for input of a gesture operation, for example. A display can be provided.

It is a perspective view of the head mounted display (HMD) concerning this embodiment. It is the figure which looked at HMD from the front. It is the figure which looked at HMD from the upper part. It is a schematic sectional drawing which shows the structure of a display unit. It is an enlarged view of a proximity sensor. It is a block diagram of the main circuit of HMD. It is a front view when a user wears HMD. It is a side view when a user wears HMD. It is a top view when a user wears an HMD. It is a figure which shows the example of the signal waveform which a some light reception area | region outputs. It is a figure which shows the other example of the signal waveform which a some light reception area | region outputs. It is a figure which shows the other example of the signal waveform which a light reception area | region outputs. It is a figure which shows the other example of the signal waveform which a light reception area | region outputs. 5 is a flowchart of luminance adjustment control executed by a control processing unit 121. It is a figure which shows typically gesture operation in which hand HD passes detection area SA in less than 2 second. It is a figure which shows typically gesture operation in which hand HD stays in detection area SA for 2 seconds or more. It is a figure which shows typically gesture operation which hand HD returns to detection area SA. It is a figure which shows typically gesture operation which hand HD stays in detection area SA for 2 seconds or more, and rotates. It is a figure which shows typically gesture operation which hand HD stayed in detection area SA for 2 seconds or more, and changed shape. It is the graph which takes the brightness | luminance of the image displayed on a display on a vertical axis | shaft, and takes time on a horizontal axis. It is another graph which shows the brightness | luminance of the image displayed on a display on a vertical axis | shaft, and takes time on a horizontal axis. It is a figure which shows the example of gesture operation which moved hand HD repeatedly up and down (or right and left, diagonal, and near) (it is called continuous action). It is a graph which takes the surrounding brightness based on the output signal of the illumination intensity sensor 112, and takes time on a horizontal axis. It is another graph which takes the surrounding brightness based on the output signal of the illumination intensity sensor 112, and takes time on a horizontal axis. It is a figure which shows the example of gesture operation which rotated hand HD. It is a graph which takes the surrounding brightness based on the output signal of the illumination intensity sensor 112, and takes time on a horizontal axis.

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a perspective view of a head mounted display (hereinafter referred to as HMD) 100 that is an electronic apparatus according to the present embodiment. FIG. 2 is a front view of the HMD 100 according to the present embodiment. FIG. 3 is a view of the HMD 100 according to the present embodiment as viewed from above. Hereinafter, the right side and the left side of the HMD 100 refer to the right side and the left side for the user wearing the HMD 100.

As shown in FIGS. 1 to 3, the HMD 100 of this embodiment has a frame 101 as a support member. A frame 101 that is U-shaped when viewed from above has a front part 101a to which two spectacle lenses 102 are attached, and

side parts

101b and 101c extending rearward from both ends of the front part 101a. The two spectacle lenses 102 attached to the frame 101 may or may not have refractive power.

A cylindrical main body 103 as a support member is fixed to the front portion 101a of the frame 101 on the upper side of the spectacle lens 102 on the right side (which may be on the left side depending on the user's dominant eye). The main body 103 is provided with a display unit 104. A display control unit 104DR (see FIG. 6 described later) that controls display control of the display unit 104 based on an instruction from the control processing unit 121 described later is disposed in the main body 103. If necessary, a display unit may be arranged in front of both eyes.

FIG. 4 is a schematic cross-sectional view showing the configuration of the display unit 104. The display unit 104 serving as an image display device includes an image forming unit 104A and an image display unit 104B. The image forming unit 104A is incorporated in the main body unit 103, and includes a light source 104a, a unidirectional diffuser 104b, a condenser lens 104c, and a display element 104d. On the other hand, the image display unit 104B, which is a so-called see-through type display member, is disposed on the entire plate so as to extend downward from the main body unit 103 and extend in parallel to one eyeglass lens 102 (see FIG. 1). The eyepiece prism 104f, the deflecting prism 104g, and the hologram optical element 104h.

Next, the operation of the display unit 104 will be described. The light emitted from the light source 104a is diffused by the unidirectional diffusion plate 104b, condensed by the condenser lens 104c, and enters the display element 104d. The light incident on the display element 104d is modulated for each pixel based on the image data input from the display control unit 104DR, and is emitted as image light. Thereby, a color image is displayed on the display element 104d.

Image light from the display element 104d enters the eyepiece prism 104f from its base end face PL1, is totally reflected a plurality of times by the inner side face PL2 and the outer side face PL3, and enters the hologram optical element 104h. The light incident on the hologram optical element 104h is reflected there, passes through the inner side surface PL2, and reaches the pupil B. At the position of the pupil B, the user can observe an enlarged virtual image of the image displayed on the display element 104d, and can visually recognize it as a screen formed on the image display unit 104B. In this case, it can be considered that the hologram optical element 104h constitutes a screen, or it can be considered that a screen is formed on the inner surface PL2. In the present specification, “screen” may refer to an image to be displayed.

On the other hand, since the eyepiece prism 104f, the deflecting prism 104g, and the hologram optical element 104h transmit almost all of the external light, the user can observe an external field image (real image) through them. Therefore, the virtual image of the image displayed on the display element 104d is observed so as to overlap with a part of the external image. In this manner, the user of the HMD 100 can simultaneously observe the image provided from the display element 104d and the external image via the hologram optical element 104h. Note that when the display unit 104 is in the non-display state, the image display unit 104B is transparent, and only the external image can be observed. In this example, a display unit is configured by combining a light source, a liquid crystal display element, and an optical system. However, instead of a combination of a light source and a liquid crystal display element, a self-luminous display element (for example, an organic EL display) is used. Element) may be used. Further, instead of a combination of a light source, a liquid crystal display element, and an optical system, a transmissive organic EL display panel having transparency in a non-light emitting state may be used. In any case, when the screen is arranged so as to fall within the visual field of the user's eye facing the image display unit 104B, and preferably at least partially overlaps the effective visual field, the user can easily visually recognize the image. Can do.

Further, in FIGS. 1 and 2, on the front surface of the main body 103, the proximity sensor 105 disposed near the center of the frame 101, the lens 106 a of the camera 106 disposed near the side of the frame 101, and the proximity sensor 105. And the illuminance sensor 112 disposed between the lens 106a and the lens 106a so as to face each other.

An illuminance sensor, which is an example of a brightness detection device, generates a photocurrent according to the ambient brightness using a photoelectric conversion element such as a photodiode, for example, and amplifies this with a current mirror amplifier to obtain a desired output. A signal (output current) is generated, and the one described in Japanese Patent Application Laid-Open No. 2009-158928 can be used. However, it is possible to detect ambient brightness including the detection region of the indicator detection device. For example, it is not limited to this.

In this specification, the “proximity sensor” which is an example of the indicator detection device is a proximity sensor for detecting that an object, for example, a part of a human body (such as a hand or a finger) is close to the user's eyes. This means that a signal is output by detecting whether or not it exists in a detection area in the proximity range in front of the detection surface of the sensor. The proximity range may be set as appropriate according to the operator's characteristics and preferences. For example, the proximity range from the detection surface of the proximity sensor may be within a range of 200 mm. If the distance from the proximity sensor is 200 mm or less, the user can easily put the palm and fingers into and out of the user's field of view with the arm bent, so that the user can easily operate with gestures using the hands and fingers. In addition, the risk of erroneous detection of a human body or furniture other than the user is reduced. Here, the control circuit determines that the object exists in the proximity range based on a signal output from the proximity sensor when the object enters the detection area in the proximity range in front of the proximity sensor. When an object enters the detection area, an effective signal may be output from the proximity sensor to the control circuit.

There are two types of proximity sensors: passive type and active type. A passive proximity sensor has a detection unit that detects invisible light and electromagnetic waves emitted from an object when the object approaches. As a passive proximity sensor, there are a pyroelectric sensor that detects invisible light such as infrared rays emitted from an approaching human body, and a capacitance sensor that detects a change in electrostatic capacitance between the approaching human body and the like. The active proximity sensor includes an invisible light and sound wave projection unit, and a detection unit that receives the invisible light and sound wave reflected and returned from the object. Active proximity sensors include infrared sensors that project infrared rays and receive infrared rays reflected by objects, laser sensors that project laser beams and receive laser beams reflected by objects, and project ultrasonic waves. Then, there is an ultrasonic sensor that receives ultrasonic waves reflected by an object. Note that a passive proximity sensor does not need to project energy toward an object, and thus is excellent in low power consumption. Active proximity sensors can improve detection reliability. For example, even if the user is wearing gloves that do not transmit detection light emitted from the human body such as infrared light, the movement of the user's hand Can be detected. A plurality of types of proximity sensors may be used in combination.

Proximity sensors are generally smaller and cheaper than cameras, and consume less power. In addition, it is difficult for the proximity sensor to perform complicated detection such as detection of the shape of the object, but it can determine the approach and separation of the object from the detection area, so that the user can pass the hand or finger or the palm of the hand. By holding it over, the HMD can be operated, and complicated image processing required for gesture recognition by analysis of the image captured by the camera is also unnecessary.

FIG. 5 is an enlarged view of the proximity sensor 105 used in the present embodiment as viewed from the front. In the present embodiment, an example in which a pyroelectric sensor is used as the proximity sensor 105 will be described. In FIG. 5, the proximity sensor 105 includes a light receiving unit 105 a that receives invisible light such as infrared light emitted from a human body as detection light. The light receiving unit 105a forms light receiving areas RA to RD arranged in 2 rows and 2 columns, and when receiving invisible light, signals corresponding to the light receiving areas RA to RD are individually output from the light receiving areas RA to RD. It has become. The output of each of the light receiving areas RA to RD changes in intensity according to the distance from the light receiving unit 105a to the object, and the intensity increases as the distance decreases. If it is a pyroelectric sensor that detects infrared light emitted from the human body, it is difficult to falsely detect an object other than the exposed human body. For example, when working in a narrow space, the false detection is effectively prevented. There is an advantage that you can.

1 and 2, the right sub-body portion 107 is attached to the right side portion 101b of the frame 101, and the left sub-body portion 108 is attached to the left side portion 101c of the frame 101. The right sub-main body portion 107 and the left sub-main body portion 108 have an elongated plate shape, and have elongated

protrusions

107a and 108a on the inner side, respectively. By engaging the elongated protrusion 107 a with the elongated hole 101 d of the side portion 101 b of the frame 101, the right sub-body portion 107 is attached to the frame 101 in a positioned state, and the elongated protrusion 108 a is attached to the side of the frame 101. By engaging with the long hole 101e of the portion 101c, the left sub-main body portion 108 is attached to the frame 101 in a positioned state.

In the right sub-body portion 107, there are a geomagnetic sensor 109 (see FIG. 6 to be described later) for detecting geomagnetism, and an angular velocity sensor 110B and an acceleration sensor 110A (see FIG. 6 to be described later) that generate an output corresponding to the posture. The left sub-main body 108 is provided with a speaker / earphone 111C and a microphone 111B (see FIG. 6 described later). The main main body 103 and the right sub main body 107 are connected so as to be able to transmit signals through a wiring HS, and the main main body 103 and the left sub main body 108 are connected so as to be able to transmit signals through a wiring (not shown). Yes. As schematically illustrated in FIG. 3, the right sub-main body 107 is connected to the control unit CTU via a cord CD extending from the rear end. A 6-axis sensor in which an angular velocity sensor and an acceleration sensor are integrated may be used. Further, the HMD can be operated by sound based on an output signal generated from the microphone 111B according to the input sound. Further, the main main body 103 and the left sub main body 108 may be configured to be wirelessly connected. The color temperature sensor 113 and the temperature sensor 114 are optional.

FIG. 6 is a block diagram of main circuits of the HMD 100. The control unit CTU generates a control signal for the display unit 104 and other functional devices, a control processing unit 121 as a luminance control device, an operation unit 122, and a GPS receiving unit 123 that receives radio waves from GPS satellites. A communication unit 124 for exchanging data with the outside, a ROM 125 for storing programs and the like, a RAM 126 for storing image data and the like, and a power source for converting the voltage applied from the battery 127 into an appropriate voltage for each unit The circuit 130 and a storage device 129 such as an SSD or a flash memory are included. Although the control processor 121 can use an application processor used in a smartphone or the like, the type of the control processor 121 is not limited. For example, if an application processor includes hardware necessary for image processing such as GPU or Codec as a standard, it can be said that the processor is suitable for a small HMD. The control processing unit 121 includes a control unit 121A, a gesture processing unit 121B, and a detection result using unit 121C in order to perform luminance adjustment described later.

Furthermore, when the light receiving unit 105a detects invisible light as detection light emitted from the human body, the signal is input from the proximity sensor 105 to the control processing unit 121, and an illuminance sensor that detects ambient brightness. The signal from 112 is input. Further, the control processing unit 121 controls image display on the display unit 104 via the display control unit 104DR.

The control processing unit 121 receives power from the power supply circuit 130, operates according to a program stored in at least one of the ROM 124 and the storage device 129, and receives an image from the camera 106 according to an operation input such as power-on from the operation unit 122. Data can be input and stored in the RAM 126, and can be communicated with the outside via the communication unit 124 as necessary. Furthermore, when the control processing unit 121 executes image control according to the output from the proximity sensor 105, the user can perform screen control of the HMD 100 by gesture operation using a hand or a finger. Examples of screen control include page turning, scrolling, selection, and determination, but details are omitted.

FIG. 7 is a front view when the user US wears the HMD 100 of the present embodiment. FIG. 8 is a side view when the user US wears the HMD 100, and FIG. 9 is a top view thereof, which is shown together with the user's hand. Here, the gesture operation is an operation in which at least the hand HD or finger of the user US passes through the detection area of the proximity sensor 105 or an action of changing the shape, and is detected by the control processing unit 121 of the HMD 100 via the proximity sensor 105. It can be done.

Next, the basic principle of gesture operation detection will be described. If nothing is present in front of the user US when the proximity sensor 105 is operating, the light receiving unit 105a does not receive invisible light as detection light, and thus the control processing unit 121 is not performing a gesture operation. Judge. On the other hand, as shown in FIG. 8, when the user US's own hand HD is approached in front of the user US, the light receiving unit 105a detects the invisible light emitted from the hand HD, and the proximity sensor 105 based on the invisible light is detected. In response to the output signal, the control processing unit 121 determines that a gesture operation has been performed. In the following description, it is assumed that the gesture operation is performed by the user's hand HD. However, the gesture operation may be performed by the user using a pointing device made of a material that can emit invisible light. May be performed. An indicator can be moved by the user's intention. Note that an imaging device (camera) can be used as the detection device instead of the proximity sensor. In such a case, the moving indicator is imaged by the imaging device, and the moving direction is determined by the image processing device (part of the control device) based on the image data output from the imaging device.

As described above, the light receiving unit 105a has the light receiving regions RA to RD arranged in 2 rows and 2 columns (see FIG. 5). Therefore, when the user US moves the hand HD closer to the front of the HMD 100 from either the left, right, up, or down directions, the output timings of signals detected in the light receiving areas RA to RD are different.

10 and 11 are examples of signal waveforms of the light receiving areas RA to RD, where the vertical axis represents the signal intensity of the light receiving areas RA to RD and the horizontal axis represents the time. For example, when the user US moves the hand HD from the right to the left in front of the HMD 100 with reference to FIGS. 8 and 9, invisible light emitted from the hand HD enters the light receiving unit 105a. In this case, the light receiving areas RA and RC first receive invisible light. Therefore, as shown in FIG. 10, first, the signals in the light receiving areas RA and RC rise, the signals in the light receiving areas RB and RD rise later, and the signals in the light receiving areas RA and RC fall, and then the light receiving areas RB, The RD signal falls. The control processing unit 121 detects the timing of this signal, and determines that the user US has performed the gesture operation by moving the hand HD from the right to the left.

In the example of FIG. 11, the signals of the light receiving areas RA and RB rise first, the signals of the light receiving areas RC and RD rise after a delay, and the signals of the light receiving areas RA and RB further fall, and then the signals of the light receiving areas RC and RD. Is falling. The control processing unit 121 detects the timing of this signal, and determines that the user US has performed the gesture operation by moving the hand HD from the top to the bottom.

Furthermore, based on the output from the proximity sensor 105, the control processing unit 121 can also detect that the hand HD has moved in the front-rear direction. FIG. 12 shows an example of a signal waveform in which the vertical axis represents the average signal intensity of the light receiving regions RA to RD and the horizontal axis represents time. The waveform indicated by the solid line is high in intensity immediately after output and decreases with time. Therefore, when such a waveform is output from the proximity sensor 105, the control processing unit 121 determines that the gesture operation is performed by moving the hand HD away from the vicinity of the face. On the other hand, the waveform indicated by the dotted line is low in intensity immediately after output and increases with time. Therefore, when such a waveform is output from the proximity sensor 105, the control processing unit 121 determines that the gesture operation is performed by moving the hand HD so as to approach the face from a distance.

In addition, based on the output from the proximity sensor 105, the control processing unit 121 can detect a shape change of the hand HD. FIG. 13 shows an example of a signal waveform in which the vertical axis represents the average signal intensity of the light receiving regions RA to RD and the horizontal axis represents time. The waveform indicated by the solid line is high in intensity immediately after output and is further lowered after a certain period of time. Therefore, when such a waveform is output from the proximity sensor 105, the control processing unit 121 determines that a gesture operation has been performed with the hand HD opened in the detection area closed. On the other hand, the waveform indicated by the dotted line is low in intensity immediately after output and increases further after a certain period of time. Therefore, when such a waveform is output from the proximity sensor 105, the control processing unit 121 determines that a gesture operation has been performed with the hand HD closed in the detection area opened.

According to the present embodiment, by using the proximity sensor, the presence / absence of the hand HD and the movement and shape change are reliably detected by positioning the detection region within the visual field of the user's eye facing the image display unit. can do. Accordingly, the user US can view the hand operation and the operation in conjunction with each other, and can realize an intuitive operation. Furthermore, since the proximity sensor is small and consumes less power than the camera, the continuous operation time of the HMD 100 can be extended.

On the other hand, the control processing unit 121 determines ambient brightness based on a signal from the illuminance sensor 112 and adjusts the luminance of an image displayed on the display unit 104 via the display control unit 104DR. More specifically, when the control processing unit 121 determines that the surrounding is relatively bright based on the signal from the illuminance sensor 112, the brightness of the image displayed on the display unit 104 is made bright and easy to see, When it is determined that the surroundings are relatively dark, the brightness of the image displayed on the display unit 104 is reduced to make it easy to see, thereby saving energy. This is a normal brightness adjustment mode (automatic brightness adjustment; also referred to as normal brightness adjustment mode). It is assumed that the luminance adjustment is not performed when the luminance adjustment mode is interrupted.

Here, when the user US brings the hand HD closer to the detection area SA of the proximity sensor 105 for a gesture operation, the shadow of the hand HD may be applied to the illuminance sensor 112. When the normal brightness adjustment mode is set, based on the output signal of the illuminance sensor 112, the control processing unit 121 determines that the ambient brightness has become dark and reduces the brightness of the image. . On the other hand, the movement of the hand HD in the gesture operation is often instantaneous, so that the shadow of the hand HD is separated from the illuminance sensor 112, and accordingly, the control processing unit 121 determines that the surrounding brightness has become brighter. This will increase the brightness of the image. As described above, when the luminance of the image fluctuates according to the gesture operation, the user US may feel uncomfortable. Therefore, in the present embodiment, the following control is performed.

FIG. 14 is a flowchart of brightness adjustment control executed by the control processing unit 121. Here, it is assumed that the control processing unit 121 executes a normal luminance adjustment mode in accordance with an output signal from the illuminance sensor 112. In this state, in step S101 of FIG. 14, the control processing unit 121 determines whether or not the hand HD as the indicator has entered the detection area SA (determination No in step S101).

When the user US causes the hand HD to enter the detection area SA for the gesture operation, the output signal of the proximity sensor 105 changes (Yes in Step S101), so that the control processing unit 121 performs a normal operation in Step S102. Suspend the brightness adjustment mode. In subsequent step S103, the control processing unit 121 determines whether or not the hand HD as the indicator has stopped in the detection area SA for, for example, two seconds or more as the first specified time. Even when the hand HD is moved slowly, it is determined to be stopped as long as it remains in the detection area SA for 2 seconds or more.

When the hand HD does not stop in the detection area SA for 2 seconds or more, for example, and passes within less than 2 seconds, the control processing unit 121 returns the hand HD as the indicator to the detection area SA again in step S105. Judge whether to come back. For example, if it is determined that the hand HD does not return to the detection area SA within 2 seconds as the third specified time, the control processing unit 121 resumes the normal brightness adjustment mode in step S109.

On the other hand, when the hand HD as the indicator is stopped in the detection area SA for, for example, 2 seconds or more as the first specified time, the control processing unit 121 keeps the hand HD in the detection area SA in step S103. In step S104, a special brightness adjustment mode corresponding to the gesture operation is set. Thereafter, the flow proceeds to step S107, and the control processing unit 121 executes the set special brightness adjustment mode. In subsequent step S108, the control processing unit 121 determines whether or not the hand HD remains in the detection area SA, and if it is determined that the hand HD remains in the detection area SA, If the flow is returned to S102, but it is determined that the hand HD does not remain in the detection area SA, the normal luminance adjustment mode is resumed in Step S109.

Further, if it is determined in step S105 that the hand HD has returned to the detection area SA again (regardless of the return direction), the control processing unit 121 sets a special brightness adjustment mode corresponding to the gesture operation in step S106. . Thereafter, the flow proceeds to step S107, and the control processing unit 121 executes the set special brightness adjustment mode. Since step S107 and subsequent steps are the same, description thereof is omitted.

FIG. 15 is a diagram schematically showing a gesture operation in which the hand HD passes through the detection area SA in less than 2 seconds, and is shown together with the proximity sensor 105. As shown in FIG. 15A, when the hand HD is placed on the right side of the detection area SA and is moved to the left from the detection area SA, only the light receiving areas RA and RB of the proximity sensor 105 are first shown in FIG. Reacts as shown by hatching. Further, as shown in FIG. 15C, when the hand HD comes to the center of the detection area SA, the entire light receiving area of the proximity sensor 105 reacts as shown by hatching, and thereafter, in FIG. When the HD is moved in the same direction, only the light receiving regions RC and RD of the proximity sensor 105 react as shown by hatching, and as shown in FIG. The region becomes unresponsive. In such a case, the control processing unit 121 interrupts the normal brightness adjustment mode when the state shown in FIG. 15B is determined, and resumes when the state shown in FIG. 15E is determined.

FIG. 16 is a diagram schematically showing a gesture operation in which the hand HD stays in the detection area SA for 2 seconds or more. FIGS. 16A to 16C are similar to FIGS. 15A to 15C, but are examples in which the hand HD remains in FIG. 16D. In such a case, the control processing unit 121 interrupts the normal brightness adjustment mode when the state of FIG. 16B is determined, and then sets the special brightness adjustment mode when the state of FIG. 16D is determined. .

FIG. 17 is a diagram schematically illustrating a gesture operation in which the hand HD returns to the detection area SA. 17 (a) to 17 (d) are the same as FIGS. 15 (a) to 15 (d), but in FIG. 17 (e), the hand HD that has escaped from the detection area SA returns to the detection area SA again. This is an example. In such a case, the control processing unit 121 interrupts the normal brightness adjustment mode at the time when the state of FIG. 17B is determined, and then performs a gesture when it is determined that the state of FIG. Set the special brightness adjustment mode according to the operation.

FIG. 18 is a diagram schematically showing a gesture operation in which the hand HD stays in the detection area SA for 2 seconds or more and rotates. 18A to 18C are the same as FIGS. 15A to 15C, but in FIG. 18D, the hand HD stays in the detection area SA and in front of the user US. It is the example rotated. In such a case, the control processing unit 121 interrupts the normal brightness adjustment mode at the time when the state of FIG. 18B is determined, and then determines the special brightness corresponding to the gesture operation at the time of determining the state of FIG. Set the adjustment mode.

FIG. 19 is a diagram schematically showing a gesture operation in which the hand HD remains in the detection area SA for 2 seconds or more and the shape is changed. FIGS. 19A to 19C are the same as FIGS. 15A to 15C, but in FIG. 19D, the hand HD remains in the detection area SA and is closed from the open state. It is an example that was changed to. In such a case, the control processing unit 121 interrupts the normal brightness adjustment mode at the time when the state of FIG. 19B is determined, and then determines the special brightness corresponding to the gesture operation at the time of determining the state of FIG. Set the adjustment mode.

Next, the special brightness adjustment mode will be described. First, the special brightness adjustment mode when the hand HD is stopped in the detection area SA for 2 seconds or more will be described. FIG. 20 is a graph showing the luminance of the image displayed on the display on the vertical axis and the time on the horizontal axis. The alternate long and short dash line is a graph showing luminance adjustment in the normal luminance adjustment mode as a comparative example, and the solid line is a graph showing luminance adjustment in the special luminance adjustment mode. This special brightness adjustment mode is suitable when the gesture operation shown in FIG. 16 is performed.

First, when the hand HD is brought close to the detection area SA as shown in FIG. 16 (a) at time T0 in FIG. 20, the proximity sensor 105 detects the hand HD in the stage shown in FIG. 16 (b). Next, when the illuminance sensor 112 detects the shadow of the hand HD at the time T1, in the luminance adjustment in the normal luminance adjustment mode as a comparative example, as shown by a one-dot chain line, the minimum luminance value MIN (for example, the same image The brightness is reduced at a stroke until the minimum value of the average brightness at the time of comparison. However, when the hand HD exits from the detection area SA in a short time, if the brightness value of the image varies greatly, the user US may feel uncomfortable and energy saving cannot be achieved.

On the other hand, in the special luminance adjustment mode indicated by the solid line, the luminance adjustment is performed with a change rate smaller than the luminance reduction rate (change rate of luminance with respect to time) in the normal luminance adjustment mode. Furthermore, when the illuminance sensor 112 no longer detects the shadow of the hand HD at time T3, the brightness value is returned to the original maximum value MAX in any mode, but at the time T3, the brightness is increased by the normal brightness adjustment mode. The brightness value in the special brightness adjustment mode is higher than the value. That is, while the gesture operation is performed, the luminance value changes less, so it can be said that the user US is less likely to feel uncomfortable. Note that the brightness value is maximized when the hand HD is not detected, and the brightness value is minimized when the hand HD is detected. However, the control is basically performed to change the brightness value according to the ambient brightness. Therefore, it is an example of the embodiment that the luminance value is maximized when the hand HD is not detected, and the luminance value is minimized when the hand HD is detected, and is not limited thereto. . The same applies to the following embodiments.

However, if the time (second prescribed time) from time T1 to time T2 when the hand HD stays in the detection area SA is relatively long, for example, 5 seconds or more, the luminance value of the image may be further reduced. There is little fear of the user US feeling uncomfortable and energy saving can be achieved. Therefore, in such a case, the special brightness adjustment mode may be terminated and the normal brightness adjustment mode may be executed as indicated by a broken line.

FIG. 21 is a graph in which the vertical axis represents the luminance of the image displayed on the display and the horizontal axis represents time. A one-dot chain line is a graph showing luminance adjustment in a normal luminance adjustment mode as a comparative example, and a solid line is a graph showing luminance adjustment in another special luminance adjustment mode. This special brightness adjustment mode is also suitable when the gesture operation shown in FIG. 16 is performed.

First, when the hand HD is brought close to the detection area SA as shown in FIG. 16 (a) at time T0 in FIG. 21, the proximity sensor 105 detects the hand HD in the stage shown in FIG. 16 (b). Next, when the illuminance sensor 112 detects a shadow of the hand HD at time T1, the luminance is reduced at a stroke to the minimum luminance value MIN2, as indicated by the alternate long and short dash line, in the luminance adjustment in the normal luminance adjustment mode as a comparative example. . On the other hand, in the special brightness adjustment mode indicated by the solid line, the brightness adjustment is performed by reducing the brightness at the same rate of change as the brightness reduction rate in the normal brightness adjustment mode, but the brightness value MID (special brightness) higher than the minimum value MIN2. The reduction is stopped at the minimum luminance value in the adjustment mode. Furthermore, when the illuminance sensor 112 no longer detects the shadow of the hand HD at time T3, the brightness value is restored to the original maximum value MAX in any mode, but even at time T3, the brightness is adjusted by the normal brightness adjustment mode. The minimum luminance value (MID) in the special luminance adjustment mode is higher than the value (MIN). In other words, while the gesture operation is performed, it can be said that the fluctuation of the luminance value is small and the user US is less likely to feel uncomfortable.

Similarly, in the special brightness adjustment mode of this example, when the time (second specified time) from time T1 to time T2 when the hand HD stays in the detection area SA is relatively long, for example, 5 seconds or more, As shown by the broken line, when the time T2 has passed, the special brightness adjustment mode may be terminated and the normal brightness adjustment mode may be resumed. Furthermore, luminance adjustment may be performed by combining the graphs of FIGS. Regardless of the example described above, for example, the control processing unit 121 executes a special brightness adjustment mode in which the brightness adjustment is not performed (the brightness value of the image is maintained) or the default brightness value is forcibly set. Also good.

Next, as shown in FIG. 17, a special brightness adjustment mode in the case of a gesture operation in which the hand HD that has passed through the detection area SA returns will be described. Also in this case, if the time that the hand HD stays in the detection area SA is less than 2 seconds, for example, brightness adjustment is not performed, and if the time that the hand HD stays in the detection area SA is 2 seconds or more, for example, The brightness adjustment mode can be set to perform the brightness adjustment shown in FIGS. 20 and 21, or the brightness adjustment can be omitted. Also, in the case of the gesture operation shown in FIGS. 18 and 19, if the time that the hand HD stays in the detection area SA is, for example, less than 2 seconds, the brightness adjustment is not performed and the time that the hand HD stays in the detection area SA. If, for example, 2 seconds or longer, the special brightness adjustment mode can be set and the brightness adjustment shown in FIGS. 20 and 21 can be performed, or the brightness adjustment can not be performed.

By the way, as shown in FIG. 22, in the case of a gesture operation in which the hand HD is repeatedly moved up and down (or left and right, diagonally, and near) (referred to as continuous operation), or in the case of a gesture operation in which the opening and closing operation of the hand HD is repeated. The illuminance sensor 112 detects periodic fluctuations in brightness. In such a case, it is important how to determine the ambient brightness as a reference for brightness adjustment in the special brightness adjustment mode. FIG. 23 is a graph showing ambient brightness based on the output signal of the illuminance sensor 112 and taking time on the horizontal axis, and shows an example in which continuous operation is performed.

In FIG. 23, the maximum brightness value MXB is obtained when the hand HD is below the illuminance sensor 112 in FIG. 22, and the other brightness values are obtained when the hand HD is above the illuminance sensor 112. It is done. Here, after obtaining the maximum value MXB of brightness, the control processing unit 121 can perform brightness adjustment. More specifically, the control processing unit 121 obtains the maximum brightness MXB during a certain sampling period ST (hereinafter referred to as a fourth specified time; for example, 2 seconds). When the maximum value fluctuates, the average value of the maximum values is obtained, and this is regarded as valid data, and the rest is regarded as invalid data. It is preferable that the control processing unit 121 performs brightness adjustment in the special brightness adjustment mode using only valid data. Since the maximum brightness value MXB indicates the brightness of the outside world and is substantially constant, appropriate brightness of the image can be realized by performing brightness control based on this value.

However, the ambient brightness may change if the lamp is turned over during indoor use. FIG. 24 is a graph showing the ambient brightness based on the output signal of the illuminance sensor 112 and taking the time on the horizontal axis, and shows an example in which the ambient brightness becomes brighter at time T1 during continuous operation. ing. Also in this case, the maximum value MXB1 or MXB2 of brightness during a certain sampling period ST (for example, 2 seconds) is obtained, and this is set as valid data, and the other is set as invalid data. It is preferable that the control processing unit 121 performs brightness adjustment in the special brightness adjustment mode using only valid data. When the sampling period ST extends over time T1, the average value of the maximum values MXB1 and MXB2 may be taken, or one of the values may be used. Using only valid data, the control processing unit 121 can perform luminance adjustment in the special luminance adjustment mode.

As an example in which the maximum value of different brightness occurs in different sampling periods as described above even when the ambient brightness is constant, as shown in FIG. 19, for example, a gesture operation for closing an open hand HD There is. Therefore, when the gesture operation shown in FIG. 19 is detected, the special brightness adjustment mode set in this example may be executed.

Next, a special brightness adjustment mode when the hand HD is rotated in the detection area SA will be described. As shown in FIG. 25, when the hand HD is rotated around the hand HD, the illuminance sensor 112 detects periodic brightness fluctuations. FIG. 26 is a graph showing the ambient brightness based on the output signal of the illuminance sensor 112 and taking the time on the horizontal axis, and shows an example in which such an operation is performed. When the hand HD is turned around, the shadow of the hand on the illuminance sensor 112 fluctuates randomly, so that the ambient brightness based on the output signal fluctuates randomly as shown in FIG. It will be.

Therefore, in such a case, the maximum value MXB of the brightness during a certain sampling period ST (for example, 2 seconds) is obtained, and this is set as valid data, and the other is set as invalid data. Using only valid data, the control processing unit 121 can perform luminance adjustment in the special luminance adjustment mode.

As described above, the present invention has been described using the HMD as an example, but the present invention is not limited to the HMD and can be applied to all electronic devices such as portable terminals.

The present invention is not limited to the embodiments described in the specification, and other embodiments and modifications are included for those skilled in the art from the embodiments and technical ideas described in the present specification. it is obvious. The description and the embodiments are for illustrative purposes only, and the scope of the present invention is indicated by the following claims.

100 HMD
101 Frame 101a Front part

101b Side part

101c Side part 101d Long hole 101e Long hole 102 Eyeglass lens 103 Main body part 104 Display 104A Image forming part 104B Image display part 104DR Display control part 104a Light source 104b Unidirectional diffuser

104c Condensing lens

104d Display element

104f Eyepiece prism

104g Deflection prism 104h Hologram optical element 104i Screen 105 Proximity sensor 105a Light receiving part 106 Camera 106a Lens 107 Right sub-main part 107a Protrusion 108 Left sub-main part 108a Protrusion 109 Acceleration sensor 111B Microphone 111C Speaker / Earphone 113 Color temperature Sensor 114 Temperature sensor 121 Control processing unit 122 Operation unit 123 GPS reception unit 124 Communication unit 125 ROM
126 RAM
127 Battery 129 Storage device 130 Power supply circuit CD Code CTU Control unit HD Hand HS Wiring PL1 Base end surface PL2 Inner side surface PL3 Outer side surface PL4 Inclined surface PL5 Inclined surface RA-RD Light receiving area SA Detection area US User

Claims

A pointer detection apparatus that has a detection region and detects whether or not a pointer moved by a user exists in the detection region, and movement and / or shape change of the pointer in the detection region in a non-contact manner; ,
A brightness detection device for detecting ambient brightness including the detection area;
An image display device for displaying an image;
A brightness control device that executes a normal brightness adjustment mode that adjusts the brightness of an image displayed on the image display device according to the ambient brightness detected by the brightness detection device;
When the indicator detection device detects that the indicator is present in the detection area, the luminance control device interrupts execution of the normal luminance adjustment mode, and the movement and / or shape of the indicator An electronic device that executes a special brightness adjustment mode different from the brightness adjustment mode according to a change.
When the indicator detection device detects that the indicator remains in the detection region for a first specified time or more, the luminance control device detects the luminance of the image as compared with the normal luminance adjustment mode. The electronic device according to claim 1, wherein the special brightness adjustment mode for gradual time change rate is executed.
When the indicator detection device detects that the indicator remains in the detection area for a first specified time or more, the brightness control device is configured to perform the normal operation when the ambient brightness is the same condition. 3. The electronic device according to claim 1, wherein the special brightness adjustment mode set to increase the minimum brightness value of the image more than the minimum brightness value of the image set by the brightness adjustment mode is executed.
When the indicator detection device detects that the indicator stays in the detection area for a second specified time longer than the first specified time, the luminance control device performs the special luminance adjustment mode. The electronic device according to claim 3, wherein the electronic device is terminated and executes the normal luminance mode.
When the indicator detection device detects that the indicator remains in the detection area for a second specified time longer than the first specified time, the luminance control device detects the luminance of the current image. The electronic device according to claim 3, wherein the special brightness adjustment mode set to maintain the value is executed.
When the indicator detection device detects that the indicator has entered the detection region again within a third specified time after passing through the detection region, the brightness control device 6. The brightness control mode is resumed when the brightness control mode is executed and when it is detected that the brightness has not entered the detection area within the third specified time. The electronic device in any one.
When the brightness detection device detects that the ambient brightness changes according to the movement and / or shape change of the indicator detected by the indicator detection device, the brightness control device detects 7. The electronic apparatus according to claim 1, wherein the special brightness adjustment mode for adjusting the brightness value of the image based on the maximum value of the ambient brightness that has been performed is executed.
The brightness control device uses the detected maximum value of the ambient brightness as valid data when adjusting the brightness value of the image, and uses the detected other ambient brightness as the brightness value of the image. The electronic device according to claim 7, wherein the special brightness adjustment mode is set as invalid data for adjustment.
The brightness detection device detects the ambient brightness over a fourth specified time, and the brightness control device is based on a maximum value of the ambient brightness detected at the fourth specified time. The electronic device according to claim 6 or 7, wherein the special brightness adjustment mode for adjusting a brightness value of an image is executed.
A head-mounted display equipped with the electronic device according to any one of claims 1 to 9, wherein the indicator detection device, the brightness detection device, and the image display device are attached to the user's head. A support member, and the support member supports the image display device so as to be positioned in front of the user's eyes, and detects the indicator so that a detection area of the indicator detection device is in front of the user's eyes. A head-mounted display that supports the device.