JP2020048134A - Display apparatus and control method of the same - Google Patents

Abstract

To provide a method for controlling an output image of a head-mount display, by which a safe use condition can be provided to a user without obstructing an imaging operation of an imaging device.SOLUTION: A display apparatus includes: an imaging device (10) capable of outputting a photographed image; a first inclination detection part (14) for detecting inclination of the imaging device (10) with respect to a state of facing the front; image display means (100) capable of displaying an output image from the imaging device (10) and an image allowing the user to check the environment, to be mounted on a user's head to display an image in front of the user's eye; and a second inclination detection part (104) for detecting inclination of the image display means (100) with respect to a state of facing the front and of being oriented in the same direction as the imaging device (10). When a relative value between angles detected by the first inclination detection part (14) and the second inclination detection part (104) exceeds a threshold, the image display means (100) is controlled to display an image allowing the user to check the environment.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a head-mounted display (hereinafter, HMD) capable of displaying an image as an external display device of an imaging device, and particularly to a method of controlling an output image of the HMD.

  2. Description of the Related Art Conventionally, there is known a technique in which an imaging device and an HMD are linked to each other, and an output image of the imaging device is displayed on the HMD and checked. HMD is often recognized as a display medium of so-called image synthesis technology, but in recent years, it is expected to be applied not only in games and entertainment but also in the range of games and entertainment due to its increased resolution, increased viewing angle, and two-sided display method. Have been. The HMD includes an immersive HMD that allows the display to check only the output image in an immersive state without transmitting external light, and a transmissive HMD that allows the display to transmit the external light and superimpose the output image on the user's field of view to check. Exists.

  In these HMDs, it is important to control output images such as the presence or absence of image superposition and switching of output images from the viewpoint of safety and ease of use. In particular, since the peripheral scene becomes invisible in an immersive HMD, an HMD capable of confirming the peripheral scene by changing (switching or superimposing) an output image by separately arranging an imaging device for capturing the peripheral scene in consideration of safety. Existing.

  In Japanese Patent Application Laid-Open No. H10-157, a peripheral scene photographing image pickup apparatus is arranged, and the display form and display position of the peripheral scene image and the additional image information are changed by either the movement of the head of the user (driver) or the movement of the line of sight. A variable HMD is disclosed.

JP-A-10-206789

  However, in the related art disclosed in the above-described patent document, there is a possibility that an appropriate display change cannot be performed in a state where the shooting direction of the imaging device cannot be controlled in conjunction with the movement of the HMD. Specifically, in a state where the image of the imaging device in a hand-held state is output to the HMD for confirmation, the imaging direction of the imaging device may not match the visual direction when the HMD is attached, and at an unintended timing, Unintended output image control may be performed. In such a case, inconvenience occurs in terms of safety and ease of use.

  Therefore, an object of the present invention is to provide a head mounted display output image control method capable of providing a user with a safe use state without hindering a shooting operation of a shooting device.

In order to achieve the above object, a display device according to the present invention includes:
An imaging device capable of outputting a captured image; a first inclination detection unit for detecting an inclination from a state in which the imaging device is directly opposed; and an output image from the imaging device and an image for confirming surroundings can be displayed. There is provided image display means mounted on the user's head and capable of displaying an image in front of the user's eyes, and detecting an inclination from a state in which the image display means faces the same direction as the imaging apparatus. An image that includes a second inclination detection unit, and that when the relative value of the angle detected by the first inclination detection unit and the second inclination detection unit exceeds a threshold value, the surroundings can be confirmed on the image display unit. Is displayed.

  ADVANTAGE OF THE INVENTION According to the display apparatus which concerns on this invention, the output image control method of the head mounted display which can provide a safe use state to a user without obstructing the imaging operation of an imaging device can be provided.

FIG. 1 is a schematic view of an imaging device and an HMD as an example of an embodiment of the present invention. FIG. 1 is a schematic diagram illustrating a rotation axis of an imaging device and an HMD that are examples of an embodiment of the present invention. FIG. 1 is a block diagram illustrating a configuration of a control system according to an embodiment of the present invention. It is the figure (the 1) which showed the typical use posture in the embodiment of the present invention. It is the figure (the 2) which showed the typical use posture in the embodiment of the present invention. 4 is a flowchart illustrating a control method according to the embodiment of the present invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The control system includes an imaging device 10 and an HMD (head mounted display) 100 as an example. The HMD 100 corresponds to an example of an image display unit according to the present invention. FIG. 1 is an external view of a state in which the HMD 100 is mounted and the imaging device 10 is held as viewed from the front. In the present embodiment, the imaging device 10 is, for example, a single-lens camera in which the lens unit 2 can be attached to and detached from the imaging device body 1. The HMD 100 is mounted on the user's head, and an image can be displayed in front of the eyes by a display unit 102 provided inside.

  The display unit 102 may be a transmissive display or an immersive (non-transmissive) display. The transmissive display displays an image superimposed on the user's field of view. An immersive display displays images in an immersive environment where the user's view is sufficiently obstructed. In the present embodiment, the display unit 102 is an immersive display. By displaying an image of the peripheral scene photographing imaging device 101 separately provided on the front of the HMD, it is possible to confirm the peripheral scene like a transmissive display. is there.

  FIG. 2 is a diagram illustrating axes of the HMD 100 and the imaging device 10 in the tilt direction. The HMD 100 and the imaging device 10 can be tilted around three axes of a pitch axis, a roll axis, and a yaw axis, respectively. The HMD 100 is defined as a pitch angle, a roll angle, and a yaw angle when each of the HMDs 100 is tilted around each axis with the upright facing state as shown in FIG.

  The image capturing apparatus 10 sets the angle of inclination about each axis as a pitch angle, a roll angle, and a yaw angle, with a state where the imaging direction is set in front of the field of view and placed on a horizontal plane as a reference 0 degree. FIG. 2 shows a state where the camera is set in a state where it is slightly tilted in the pitch direction and the yaw direction.

  FIG. 3 is a block diagram illustrating a configuration of the control system according to the present embodiment.

  The imaging device 10 includes a system control unit 13 that controls the entire imaging device 10. The system control unit 13 is connected to the image processing unit 4, the recording medium 5, the operation unit 6, the inclination detection sensor 14, the rear display 12, the memory 15, and the communication unit 16, and controls the behavior of each unit. The image pickup device 3 photoelectrically converts a subject image incident through the lens unit 2 and includes a CCD, a CMOS device, and the like. The lens unit 2 is detachable, and can communicate lens information with the system control unit 13 via a lens communication contact (not shown).

  The image processing unit 4 performs all image processing by hardware, such as gamma / knee processing, filter processing, and information synthesis processing for monitor display, on digitized image data. The image processing section 4 can also perform image data compression processing such as JPEG.

  As the recording medium 5, a flash memory or the like that is detachable from the main body is used, and records photographed image data, photographing information, and the like.

  The operation unit 6 includes various operation members such as a shooting switch and a touch panel unit, and transmits various operations of the user to the system control unit 13.

  The tilt detection sensor 14 as a tilt detection unit detects a change in tilt of the imaging device 10 in each axis illustrated in FIG. The type and quantity of the inclination detection sensor 14 are not limited as long as the inclination detection sensor 14 is a device that can detect a change in inclination of the imaging device 10 such as an acceleration sensor or a gyro sensor.

  The rear display 12 is composed of, for example, a TFT liquid crystal panel having a size of about 3.0 inches, and is used for displaying settings of the imaging device 10 and displaying captured images.

  The memory 15 is provided in the imaging device main body 1 and stores a control program and the like. The system control unit 13 executes a predetermined process according to a control program stored in the memory 15.

  The communication unit 16 is a wired or wireless communication circuit that exchanges data with the communication unit 106 of the HMD 100 under the control of the system control unit 13, and is a wireless communication circuit in the present embodiment.

  The HMD 100 includes a system control unit 103 that controls the entire HMD 100. The system control unit 103 is connected to the peripheral scene photographing imaging device 101, the tilt detection sensor 104, the display unit 102, the memory 105, and the communication unit 106, and controls the behavior of each unit.

  The tilt detection sensor 104 as a tilt detection unit detects a change in tilt of the HMD 100 in each axis illustrated in FIG. The type and quantity of the inclination detecting sensor 104 are not limited as long as the inclination detecting sensor 104 can detect a change in inclination of the HMD 100 such as an acceleration sensor or a gyro sensor.

  The memory 105 is provided in the HMD 100 and stores a control program and the like. The system control unit 103 executes a predetermined process according to a control program stored in the memory 105.

  The communication unit 106 is a wired or wireless communication circuit that exchanges data with the communication unit 16 of the imaging device 10 under the control of the system control unit 103. In the present embodiment, the communication unit 106 is a wireless communication circuit.

  The display unit 102 is provided inside the HMD 100 as described above, and displays an image in front of the user's eyes. The system control unit 103 of the HMD 100 receives the data of the communication unit 106, determines an output image to the display unit 102, and performs display control.

  4A and 4B are examples of typical use postures using the imaging device 10 and the HMD 100 in the present embodiment. FIG. 4A illustrates a state in which the imaging device 10 is held at a normal position in the line of sight, and is in a usage posture substantially equal to a posture in which the imaging device 10 is used alone to take a picture while checking the rear display 12. In the present embodiment, this shooting posture is referred to as “basic shooting posture”. At this time, the relative values of the pitch angle, the roll angle, and the yaw angle of the imaging device 10 and the HMD 100 are all substantially 0 degrees. In this basic photographing posture, photographing work in a concentrated environment becomes possible by displaying a photographed image of the imaging device 10 on the display unit 102 of the HMD 100. At this time, it is preferable that the rear display 12 of the imaging device 10 is not displayed from the viewpoint of energy saving.

  4B is a posture in which the arm holding the imaging device 10 is slightly lowered from the basic photographing posture of FIG. 4A and the imaging device 10 is tilted in the pitch direction. This posture is a posture often used when confirming the setting of the imaging device 10 on the rear display 12 or the like, and the relative value of the pitch angle between the imaging device 10 and the HMD 100 at this time is, for example, about 40 degrees. In this embodiment, this photographing posture is referred to as a “setting confirmation posture”, and the relative value of the pitch angle at this time is referred to as a “first threshold”.

  In this setting confirmation posture, the setting of the imaging device 10 can be easily confirmed on the rear display 12 or the like by displaying the image of the imaging device 101 for photographing the peripheral scene on the display unit 102 of the HMD 100. Therefore, it is preferable that a screen on which the settings of the imaging device 10 can be confirmed is displayed on the rear display 12 of the imaging device 10.

  4C illustrates a posture in which the arm holding the imaging device 10 is further lowered from the setting confirmation posture of FIG. 4B and the imaging device 10 is tilted in the pitch direction. This posture is mainly a posture at the time of confirming a peripheral scene or moving, and is in a state where there is no intention to shoot or to set the imaging device.

  The relative value of the pitch angle between the imaging device 10 and the HMD 100 in this state is larger than the setting confirmation posture of FIG. In the present embodiment, this photographing posture is referred to as a “peripheral scene confirmation posture”, and the relative value of the pitch angle at this time is referred to as a “second threshold”.

  In this peripheral scene confirming posture, displaying the image of the peripheral scene photographing imaging device 101 on the display unit 102 of the HMD 100 enables the peripheral scene to be confirmed and the photographing place to be moved safely. At this time, it is preferable that the rear display 12 of the imaging device 10 is not displayed from the viewpoint of energy saving.

  FIG. 4D illustrates a state in which the imaging apparatus 10 is tilted by about 90 degrees in the roll direction until the right hand comes up from the basic shooting posture of FIG. Is the posture when performing.

  In the present embodiment, this shooting posture is referred to as “vertical position shooting posture”. In FIG. 4D, a state in which the right hand is held up is taken as an example, but a posture in which the left hand is held up equally is also referred to as a “vertical position shooting posture”. The relative value of the roll angle between the imaging device 10 and the HMD 100 in the vertical shooting orientation is about 90 degrees. Further, the relative value of the pitch angle between the imaging device 10 and the HMD 100 in the vertical shooting posture is almost 0 degrees.

  In this vertical shooting posture, by displaying the image captured by the imaging device 10 on the display unit 102 of the HMD 100, it is possible to perform a shooting operation in a concentrated environment. At this time, it is preferable that the rear display 12 of the imaging device 10 is not displayed from the viewpoint of energy saving.

  The yaw axis of the imaging device 10 in the vertical shooting posture substantially matches the pitch axis of the basic shooting posture, and the pitch axis of the imaging device 10 in the vertical shooting posture substantially matches the yaw axis of the basic shooting posture. Therefore, the setting confirmation posture and the peripheral scene confirmation posture in the vertical position state are determined in the same manner as (b) of FIG. 4A and (c) of FIG. 4B by replacing the yaw axis of the imaging device 10 with the pitch axis. It is possible to do.

  Next, the image display control method according to the present embodiment will be described in more detail with reference to the flowchart of FIG.

  First, in steps S1 to S3, the state of the photographing device 10 is checked.

  In step S1, it is checked whether the power of the imaging device 10 is on. If the power is ON, the process proceeds to step S2. If the power is OFF, the process proceeds to step S16, and the image of the peripheral scene photographing imaging device 101 is displayed on the HMD 100.

  In step S2, it is confirmed whether or not the lens unit 2 is mounted on the imaging device main body 1. If the lens unit 2 is mounted, the process proceeds to step S3. If the lens unit 2 is not mounted, the process proceeds to step S16, and the image of the peripheral scene photographing imaging device 101 is displayed on the HMD 100.

  The state where the power of the imaging apparatus is turned off or the state where the lens unit 2 is not attached is a state where the imaging apparatus 10 is not ready for imaging. Therefore, by controlling as described above, the surrounding scene can be confirmed. Preparation for shooting can be easily performed.

  In step S3, the roll angle of the imaging device 10 is confirmed by the inclination detection sensor 14 of the imaging device 10, and it is determined whether or not the camera is in the vertical position shooting state. In this embodiment, the case where the absolute value of the roll angle becomes 80 to 110 degrees is determined to be the vertical position shooting state. If the camera is in the vertical position shooting state, the process proceeds to step S4. If the camera is not in the vertical position shooting state, the process proceeds to step S5.

  As described above, since the yaw axis of the imaging device 10 in the vertical shooting posture substantially coincides with the pitch axis of the basic shooting posture, the pitch angle and the yaw angle obtained by the tilt detection sensor 14 of the imaging device 10 are switched in step S4. The control is performed, and the process proceeds to step S5.

  Next, in steps S5 to S6, the inclination relative value of the pitch angle between the imaging device 10 and the HMD 100 is confirmed.

  In step S5, it is checked whether the relative value of the pitch angle inclination exceeds the first threshold value. The first threshold is a threshold for judging whether there is a photographing intention. If this value is set too small, photographing cannot be performed if the posture is slightly different from the basic photographing posture. In the present embodiment, the angle is set to 40 degrees as an example in order to enable photographing in a posture with a high degree of freedom.

  If the inclination relative value of the pitch angle exceeds the first threshold, the process proceeds to step S6. If the inclination relative value of the pitch angle does not exceed the first threshold, the process proceeds to step S13, and it is determined that the camera is in the basic shooting posture. If it is determined that the photographing posture is the basic photographing posture, control is performed to display the image of the imaging device 10 on the HMD 100 in step 14, and the image on the rear display 12 of the imaging device 10 is controlled to be in a non-display state in step 15.

  In step S6, it is checked whether the relative value of the pitch angle inclination exceeds the second threshold value. The second threshold value is a threshold value for determining whether there is an intention to confirm or change the setting of the imaging device 10, and is set to 80 degrees as an example in the present embodiment.

  If the inclination relative value of the pitch angle exceeds the second threshold value, the process shifts to step S7 to determine the surrounding scene confirmation posture. If it is determined that the surrounding scene is in the posture, the control of displaying the image of the surrounding scene photographing imaging device 101 on the HMD 100 is performed in step 8, and the image of the rear display 12 of the imaging device 10 is set to the non-display state in step 9. Control.

  If the inclination relative value of the pitch angle does not exceed the second threshold value, the process shifts to step S10 to determine the setting confirmation posture. If it is determined that the posture is the setting confirmation posture, control is performed to display the image of the peripheral scene photographing imaging device 101 on the HMD 100 in step 11, and the setting of the imaging device 10 is confirmed on the rear display 12 of the imaging device 10 in step 12. Control to display a possible screen.

  In the image display control method according to the present embodiment, the relative value of the yaw angle between the imaging device 10 and the HMD 100 is not specified, but the relative value of the yaw angle can be used for image display control. However, since the tilt in the yaw direction has a small causal relationship with the setting confirmation of the imaging device 10, in order to allow photographing in a more free posture, the peripheral scene confirmation posture is determined by setting a large threshold value of, for example, about 90 degrees. It is preferable to use only for

  Further, in the present embodiment, a two-step threshold is used for the pitch relative value of the inclination of the pitch angle, and fine display control can be performed in accordance with the situation.However, if it is desired to prioritize only shooting in a free posture, The threshold value can be set to one step.

  For example, if the first threshold value of the pitch angle relative value used in step S5 is set to be larger than the second threshold value used in step S6, as a result, the determination of the intermediate posture of the setting confirmation posture does not occur, and the threshold is set to one step. Can be controlled.

  By performing the image display control as described above, the photographing operation when it is determined that there is a photographing intention is not hindered, and the surroundings can be confirmed during photographing preparation, setting confirmation, moving, etc. Each work can be performed safely.

  The preferred embodiments of the present invention have been described above. However, the present invention is not limited to these embodiments, and should be appropriately modified and improved within the scope of the technical idea of the gist. For example, the digital camera described as the imaging device in the embodiment described above can be applied to a still camera or a video camera. In addition, the material, shape, size, number, arrangement location, and the like of each component in the above-described embodiment are arbitrary and are not limited as long as the present invention can be achieved.

2 lens unit, 3 image sensor, 4 image processing unit, 5 recording medium, 6 operation unit,
10 imaging device, 12 rear display, 13: system control unit,
14 tilt detection means (tilt detection sensor), 15 memory, 16 communication unit,
100 image display means (head mounted display),
101 image capture device for surrounding scene shooting, 102 display unit,
104 tilt detection means (tilt detection sensor),
105 memory, 106 communication unit

Claims (10)

Imaging means for imaging a subject to obtain an image;
First inclination detection means for detecting an inclination from a state in which the imaging means is directly facing,
Display means capable of displaying an image obtained from the imaging means and information about the periphery of the imaging means, and being mounted so as to be able to display the image in front of the user's eyes;
A second tilt detection unit that detects a tilt from a state in which the display unit faces the same direction as the imaging unit,
A first display control for controlling to display information about the periphery on the display means when a relative value of the angle detected by the first inclination detection means and the second inclination detection means exceeds a threshold value; Means,
A display device comprising: Having second display means,
A second display control means for controlling the output image of the second display means by comparing the relative value of the angle detected by the first inclination detection means and the second inclination detection means,
The display device according to claim 1, further comprising:   Wherein the first control means and the second control means compare the relative values of the angles detected by the first inclination detection means and the second inclination detection means with a two-stage threshold value The display device according to claim 1.   The first control means and the second control means perform comparison with the relative value of the angle in the pitch direction detected by the first inclination detection means and the second inclination detection means. The display device according to claim 1.   The first control means and the second control means perform a comparison based on the relative value of the angle in the yaw direction detected by the first inclination detection means and the second inclination detection means. The display device according to claim 1.   The first control means and the second control means, when the absolute value of the angle in the roll direction detected by the first tilt detection means exceeds a threshold, the first tilt detection means used for comparison The display device according to claim 1, wherein the detection angle in the pitch direction and the detection angle in the yaw direction are interchanged.   The said 1st display means performs control which displays the information regarding the said periphery on the said display means, when the lens unit is not mounted | worn with the said imaging means, The one of Claim 1 thru | or 6 characterized by the above-mentioned. The display device according to claim 1.   The display device according to claim 1, wherein the display unit is a non-transmissive head mounted display. A control method for a display device, comprising: a display unit that can display an image acquired from the imaging unit and information about a periphery of the imaging unit, and has a display unit attached so as to display the image in front of a user's eyes,
Capturing an image of a subject to obtain an image;
Detecting an inclination from a state in which the imaging unit is directly facing;
A step of detecting an inclination from a state in which the display unit is directly opposed to the imaging unit in the same direction,
When the relative value of the angle detected by the first inclination detection means and the second inclination detection means exceeds a threshold, displaying information about the periphery on the display means,
A control method for a display device, comprising:   The method according to claim 9, wherein the display device is a non-transmissive head-mounted display.