WO2019031593A1

WO2019031593A1 - Video and sound reproduction device and method

Info

Publication number: WO2019031593A1
Application number: PCT/JP2018/029984
Authority: WO
Inventors: 修一田坂
Original assignee: イアフレド株式会社
Priority date: 2017-08-09
Filing date: 2018-08-09
Publication date: 2019-02-14
Also published as: JP2019033426A; JP6613429B2

Abstract

Provided is a video and sound reproduction device with which it is possible to view and listen to content in both VR and AR. This video and sound reproduction device comprises a reproduction circuit unit capable of reproducing virtual-reality content and augmented-reality content, a control circuit unit that selects either the virtual-reality content or the augmented-reality content and causes the reproduction circuit unit to reproduce the selected content, a first display unit that displays an image of the virtual-reality content, a second display unit that displays an image of the augmented-reality content, and a head-mounted mechanism in which the first display unit and the second display unit can be attached/removed and replaced.

Description

Audiovisual reproduction apparatus and method thereof

The present invention relates to a video and audio reproduction apparatus, and more particularly to a video and audio reproduction apparatus using a head mounted display.

Virtual Reality (hereinafter referred to as "VR") refers to a goggle type display called a head mounted display (HMD) that displays 360-degree images, a goggle type display, an eyepiece, and 3D sound technology to play games, travel, sports or This is a technology for reproducing contents such as a movie as if it were a reality, and it is possible to obtain a drastic sense of immersion compared to a conventional flat television picture.

In addition, Augmented Reality (hereinafter referred to as "AR") is a technology that combines the real world and virtual reality, and since it is a content that can be used outdoors as, for example, portable game software, it is better than VR. It is expected to be a big market. AR is also called Mixed Reality, and it is created by real images taken with a camera and computer graphics by placing an environment recognition camera and a space recognition camera in the head mount display housing. A holographic optical technology is also put to practical use, which combines an image with a video image in real time and displays the image on a microdisplay again.

In WO 2014/115393 “Image display device and image display method, and image display system” published on July 31, 2014, the image display device is a light shield type used for VR and a transmission type used for AR You can understand the outline of HMD.

An example of the details of a VR can be referred to in its entirety in US Patent Application Publication No. 2016/0361658, "Expanded Field of View re-rendering for VR Spectating," published Dec. 15, 2016. According to the present invention, when a user wearing an HMD executes a video game, the camera's motion is detected by the camera so that the video and audio latency does not occur due to the motion, and a video server on the network is newly added. Is used to generate images with a wider field of view (FOV).

Also, an example of the details of the AR can be referred to in its entirety in US Patent Application Publication No. 2014/0168261 “Direct interaction system mixed reality environments” filed on Dec. 13, 2012. In accordance with the present invention, a configuration of a system for interacting with virtual objects in a mixed reality environment tracks a three dimensional scene map generated by a head mounted display device (HMD) including a camera capturing a user's field of view; The mobile processing unit can display virtual objects in the real world within the user's field of view (FOV). The transparency of the HMD provides the ability to project a virtual image within the user's field of view so that the virtual image is displayed alongside the real object. It also automatically tracks where the user is looking and allows the system to determine where to insert the virtual image into the user's FOV. Once the system knows where to project the virtual image, the display element is used to project the image.

International Publication No. 2014/115393 US Patent Application Publication No. 2016/0361658 U.S. Patent Application Publication No. 2014/0168261

However, since the view of a human being is blocked by the display in order to enjoy 360-degree video, VR is mainly used indoors from the viewpoint of safety. On the other hand, AR uses a transmissive display and is safe, but it is unsuitable for viewing dynamic images of 360 degrees because of limited viewing angle. The reason is that VR is required to have a high immersive feeling of FOV over 100 degrees, while AR is used for PC applications such as character recognition even if FOV is about 50 degrees and the field of view is narrow, in order to combine virtual reality in the real world High resolution is required because there are many cases. The number of pixels per degree is used as a criterion for resolution, but in the case of VR, the viewing angle is about 110 degrees, so the resolution in the horizontal direction is about 20 pixels / degree (2160 pixels) when using 2K organic EL Since the viewing angle is about 50 degrees in the case of AR or MR, the resolution in the horizontal direction is about 43 pixels / degree (2160 pixels / 50 degrees = 43.2) in the case of 2K. , It can be seen that the resolution of AR is high. That is, due to different applications, the optical specifications and configuration of the display used will differ between VR and AR.

At this stage, it is necessary to improve the resolution of the image in order to get closer to the real world, but high-resolution organic EL displays (for example, 8K and 12K panels) have no mass productivity and have a high manufacturing cost. In addition, when the head movement of the user is detected and the video is reproduced on the high resolution panel, when the video time delay (Latency) occurs, a phenomenon called VR sickness occurs, which hinders viewing for a long time. Therefore, high performance signal processing capability is required to realize a high frame rate of 90 Hz or more, but a large problem arises for the portable terminal that the load on the signal processing semiconductor is large and the power consumption is also increased.

In addition, since VR and AR are different in the content to be reproduced, it is burdensome for the user to purchase two VR head mounted display devices with a view angle priority and two AR head mounted display devices with a resolution priority. In addition, composite displays that use both VR and AR technology have been studied, but they have not been put to practical use yet. Such a composite display, even if realized, is likely to be heavy, and fatigue due to prolonged wearing is also conceivable.

An object of the present invention is to provide a video and audio reproduction apparatus capable of viewing and listening to both VR and AR contents.

In order to achieve the above object, the video and audio reproduction apparatus of the present invention is
A reproduction circuit unit capable of reproducing virtual reality content and augmented reality content;
A control circuit unit that selects one of the virtual reality content and the augmented reality content and causes the reproduction circuit unit to reproduce the selected content;
A first display unit for displaying an image of virtual reality content;
A second display unit for displaying an image of augmented reality content;
And a head mount mechanism capable of attaching and detaching the first display unit and the second display unit.

According to the configuration of the present invention, it is possible to provide a video and audio reproduction apparatus capable of viewing and listening to both VR and AR contents.

Conceptual diagram of a head mounted display (HMD) according to the first embodiment Side view of the appearance configuration of the HMD when playing back VR content according to the first embodiment Side view of the appearance configuration of the HMD when playing back AR content according to an embodiment Block diagram showing functional components of the HMD of the first embodiment A schematic view showing an example of a mechanism for attaching and detaching an AR unit to and from an HMD device Side view of HMD device of second embodiment Block diagram of the second embodiment Conceptual diagram when detecting the position of the earphone unit A graph showing the relationship between the distance between the beacon terminal incorporated in the earphone and the signal detection unit and the radio wave strength RSSI A schematic diagram showing an example in which the distance to the beacon signal source is displayed in text on the display A schematic diagram showing an example in which the direction of the beacon signal source is graphically displayed on the display Side view of HMD of third embodiment Block diagram of the third embodiment

According to a first aspect of the present invention, there is provided an audiovisual reproduction apparatus including:
A reproduction circuit unit capable of reproducing virtual reality content and augmented reality content;
A control circuit unit that selects one of the virtual reality content and the augmented reality content and causes the reproduction circuit unit to reproduce the selected content;
A first display unit for displaying an image of virtual reality content;
A second display unit for displaying an image of augmented reality content;
And a head mount mechanism capable of attaching and detaching the first display unit and the second display unit.

According to this configuration, the first display unit displaying the image of the virtual reality content and the second display unit displaying the image of the augmented reality content are exchanged according to the content to be reproduced. It is possible to view and listen to both VR and AR content on one device.

In the first configuration, the video and audio reproduction apparatus according to the second configuration further includes:
It further comprises detection means for detecting which of the first display unit and the second display unit is attached to the head mount mechanism.
When it is detected that the first display unit is attached, the control circuit unit causes the first display unit to display a selection screen of virtual reality content reproducible by the reproduction circuit unit.
When it is detected that the second display unit is attached, the control circuit unit causes the second display unit to display a selection screen of the augmented reality content reproducible by the reproduction circuit unit.

According to this configuration, the detection means automatically detects which of the first display unit and the second display unit is attached to the head mount mechanism, and according to the attached display unit, A screen for selecting VR content and AR content can be displayed. This saves the user the trouble of selecting which one of the VR content and the AR content to be reproduced.

In the first or second configuration, the video and audio reproduction apparatus according to the third configuration further includes:
It further comprises an acoustic reproduction device that can be deployed near at least one of the left and right pinna or in the ear canal,
The control circuit unit is provided in the sound reproduction device.

According to this configuration, the head mounting mechanism can be reduced in weight by providing the control circuit unit in the sound reproducing apparatus. As a result, it is possible to provide a video and audio reproduction apparatus with less fatigue even in long-term viewing.

The video and audio reproduction apparatus according to the fourth configuration further includes, in the third configuration:
The sound reproduction apparatus includes an input device for inputting an operation instruction to the control circuit unit.

According to this configuration, the input device is provided to the sound reproducing apparatus capable of being disposed in the vicinity of at least one of the left and right auricles or in the external ear canal, for example, to operate by holding a separate remote controller etc. Compared to the case, the freedom of both hands is improved.

In the third or fourth configuration, the video and audio reproduction apparatus according to the fifth configuration further includes:
The sound reproducing apparatus further includes a position transmitting unit that transmits a position signal indicating the current position of the sound reproducing apparatus.
The video and audio reproduction apparatus further includes a position detection unit that detects a position signal transmitted from the position transmission unit.
The control circuit unit causes the first display unit or the second display unit to display the position of the sound reproduction device based on the position signal detected by the position detection unit.

According to this configuration, when the sound reproduction device is lost, the sound reproduction device can be detected based on the position signal transmitted from the sound reproduction device, and the position can be displayed on the display unit. Thereby, even if the sound reproducing device is lost, it can be easily found out.

A video and audio reproduction apparatus according to a sixth configuration is the same as the fifth configuration.
The position transmitting unit is configured to transmit a beacon signal as the position signal.

The video and audio reproduction apparatus according to the seventh configuration is the same as the video and audio playback apparatus according to the sixth configuration.
The position detection unit is provided at both left and right sides of the video and audio reproduction apparatus.

In the seventh configuration, the video and audio reproduction device according to the eighth configuration further includes:
The distance to the sound reproduction device detected by the position detection unit provided on the left side of the video and audio reproduction device, and the sound reproduction device detected by the position detection unit provided on the right side of the video and audio reproduction device The control circuit detects the position of the sound reproducing device using the difference between

According to this configuration, based on the difference between the distance to the sound reproduction device detected on the left side of the video and audio reproduction device and the distance to the sound reproduction device detected on the right side of the video and audio reproduction device The direction of the sound reproduction device and the distance to the direction can be obtained.

A video and audio reproduction apparatus according to a ninth configuration is any one of the sixth to eighth configurations:
The control circuit unit causes the first display unit or the second display unit to display the distance to the position transmitting unit, which is a source of the position signal, as text.

As described above, by displaying the position of the sound reproducing device as text, it is possible to easily find the sound reproducing device.

A video and audio reproduction apparatus according to a tenth configuration is any one of the sixth to ninth configurations:
The control circuit unit causes the first display unit or the second display unit to graphically display the direction of the position transmitting unit which is a source of the position signal.

An image and sound reproducing apparatus according to an eleventh configuration is any one of the first to tenth configurations:
It further comprises a first microphone capable of detecting an external noise signal, and a second microphone capable of detecting an external audio signal.

Furthermore, in the eleventh configuration, the video and audio reproduction apparatus according to the twelfth configuration further includes:
When the reproduction circuit unit reproduces virtual reality content, the first microphone operates preferentially.
When the reproduction circuit unit reproduces augmented reality content, the second microphone operates preferentially.

According to this configuration, when the VR content is being reproduced, an external noise signal is detected by the first microphone and, for example, noise cancellation processing can be performed to create a situation in which the VR content is likely to be absorbed. Also, when playing back AR content, it is possible to capture the sound of the real world by detecting the external audio signal with the second microphone.

The sound reproducing apparatus according to the thirteenth configuration of the present invention is
An audio reproduction apparatus which is used together with an audiovisual reproduction apparatus having reproduction circuitry capable of reproducing virtual reality content and augmented reality content, and which can be deployed in the vicinity of at least one of the left or right pinna or the external ear canal,
The control circuit unit is configured to select one of the virtual reality content and the augmented reality content and cause the reproduction circuit unit to reproduce the selected content.

In this configuration, the reproduction circuit unit is provided in the sound reproduction device which can be disposed in the vicinity of at least one of the left and right auricles or the ear canal. Thus, the audio and video reproduction apparatus can be controlled by the audio reproduction apparatus.

The sound reproducing apparatus according to the fourteenth configuration is the thirteenth configuration,
A control circuit and a sensor used to operate the video and audio reproduction apparatus are provided.

According to this configuration, by providing the control circuit unit and the sensor in the sound reproduction device, the weight of the head mount mechanism can be reduced. As a result, it is possible to provide a video and audio reproduction apparatus with less fatigue even in long-term viewing.

Hereinafter, more specific embodiments of the present invention will be described with reference to the drawings.

First Embodiment
FIG. 1 is a conceptual view of a head mounted display (HMD) according to an embodiment of the present invention, which shows that an AR unit and a VR unit are interchangeable. FIG. 2 is a side view of the HMD apparatus when the VR unit of the first embodiment is mounted. FIG. 3 is a side view of the HMD apparatus when the AR unit of the first embodiment is attached. FIG. 4 is a block diagram showing functional components of the HMD of the first embodiment.

FIG. 1 shows the concept of the HMD of this embodiment. The VR unit 11 and the AR unit 12 are replaceable, and are joined by the AR / VR common unit 10 and unit engaging portions 14a and 14b. The VR unit 11 and the AR unit 12 mainly include an optical module, and the AR / VR common unit 10 includes a signal processing circuit, a battery, an attitude detection sensor, and the like. The unit engaging portions 14a and 14b are electrically coupled by a connector.

The user wears the VR unit 11 when viewing immersive content (VR content) such as movies, sports, concerts, competitive games, and the like. In addition, when viewing AR content for collaborative work with a plurality of people such as product development in which a virtual image is three-dimensionally displayed or in-house education, or AR content such as a navigation type game, AR unit 12 Replace and wear. The VR unit 11 and the AR unit 12 are configured to be lightweight, so they can be carried as a kit on the go.

In the configuration example of FIG. 1, the AR image capture camera 107 a is provided in the AR unit 12, and the VR image capture camera 107 b is provided in the VR unit 11. This is because the image recognition function and the number of cameras differ between AR and VR. However, there is no problem if the image capturing camera is provided not in the VR unit 11 and the AR unit 12 but in the AR / VR common unit 10.

The AR unit 12 is provided with a depth camera unit 108. The depth camera unit 108 is formed by a CCD sensor or a CMOS sensor capable of capturing depth information. The provision of the depth camera unit 108 enables three-dimensional arrangement of virtual images in a real environment, and can also recognize gestures on the three-dimensional space of the user, thus having a controller function as well. it can.

FIG. 2 shows a configuration example for attaching and detaching the VR unit 11 and the AR unit 12 to and from the AR / VR common unit 10. Although FIG. 2 exemplifies how the AR unit 12 is attached, the VR unit 11 can be configured in the same manner.

The unit engagement portion 14a on the AR unit 12 side includes a right side engagement portion 14aR having an engagement hinge portion 18 and a left side engagement portion 14aL having a connector portion 16a. The unit engagement portion 14b on the AR / VR common unit 10 side is a right engagement portion 14bR engaged with the right engagement portion 14aR of the AR unit 12, and a left engagement engaged with the left engagement portion 14aL of the AR unit 12. And a joint portion 14bL. The right side engaging portion 14bR has a guide space 19 for receiving the engaging hinge portion 18. The left side engaging portion 14bL includes a connector portion 16b connected to the connector portion 16a, and a magnet adsorbing portion 15 which adsorbs the left side engaging portion 14aL with a magnetic force. With this configuration, when the engagement hinge portion 18 is inserted into the guide space 19 and the left engagement portion 14aL is engaged with the left engagement portion 14bL, the magnet adsorption portion 15 fixes the engaged state. There is. In this example, the connector portion and the magnet adsorbing portion are provided only in the left engaging portion, but the connector portion and the magnet adsorbing portion may be provided in the right engaging portion as well. Further, the method of engaging the VR unit 11 and the AR unit 12 with the AR / VR common unit 10 is not limited to the example shown in FIG. In addition, the method of fixing using elastic force of plastic, the method of fastening both with a screw etc., etc. can be considered.

FIG. 3 shows a side view of the appearance of the head mounted display when playing back VR content. Although FIG. 3 shows only the left side surface, the right side surface also has the same configuration. The head mounted display is used by a user wearing on the head. The VR unit 11 is configured such that light from the outside of the display does not enter the left and right eyes, so the display unit with a wide viewing angle improves the user's sense of immersion. The user can not directly view the real landscape, but can indirectly view the external image captured by the camera.

The AR / VR common unit 10 includes a circuit board 5 on which electronic circuits such as signal processing are mounted, an HMD posture detection unit 102, and a video battery 106. The AR / VR common unit 10 can be stabilized by being supported near both ears by a frame structure such as glasses in order to be fixed to the head of the user. The circuit board 5 includes a video signal processing unit described later with reference to FIG. 5, a communication unit with the outside, an AR / VR switching unit, a memory, and the like. VR video content receives video and audio signals from the Internet via wireless LAN (IEEE 802.11a / b / g / n / ac), Bluetooth (registered trademark) 4.0, etc., and can be used as semiconductor memory, microSD card, etc. It is also possible to save. The HMD posture detection unit 102 constantly monitors the position state centering on the head of the user by, for example, a six-axis detection sensor (three-axis gyro, three-axis acceleration), a three-axis geomagnetic sensor, or the like.

The VR unit 11 includes a VR display unit 141, a lens unit 142, and an image capturing camera unit 107. The VR display unit 141 is configured of an organic EL or liquid crystal display, and in order to be closer to the real world, it is preferable to improve the resolution of the image to 4K or more. The VR display unit 141 has a structure in which the display device is separated to the left and right, like a lens of glasses, for example, when a 3.5-inch organic EL is composed of two surfaces, or 5.5-inch like a smartphone The left and right integrated structure using the organic EL is also possible.

The earphone unit 13 includes a speaker unit 120, a microphone A 121, a microphone B 122, an audio battery 123, and a controller touch panel unit 124. Although only one earphone unit 13 is illustrated in FIG. 3, both left and right earphone units 13 may have the same function. The speaker unit 120 can be inserted into the pinna and is connected to the AR / VR common unit 10 wirelessly. The microphone A 121 is for noise cancellation, and by picking up external noise to electrically generate a phase difference signal, there is an effect of enhancing the sense of immersion by effectively shielding the external sound. Therefore, an audio battery 123 for operating a circuit having a wireless transmission function and a noise cancellation function is incorporated in the earphone unit 13. The microphone B 122 picks up an instruction command with the user's voice, for example, video playback start, stop, fast-forward, for the voice recognition function. Thereby, the operation is possible without using the remote control. When reproducing the VR content 11, it is also possible to detect external noise and activate a noise canceling function.

Also, the controller touch panel unit 124 performs a touch operation on the touch panel with a finger to execute a cursor or command of a graphic layer that appears on the VR reproduction screen of the VR display unit 141 as a GUI (graphic user interface). , Easy to operate. When operating with the remote control in both hands as seen in VR game machines etc., there is no problem if it is watching for a short time, but the remote control is heavy and tired, there is a risk of falling, there is a danger of colliding with furniture For example, the user is not necessarily satisfied. Since the earphone unit 13 can be controlled in the vicinity of the pinna by the controller touch panel unit 124, the user can control the screen with the minimum movement. Further, since the earphone unit 13 is light in weight, it is suitable for viewing for a long time. If the touch panel portion 124 for controller does not have a flat surface but has, for example, a hemispherical shape, execution of a cursor or button of a GUI displayed overlapping an image of a 360-degree three-dimensional space can be easily viewed. Moreover, although FIG. 3 etc. illustrate the earphone unit 13 as a specific example of a sound reproduction apparatus, you may implement a sound reproduction apparatus as a headphone type which covers the whole ear. If the headphone type is used, since the controller touch panel portion can be formed relatively large, it becomes relatively easy to have the same function as described above.

FIG. 4 shows a side view of the appearance of the HMD when the AR unit 12 is attached. In the VR unit 11, since the image display units disposed in both the left and right eyes are of a transmissive type, it is possible to provide the user with a combined experience in which a virtual object image appears on an actual landscape. Similar to FIG. 3, the left side is displayed, but the right side has the same configuration.

The AR unit 12 includes an AR display unit 132, a reflective surface 133, a light guide plate 134, an opaque filter 135, a lens unit 136, and an image capturing camera unit 107.

The AR display unit 132 emits a virtual image source. The image is reflected by the reflective surface 133 and reaches the user's eye through the light guide plate 134. As the AR display unit 132, a high output type LED also called a micro display, an organic EL, a digital light processing (DLP) display, a liquid crystal on silicon (LCOS) in which liquid crystal and a driving circuit are integrated, or the like is used. Although the light guide plate 134 is substantially transparent, a diffraction grating such as a holographic film is provided on the surface, and the light from the AR display unit 132 is expanded to play a role as a screen of an image projector.

The opacity filter 135 removes natural light to improve the contrast of the image of the virtual object image. The opaque filter 135 is aligned with the light guide plate 134 to selectively transmit or block natural light from the front of the HMD and guide it to the eye for the entire screen or for each pixel. For example, when rendering the virtual image, by adjusting the degree of transparency of the opaque filter 135 according to the positional relationship between the virtual image and the real image, the image is displayed as if the virtual image actually exists. Can. The lens unit 136 is a standard transparent lens as commonly used in eyeglasses. The image capturing camera unit 107 has an environment recognition function.

The microphone B 122 can pick up an instruction instructed by the user as a voice input function with a microphone. The microphone B 122 can also be used to actively capture the sound of the external environment in order to be able to hear external sounds even when the earphone unit 13 is inserted in the pinna. In addition, when the reproduction of the AR content is started, the microphone B 122 may be automatically activated by detecting metadata of the AR content.

FIG. 5 is a block diagram showing functional components of the video and audio reproduction apparatus according to the present embodiment. The AR / VR common unit 10 includes a memory unit 100, a reproduction signal processing unit 101, an HMD posture detection sensor unit 102, a communication unit 104, an AR / VR switching unit 105, a video battery 106, and an image capturing camera unit 107. ing.

The VR unit 11 includes a VR reproduction I / F (interface) unit 113, a VR display drive unit 140, and a VR display unit 141. When playing back VR content, as shown in FIG. 1, by mounting the VR unit 11 to the AR / VR common unit 10, the user can view and experience the world of an immersive image.

The AR / VR switching unit 105 determines which of AR and VR the content to be reproduced is. Here, as a method for determining whether the content to be reproduced is AR or VR, for example, (1) detecting which one of the AR unit 12 or the VR unit 11 is attached to the AR / VR common unit 10 is detected (2) There is a method of detecting which reproduction of AR / VR content is selected by the user on the initial screen or the like. That is, in the case of the above (1), the VR unit 11 and the AR unit 12 are provided with a mechanism for generating a signal that can identify the unit, or the shapes of the unit engaging portions 14 in the VR unit 11 and the AR unit 12 are mutually different. It is conceivable to make it possible to detect which one of the VR unit 11 and the AR unit 12 is attached on the AR / VR common unit 10 side by making the difference.

When it is determined which of AR and VR the content to be reproduced is, the VR reproduction I / F unit 113 displays the thumbnails of the reproducible VR content on the VR display unit 141 to allow the VR content to be displayed to the user. Can be selected. When the VR unit 11 is used, it is possible to select and display and reproduce only the VR content among the contents stored in the memory unit 100.

The AR unit 12 includes an AR reproduction I / F unit 112, an AR display drive unit 130, an opacity filter control unit 131, and an AR display unit 132. When playing back AR content, as shown in FIG. 4, by attaching the AR unit 12 to the AR / VR common unit 10, a plurality of virtual images are seamlessly merged with the viewpoint of the user in the real world, It can provide the user with a mixed reality experience.

The user receives an image recorded in the memory unit 100 and an instruction on a virtual image obtained from the communication unit 104, and receives the user's head by the 3-axis gyro, 3-axis acceleration, and 3-axis geomagnetic sensor obtained from the HMD posture detection sensor unit 102. Position information centering on the unit is passed to the signal processing unit 101. The signal processing unit 101 determines when and where to provide a virtual image.

An image received from the image capture camera 107 is stored in the memory unit 100. The AR display drive unit 130 drives the AR display unit 132 (micro display). The information on the virtual image displayed on the AR display unit 132 is provided to the opacity filter control unit 131 that controls the opacity filter. At this time, a video is buffered in the AR reproduction I / F unit 112 to provide system timing data, and a timing circuit and a buffer memory circuit are configured.

When the AR / VR switching unit 105 identifies that the AR unit 12 is attached to the AR / VR common unit 10, or when the user selects the playback of AR content, the AR playback I / F unit 112 performs playback By displaying thumbnails of possible AR content on the AR display unit 132, the user can select AR content to be reproduced. That is, when the AR unit 12 is used, only the AR content stored in the memory unit 100 can be displayed and reproduced.

The earphone unit 13 includes an audio amplifier / DAC unit 110, an audio processing unit 111, an earphone speaker unit 120, a microphone A 121, a microphone B 122, an audio battery 123, and a controller touch panel unit 124. The earphone unit 13 can wirelessly communicate with the communication unit 104, but can also be connected via a wire cable.

A Bluetooth codec such as AAC or aptX transmitted from the communication unit 104 is decoded by the audio amplifier / DAC unit 110, and audio is output from the earphone speaker unit 120. When the voice processing unit 111 operates the noise cancellation function, the voice processing unit 111 picks up external noise from the microphone A 121 and generates a phase difference signal electrically. Further, the microphone B 122 picks up an instruction command with voice instructed by the user as a voice recognition function.

When the AR / VR switching unit 105 detects that the VR unit 11 is attached to the AR / VR common unit 10, or when the user selects playback of VR content, the microphone A121 automatically generates an external noise. It is possible to detect the noise and activate the noise canceling function. When the AR unit 12 is detected, it is possible to automatically activate the microphone B122.

Further, the control panel can be easily operated by performing a touch operation on the touch panel with a finger in order to execute a cursor or a command of a graphic layer floating on the VR reproduction screen as a GUI (graphic user interface).

Second Embodiment
FIG. 6 is a side view of the HMD apparatus when the AR unit of the second embodiment is mounted. FIG. 7 is a block diagram showing components of the HMD of the second embodiment.

The AR / VR common unit 10 is provided with an HMD posture detection sensor 102, an image capturing camera unit 107, a video battery 106, an AR / VR switching unit 105, and a memory unit 100a. The earphone unit 13 includes a circuit board 5 as shown in FIG. As shown in FIG. 7, the communication unit 104, the memory unit 100 b, and the reproduction signal processing unit 101 are mounted on the circuit board 5. The communication unit 104, the memory unit 100, and the reproduction signal processing unit 101 are circuits mounted in the AR / VR common unit 10 in the first embodiment, as shown in FIG.

Since the AR / VR unit 10 and the earphone unit 13 are connected wirelessly, the memory unit 100 a of the AR / VR common unit 10 is used as a buffer memory for video data for communication. The memory unit 100b of the earphone unit 13 is an external memory such as a microSD card storing video content.

At the time of reproduction of the AR content, the communication unit 104 of the earphone unit 13 downloads the AR video content received via the Internet or the like to the memory unit 100 b and stores the downloaded content. The downloaded content is decoded by the reproduction signal processing unit 101. The decoded video signal is transmitted to and stored in the memory unit 100 a of the AR / VR common unit 10 via the communication unit 104. The object video is displayed on the AR display unit 132 via the AR reproduction I / F unit 112 of the AR unit 12 in a state where a sufficient amount of data for reproducing the video is secured. The audio signal is reproduced by the earphone speaker unit 120 in synchronization with the video signal via the audio amplifier and the DAC unit 110.

Similarly, during reproduction of VR content, the information determined by the AR / VR switching unit 105 of the earphone unit 13 is transmitted from the reproduction signal processing unit 101 to the communication unit 104 and displayed on the VR display unit 141 of the VR unit 11 . Signals of the HMD posture detection sensor unit 102 and the image capture camera unit 107 are calculated by the reproduction signal processing unit 101 of the earphone unit 13 via the communication unit 104.

That is, in the present embodiment, by incorporating the circuit board 5 on which the signal processing circuit, the memory circuit, and the like are integrated into the earphone unit 13, the weight of the AR / VR common unit 10 can be reduced. As a result, there is no fatigue even when watching for a long time.

If the earphone unit 13 is lost, the communication unit 104 obtains positional information of the paired earphone unit 13 and three-dimensionally displays the position of the earphone unit on the display of the VR unit 11 or the AR unit 12 It is possible to investigate.

FIG. 8 is a conceptual diagram when detecting the position of the earphone unit 13. The earphone unit 13 includes a beacon terminal 150 (not shown in FIG. 7) in addition to the configuration shown in FIG. Further, the AR / VR common unit 10 is provided with two detection units 151 and 152 (not shown in FIG. 7) at positions corresponding to the left and right of the user's face. When the earphone unit 13 is lost, when the power of the earphone unit 13 is turned on, for example, using the GPS function, the location information transmitted by the communication unit 104 is identified via a smartphone or PC via the Internet You can also However, in many cases, the earphone may be lost or the power of the earphone may be turned off in places such as a room where the GPS function does not work. The beacon terminal 150 can be used for about one year even with a button battery, and by being incorporated in the earphone unit 13, detection at a short distance becomes possible. In the present embodiment, since two detection units 151 and 152 are arranged on the left and right of the AR / VR common unit 10, the difference between the signals detected by the detection units 151 and 152 is calculated. The position can be identified more accurately. In particular, by arranging the detection units 151 and 152 on the left and right of the user's head, one of the detection units 151 and 152 can obtain stronger signal strength because it is affected by radio interference from the user's head. .

FIG. 9 shows the relationship between the distance from the beacon terminal 150 built in the earphone unit 13 to the detection units 151 and 152 and the received signal strength indication (RSSI). For example, as shown in FIG. 9, when the detection value by the detection unit 152 on the right side is larger than the detection value by the detection unit 151 on the left side, it is known that the earphone unit 13 is on the right side of the head of the user. Also, from this state, when the user turns the head to the right and the detection value by the detection unit 151 and the detection value by the detection unit 152 become equal, it is determined that the beacon terminal 150 is in front of the user. be able to. Also, in this state, the distance to the beacon terminal 150 can be calculated based on the strength of the beacon signal.

FIG. 10 shows an example in which the distance and the direction to the earphone unit 13 incorporating the beacon terminal 150 are displayed in text on the display. Since the direction and distance are displayed, the user can find out the earphone unit 13 in a short time.

FIG. 11 shows an example in which the distance and the direction with the earphone unit 13 incorporating the beacon terminal 150 are graphically displayed on the display. The arrow indicates the direction of the earphone unit 13, and when approaching, the user can intuitively find the earphone unit 13 by displaying the position of the earphone unit 13 like a ripple.

The display modes shown in FIGS. 10 and 11 are merely examples, and any display mode can be taken.

Although the earphone unit 13 illustrated so far is a completely cordless wireless form, it is also conceivable to separate the circuit board 5 and the audio battery 123 from the earphone unit 13 and connect them with a cable. It is also conceivable to incorporate the HMD posture detection sensor 102 or the like into the earphone unit 13 by further integration. If the video battery 106 can also be integrated with the audio battery 123 of the earphone unit 13 if wireless power supply technology becomes possible, further weight reduction of the HMD device proceeds, and a comfortable viewing environment can be realized. realizable.

Third Embodiment
FIG. 12 is a side view showing the HMD device according to the third embodiment, in which the AR unit 12 is mounted. FIG. 13 is a block diagram showing a functional configuration of the third embodiment.

The earphone unit 13 of this embodiment is mechanically and electrically coupled to the VR unit 12 and the AR unit 11. Therefore, the video and audio batteries can be integrated and stored in the earphone unit 13 as the video and audio battery 125. That is, in the present embodiment, the video battery 106 and the audio battery 123 are omitted, and the HMD device can be further reduced in weight by mounting the video and audio batteries 125 in the earphone unit 13. Further, as in the second embodiment, a circuit board 5 on which a signal processing circuit, a memory circuit, and the like are integrated, and an HMD posture detection sensor unit 102 are also incorporated in the earphone unit 13. As a result, the weight of the HMD device can be reduced, and therefore fatigue does not occur even for a long time viewing. Also, since it is not wireless, there is little concern that the earphone unit 13 will be lost.

The left and right earphone units 13 include an HMD posture detection sensor 102, a battery 125, a main circuit board 5, a microphone A121, a microphone B122, a speaker unit 120, a controller touch panel unit 124, and the like. Even if the component configuration of the left and right earphone units 13 is asymmetrical, there is no problem if constructed considering the weight distribution on the left and right.

Further, since the earphone unit 13 is not a wireless earphone, unlike the above embodiment in which audio is received wirelessly such as Bluetooth, it is not necessary for the communication unit 104 to decode the audio signal.

The technology disclosed in the specification has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the scope of the technology disclosed herein.

For example, the input device is not limited to the controller touch panel unit 124 of the earphone unit 13. A microphone or an infrared sensor may be used as the input device.

Claims

A reproduction circuit unit capable of reproducing virtual reality content and augmented reality content;
A control circuit unit that selects one of the virtual reality content and the augmented reality content and causes the reproduction circuit unit to reproduce the selected content;
A first display unit for displaying an image of virtual reality content;
A second display unit for displaying an image of augmented reality content;
A video and audio reproduction apparatus comprising: a head mount mechanism capable of attaching and detaching the first display unit and the second display unit.
It further comprises detection means for detecting which of the first display unit and the second display unit is attached to the head mount mechanism.
When it is detected that the first display unit is attached, the control circuit unit causes the first display unit to display a selection screen of virtual reality content reproducible by the reproduction circuit unit.
When it is detected that the second display unit is attached, the control circuit unit causes the second display unit to display a selection screen of augmented reality content reproducible by the reproduction circuit unit. The video and audio reproduction apparatus according to 1.
It further comprises an acoustic reproduction device that can be deployed near at least one of the left and right pinna or in the ear canal,
The video and audio reproduction apparatus according to claim 1, wherein the control circuit unit is provided in the audio reproduction apparatus.
The video and audio reproduction apparatus according to claim 3, wherein the audio reproduction apparatus comprises an input device for inputting an operation instruction to the control circuit unit.
The sound reproducing apparatus further includes a position transmitting unit that transmits a position signal indicating the current position of the sound reproducing apparatus.
The video and audio reproduction apparatus further includes a position detection unit that detects a position signal transmitted from the position transmission unit.
The control circuit unit causes the first display unit or the second display unit to display the position of the sound reproduction device based on the position signal detected by the position detection unit. A video and audio reproduction apparatus according to any one of the above.
The video and audio reproduction device according to claim 5, wherein the position transmitting unit transmits a beacon signal as the position signal.
The video and audio reproduction apparatus according to claim 6, wherein the position detection unit is provided on both left and right sides of the video and audio reproduction apparatus.
The distance to the sound reproduction device detected by the position detection unit provided on the left side of the video and audio reproduction device, and the sound reproduction device detected by the position detection unit provided on the right side of the video and audio reproduction device The video and audio reproduction apparatus according to claim 7, wherein the control circuit unit detects the position of the audio reproduction apparatus using a difference between the distance and the distance.
The control circuit unit causes the first display unit or the second display unit to display the distance to the position transmitting unit, which is a source of the position signal, in the form of text. The audiovisual reproduction apparatus as described in a term.
The control circuit unit causes the first display unit or the second display unit to graphically display the direction of the position transmitting unit which is a source of the position signal. The audiovisual reproduction apparatus described in.
The video and audio reproduction device according to any one of claims 1 to 10, further comprising: a first microphone capable of detecting an external noise signal; and a second microphone capable of detecting an external audio signal.
When the reproduction circuit unit reproduces virtual reality content, the first microphone operates preferentially.
12. The video and audio reproduction apparatus according to claim 11, wherein the second microphone operates preferentially when the reproduction circuit unit reproduces augmented reality content.
An audio reproduction apparatus which is used together with an audiovisual reproduction apparatus having reproduction circuitry capable of reproducing virtual reality content and augmented reality content, and which can be deployed in the vicinity of at least one of the left or right pinna or the external ear canal,
A sound reproduction apparatus comprising: a control circuit unit which selects one of the virtual reality content and the augmented reality content and causes the reproduction circuit unit to reproduce the selected content.
The sound reproduction device according to claim 13, comprising a control circuit and a sensor used for operating the video and audio reproduction device.