WO2015145863A1

WO2015145863A1 - Display system, attachment, display method, and program

Info

Publication number: WO2015145863A1
Application number: PCT/JP2014/080393
Authority: WO
Inventors: 直敬藤井
Original assignee: 国立研究開発法人理化学研究所
Priority date: 2014-03-28
Filing date: 2014-11-17
Publication date: 2015-10-01
Also published as: JP6278490B2; JPWO2015145863A1

Abstract

The present invention provides an inexpensive and simple display system suitable for a head proximity-type video display system capable of giving a user an impression as though the information amount of the actual visual information space were expanded. A portable terminal (2) is accommodated in an attachment comprising a proximity unit (41) and a terminal accommodation unit (42), and a display panel (62) is magnified by a lens (111) to allow the user to visually confirm (C) the display panel (62). The portable terminal (2) detects, using a sensor of the portable terminal (2), an action (D) of the user that involves touching the attachment, and controls video display on the display panel (62) in accordance with the detected action.

Description

Display system, attachment, display method, and program

The present invention relates to a display system, an attachment, a display method, and a program. The present invention relates to a head proximity image display system that displays an image for a user who has placed a head proximity image display device close to the head, a head proximity body used in the head proximity image display system, and a head proximity image display Suitable for programs.

Virtual reality (VR) technology, which has been studied extensively, creates virtual worlds and remote spaces using various images and computer graphics (CG), and displays images by computers according to human actions. It is something to change. This makes it possible for viewers to feel as if they were there. However, even though a certain level of presence was obtained, it was not possible to make it feel as if it was in front of you. In order to solve such problems of the VR technology, a new technology called SR (Substitutional Reality System) system has been developed.

The SR system is a technology that keeps in mind that the virtual alternative world and the reality are replaced and the subject can experience the virtual and the reality without distinction. In order to realize this SR technology, it is so-called metacognition that is convinced that an event that occurs in reality is true, or that there is a suspicion when an inconsistent event occurs in reality. It is necessary to operate well the human higher cognitive function called. Conventionally, the SR system has not yet been put into practical use due to various technical limitations associated with this meta-cognition operation. In other words, in considering the practical application of this SR system, for example, virtual substitute events that have not occurred in front of the eyes, such as past images or images of different locations, can be seen in front of the eyes. A major issue is how to make viewers recognize that this is happening.

技術 Technologies that have been tried in the past to actually realize such an alternative reality state include “augmented reality”, “telepresence technology”, and “Google glass (registered trademark)”.

“Augmented reality” is a technology that superimposes a visual image based on CG (Computer Graphics) technology on an actual live image that can be seen from the user's viewpoint, and displays the image via a display device such as a head mounted display. This “augmented reality” is a technology that can newly add information that does not actually exist in reality. For example, disclosed techniques of Patent Documents 1 to 3 have been proposed as conventional techniques studied with respect to this “augmented reality”. However, this “augmented reality” has a problem in that since the visual image to be superimposed is a monolayer, the information of the real image displayed on the back of the CG superimposed thereon is deleted.

“Telepresence technology” is a real-time display of images captured by a camera provided at a remote location via a head-mounted display mounted on the viewer's head. At the same time, in this “telepresence technology”, the imaging direction of the camera is remotely controlled in conjunction with the movement of the viewer's head. As a result, it is possible to provide a sense of presence as if they are facing a remote member on the spot. However, this "telepresence technology" is intended to give viewers imaging information from a camera placed at a remote location, and it is a technology intended to add new information to the real space. Absent.

Recently, a glasses-type augmented reality wearable computer called “Google glass (registered trademark)” has been proposed. Google glass (registered trademark) is a concept of embedding various information in the real space visually recognized by viewers through glasses, so that information can be displayed hands-free and the Internet can be used with natural language voice commands. It has become. However, this Google glass (registered trademark) has a narrow information display area, so it must be exclusive when displaying multiple information, and when they are superimposed on each other, the information hidden underneath There is a problem that the viewer cannot be visually recognized. In addition, the effect of making the displayed image look real is not taken into consideration, and it does not exceed the use of a portable small monitor.

Japanese Patent Laid-Open No. 10-51711 JP 2000-82107 A JP 2006-48672 A

As described above, in the conventional “augmented reality”, “telepresence technology”, and “Google glass (registered trademark)”, a virtual substitute event that does not occur in front of the eyes actually occurs in front of the eyes. It does not lead to let the viewer recognize it as an event. That is, there are various meta-cognitive obstacles for recognizing the virtual substitute event as a real event due to the CG of the video describing the virtual substitute event or the unnaturalness of the video scene. For example, in “augmented reality”, the viewer determines that the CG portion is artificial because of the unnaturalness of the CG superimposed on the live video. In “telepresence”, video from a remote place is projected on the screen in the first place, so even if the viewer views the video, it is not interpreted as real.

Regardless of how sophisticated the CG video is to show the alternative video depicting a virtual alternative event, in most cases, the viewer is watching a video that clearly shows an alternative event that is different from reality. There was a problem that recognition could not be wiped away.

In addition, the above-described conventional techniques all require special devices such as a head-mounted display and a glasses-type augmented reality wearable computer, so that the entire system becomes expensive, and time for dissemination becomes long. There was a problem that it took.

Accordingly, the present invention has been devised in view of the above-described problems, and its object is to provide a display system, a display method, and an attachment that can be easily and inexpensively replaced with a head-mounted display or the like. As well as providing a program. The present invention allows a viewer to recognize an event that has not actually occurred in front of the user as an event that has actually occurred in front of the user, and has already been used in smartphones and the like. For providing a head proximity type video display system that can be easily realized by using a portable terminal and that can be provided to a user as an inexpensive system, a head proximity body used in the system, and a head proximity type video display program Is preferred.

In order to solve the above-mentioned problems, the present inventor has made it popular that the viewer recognizes an event that has not actually occurred in front of the eyes as an event that has actually occurred in front of the eyes. A head proximity type video display system implemented as an application program of a mobile terminal such as a smartphone and the like, a head proximity body used therefor, a display system suitable for application to a head proximity type video display program, Invented display methods, attachments, and programs.
Here, the display system of the present invention is:
A mobile terminal having a screen;
An attachment that houses the mobile terminal so that the screen is visible to the user;
With
Mobile devices
The sensor of the mobile terminal detects the action by the user through the attachment,
The video displayed on the screen is controlled according to the detected action.

The head proximity type video display system of the present invention detects a movement of a recording means in which one or more alternative videos are recorded, a display means for displaying at least the alternative videos recorded in the recording means, and its own movement. A mobile terminal having a motion sensor; and a head proximity body having an accommodating means in which the portable terminal is accommodated so that the display means is positioned in front of the user's eyes during video viewing. The display of the substitute video is started or stopped according to the motion detected by the motion sensor.

Moreover, in the head proximity type video display system of the present invention, the portable terminal further includes an imaging unit that captures a real space as a live video from substantially the same viewpoint as the user, and the display unit includes the motion sensor. Depending on the detected movement, the alternative video recorded in the recording unit or the live video captured by the imaging unit is displayed and stopped, or the display of the alternative video and the live video is switched.

Further, in the head proximity type image display system of the present invention, the head proximity body includes a head frame that is close to the user's head when viewing the image, and the head frame that is closer to the user's head than the head frame. The head frame has openable and closable protruding pieces protruding from both sides, and the motion sensor detects a movement according to the opening / closing operation of the protruding piece by the user.

Further, in the head proximity type video display system of the present invention, the motion sensor moves the head proximity body to the front of the head at the start of visual recognition, and opens and closes the protruding piece in front of the head. Detects deceleration accompanying movement.

In the head proximity image display system of the present invention, the head proximity body further includes a lens for enlarging an image displayed by the display means of the portable terminal accommodated in the accommodation means.

Further, in the head proximity type video display system of the present invention, in the invention, the image taken in time series in advance in all directions centered on the user's head or computer graphics created in real time, etc. Alternative video acquisition means for acquiring the virtual video as one alternative video, and when the portable terminal reproduces the one alternative video, the mobile terminal is based on the user's line of sight or the movement of the head. Video control means for cutting out the video to be displayed from the one alternative video along the time series is further provided.

In addition, in the head proximity type video display system of the present invention, the alternative video acquisition means processes information acquired by accessing the public communication network as needed or information stored in advance into one alternative video.

The head proximity type video display system of the present invention further includes audio data acquisition means for acquiring audio data, wherein the video control means is an alternative to reproduce the audio data acquired by the audio data acquisition means. Play in conjunction with the video.

Further, the head proximity body of the present invention is used in the head proximity type video display system, and the storage means in which the portable terminal is stored so that the display means is positioned in front of the user's eyes when viewing the video. Have

Further, in the head proximity body of the present invention, a head frame that is close to the user's head when viewing an image, and an open / close projecting from both sides of the head frame at a smaller interval than the user's head And a free protruding piece.

The head proximity type video display program according to the present invention is a head proximity type video display program that causes a mobile terminal to display a video for a user whose head proximity body is close to the head. It is detected by the motion detection step that an alternative video recording step for recording the alternative video, a motion detection step for detecting the movement of the portable terminal, and displaying one or more alternative videos recorded in the alternative video recording step. The display step of starting or stopping according to the movement is executed by the portable terminal.

本 According to the present invention configured as described above, a display system, a display method, an attachment, and a program can be provided. According to the present invention, it is possible to expand the visual information space by using an alternative reality technology (a technology for experiencing current and past images seamlessly as an extension of reality). In particular, according to the present invention, since it can be easily realized by using a portable terminal such as a smartphone that has already been widely used, it can be provided to the user as an inexpensive system, and rapid spread is expected. it can.

In particular, according to the present invention having the above-described configuration, a user feels as if a head-mounted display is attached to the user by allowing the user to use the portable terminal housed in the head proximity body. In addition, the start of video display on the display panel can be automatically controlled through the proximity movement of the head proximity object.

In particular, in the present invention, the portable terminal is used in a state of being accommodated in the head proximity body. For this reason, it is very troublesome for the user to operate the mobile terminal covered with the head proximity object. However, in the present invention, the video can be automatically started only by a natural proximity operation. Therefore, the operability is extremely excellent. Further, according to the present invention, since it can be operated via a portable terminal that operates wirelessly, it is possible to realize complete wirelessization of the entire system.

視覚 Such expansion of the visual information space allows the user to independently operate the information space and display the substitute video to access necessary information. According to the conventional system, information can only be accessed through a screen on a single layer of the device. Therefore, to add new information, the original information must be deleted by overwriting it. The accessibility was low. On the other hand, according to the present invention, it is possible to embed various information in a real space displayed as a live video through an alternative video with almost no loss of original information. Giving the user the impression that the amount of information has expanded can be realized by the portable terminal.

1 is an overall configuration diagram of a head proximity type video display system to which the present invention is applied. It is a figure for demonstrating the structure of a panoramic video camera. It is a figure which shows the external appearance shape of a head proximity body. It is a disassembled perspective view of a head proximity body. It is a figure for demonstrating the usage method of the head proximity | contact type | formula video display system to which this invention is applied. (a), (b) is a figure for demonstrating the external appearance shape of a portable terminal. It is a figure which shows the block configuration of a portable terminal. It is a figure for demonstrating the step structure of a control application. (a), (b) is a figure for demonstrating the example of a detection of the proximity | contact action of the head proximity | contact body by a motion sensor. (a), (b) is a figure for demonstrating the example of a detection of the proximity | contact operation | movement of the head proximity | contact body by a motion sensor. It is a figure for demonstrating the detail of the motion detection of the portable terminal by a motion sensor. (a), (b) is a figure for demonstrating the example which applies the past image | video imaged in the past as a substitute image | video. (a), (b), (c), (d) is explanatory drawing of the procedure which comprises an attachment with the packaging box of a portable terminal. (a), (b), (c) is explanatory drawing of the procedure which comprises an attachment with the packaging box of a portable terminal. (a), (b) is explanatory drawing of the attachment comprised by the packaging box of the portable terminal.

Hereinafter, an embodiment of a head proximity type video display system in which the display system according to the present invention is applied to provision of alternative reality content will be described in detail with reference to the drawings.

FIG. 1 shows an overall configuration diagram of a head proximity image display system 1 to which the present invention is applied. This head proximity type image display system 1 includes a portable terminal 2, a recording module 3 connected thereto, and a head proximity body 4, and is further connected to a communication network 5 via the portable terminal 2. ing.

The portable terminal 2 serves as a so-called central control device that controls the entire head proximity image display system 1. The mobile terminal 2 is embodied as a smartphone, a mobile phone, a tablet terminal, or the like, for example. However, the mobile terminal 2 is not limited to this, and any mobile terminal such as a mobile game device, a mobile music player, or a wearable terminal can be used. It is a concept that includes a type electronic device terminal.

The communication network 5 may be any wired or wireless communication network. For example, the Internet, an intranet, an extranet, a LAN (Local Area Network), an ISDN (Integrated Services Digital Network), a VAN (value added network). ), CATV (Community Antenna Television) communication network, virtual private network (Virtual Private Network), telephone line network, mobile communication network, satellite communication network, and the like.

The recording module 3 is used to record a substitute video based on a past event separately from an actual event, and includes a panoramic video camera 31.

The panoramic video camera 31 has a base 311 and a camera array 312 provided on the base 311 as shown in FIG. The height of the base 311 can be adjusted as necessary, and is set as needed to record a desired video. The camera array 312 includes a main body 320 and a plurality of imaging devices 321 mounted inside the main body 320. An imaging device 321 is mounted in each of the openings formed in the main body 320, and an object can be imaged through the openings. The plurality of imaging devices 321 are set with an angle of view and an imaging direction in order to enable imaging in different directions. Further, each imaging device 321 may be mounted so as to be able to capture all directions in the main body 320 (360 ° in the horizontal direction and 360 ° in the vertical direction) without leakage as shown in FIG. 2, for example. In addition, according to the imaging device 321, it is possible to perform imaging in all directions at the same time. For this reason, when imaging a person in a room, if the person moves, the person who has moved is imaged around the entire virtual viewpoint built in the camera body. A video is generated. Incidentally, the recording module 3 is not limited to the configuration described above, and may be replaced by any configuration as long as it can capture all directions of the main body 320.

The imaging device 321 uses a solid-state imaging device such as a CCD (Charge Coupled Device) for each of the images captured in this manner, forms a subject image incident through the lens on the imaging surface, and performs photoelectric conversion. A video signal is generated and transmitted to the portable terminal 2 or the recording computer via the interface 330.

In some cases, a microphone (not shown) may be mounted on the imaging device 321. The microphone (not shown) collects peripheral sounds and converts them into audio signals. The microphone (not shown) transmits the converted audio signal to the portable terminal 2 or the recording computer via the interface 330.

The head proximity body 4 is an attachment of the mobile terminal 2, and the mobile terminal 2 is accommodated in the attachment, so that the head close to the so-called viewer as shown in the perspective view of FIG. 3 and the assembly drawing of FIG. It is configured as a pseudo head-mounted display that can be placed close to and can be mounted. By accommodating the mobile terminal 2 in the head proximity body 4, a glasses-type or goggles-type display device is realized. The head proximity body 4 is roughly classified into a proximity unit 41 and a terminal storage unit 42. I have. Each component of the attachment head proximity body 4 (attachment) can be provided with a pattern, a color, a decoration, and the like according to the use and function.

The proximity unit 41 is made of paper such as cardboard or resin, but is not limited to this, and may be made of any material such as a ceramic material or metal. A head frame 102 close to the head and projecting

pieces

101a and 101b projecting from both sides of the head frame 102 are provided.

The heel head frame 102 is formed of a cylindrical body having a rectangular cross section. A terminal accommodating unit 42 can be inserted into the cylindrical head frame 102. The proximity unit 41 is brought close to the user's head in the direction C in the head frame 102 and is therefore attached. The head frame 102 is formed with

side plates

102a and 102b facing each other, and is attached to the side plate 102a such that the protruding piece 101a protrudes in the C direction, and the protruding piece 101a is provided on the side plate 102b. It is attached to protrude in the direction. The distance between the

side plates

102a and 102b in the head frame 102 is substantially equal to or less than the general width of the user's head.

Since the protruding piece 101a and the protruding piece 101b are respectively attached to the

side plates

102a and 102b, the interval between the protruding piece 101a and the protruding piece 101b is equal to or less than the general head of the user. Yes. The protruding piece 101a is openable and closable in the D direction with respect to the side plate 102a via the hinge mechanism 103. Similarly, the protruding piece 101b can be opened and closed in the D direction via the hinge mechanism 103 with respect to the side plate 102b. Note that the protruding

pieces

101a and 101b may be connectable to each other via rubber, string, or the like, and can be firmly attached by covering them on the user's head. The user can freely adjust the angle D of the projecting piece to minimize the influence from outside light.

ヒンジ When the proximity unit 41 is made of cardboard, the hinge mechanism 103 only needs to be configured as a bent portion of the cardboard. Further, when the proximity unit 41 is made of metal or resin, the hinge mechanism 103 is provided with a through hole (not shown) at the joint position of the protruding piece 101 and the side plate 102 in the vertical direction. You may make it comprise the hinge which can be opened and closed by inserting. However, it is desirable that the protruding pieces 101 are not opened in the D direction in a so-called normal state in which an image is not visually recognized, and the protruding

pieces

101a and 101b are narrower than the user's head.

The terminal housing unit 42 is made of paper such as cardboard or resin, but is not limited to this, and may be made of any material such as ceramic material or metal and is inserted into the proximity unit 41. The rear plate 110 provided in the C direction, the lens 111 formed on the rear plate 110, the rear plate 110, and the lens 111 are opposed to each other, in other words, in a direction opposite to the C direction. A front plate 112 is provided, and a pressing piece 113 is provided between the front plate 112 and the rear plate 110, and in particular, is disposed closer to the front plate 112. Between the front plate 112 and the pressing piece 113, an accommodating portion 114 for accommodating the portable terminal 2 is formed.

The lens 111 is a lens medium that can refract visible light emitted from the display panel 62 of the mobile terminal 2. An image displayed on the display panel 62 of the mobile terminal 2 can be enlarged and displayed in the user's field of view through the lens 111.
As the lens 111, an aspherical lens, for example, a convex lens, a plano-convex lens, a Fresnel lens, or the like can be adopted. However, the Fresnel lens is particularly suitable because it is light and thin and inexpensive.
In addition, when a stereoscopic image using parallax or a stereoscopic image using random dots is presented from the mobile terminal 2, it is easier for the user to visually recognize the left and right visual fields. In this case, a two-lens lens can be configured using one lens by opening two eyepiece openings on the left and right, or a two-lens lens by arranging two lenses on the left and right Can also be configured. As described above, when multi-lens lenses are employed, the same number of lenses may be prepared, or individual lenses may be fitted into the eyepiece openings. Furthermore, a wall that blocks the left and right fields of view may be provided in the proximity unit 41.

The bag accommodating portion 114 is configured with an interval that allows the front plate 112 and the pressing piece 113 to accommodate the portable terminal 2. The portable terminal 2 is accommodated in the accommodating portion 114 in a state where the display panel 62 side is directed in the C direction in the drawing and the imaging unit 44 side is directed in the direction opposite to the C direction. The holding pieces 113 protrude from the

side plates

132a and 132b in the terminal housing unit 42 toward the inside, respectively, and have a size and shape that can support the end portion of the mobile terminal 2. As a result, it is possible to prevent the display image from the display panel 62 in the mobile terminal 2 from being blocked by the pressing piece 113. Further, the front plate 112 is provided with

grooves

123 and 124 at two locations. The groove 123 is provided to make it easier for the user to grasp the portable terminal 2 when taking it out. The groove 124 is provided at a location corresponding to the position of the imaging unit 44 when the mobile terminal 2 is accommodated. As a result, the shooting direction by the imaging unit 44 is released by the groove 124 and is not blocked by the front plate 112.

As shown in FIG. 3, the head proximity body 4 accommodates the portable terminal 2 in the accommodation portion 114 in the terminal accommodation unit 42 having the above-described configuration, and inserts the terminal accommodation unit 42 into the proximity unit 41. Be available. When the head proximity body 4 is actually brought close to the user, the protruding

pieces

101a and 101b are opened as shown in FIG. 5, and the head frame 102 in the proximity unit 41 is brought close to the user's head. Thereby, the user's field of view is shielded through the protruding piece 101 and the head frame 102, and can be concentrated over a long time on the display image of the display panel 62 of the mobile terminal 2 that is enlarged and displayed through the lens 111. It becomes possible. As a result, the user can experience the same immersive feeling as if he / she was wearing a head mounted display. In this state, for example, the user may make the terminal accommodating unit 42 inserted in the proximity unit 41 approach or separate from the user. Thereby, the distance between the user's eyes and the lens 111 can be changed, and the focus on the image displayed on the display panel 62 can be freely adjusted.

FIG. 6 (a) is a plan view of the portable terminal 2, and FIG. 6 (b) is a bottom view thereof. As shown in FIG. 6, the mobile terminal 2 includes a display panel 62, headphones 43, an imaging unit 44, and a motion sensor 59.

Headphone 43 can be worn on the user's ear. The headphone 43 is not limited to a shape and size that completely envelops the user's ears, and may be a small earphone type. The configuration of the headphone 43 is not essential and may be omitted as necessary. However, it is desirable to have a noise canceling function when in use.

FIG. 7 shows a block configuration of the portable terminal 2. The mobile terminal 2 further includes a microphone 60 and a recording unit 69 in addition to the motion sensor 59, the display panel 62, the headphones 43, and the imaging unit 44 described above. The portable terminal 2 includes a peripheral interface (I / F) 57 and a control application 20. The power switch 58 and the operation unit 65 are connected to the peripheral I / F 57. The motion sensor 59, the microphone 60, the display panel 62, the headphones 43, the imaging unit 44, and the recording unit 44 are each connected to the peripheral I / F 57.

The display panel 62 displays the video imaged by the imaging unit 44 or the video image transmitted from the control application 20. When a video signal is input, the display panel 62 generates signals (R, G, and B primary color signals) that are elements for generating an image based on the video signal. Further, light based on the generated RGB signals is emitted and combined, and the light is scanned two-dimensionally. The two-dimensionally scanned light is converted so that its center line converges on the user's pupil and projected onto the retina of the user's eye.

The headphone 43 receives the audio signal transmitted from the mobile terminal 2. The headphone 43 outputs sound based on the input sound signal.

The heel peripheral I / F 57 is an interface that plays a role of relaying transmission / reception of various information for transmitting information acquired from the motion sensor 59, the microphone 60, the operation unit 65, and the like to the control application 20.

The power switch 58 is constituted by, for example, a pushable button-type switch exposed to the outside, and when the user presses the switch, the processing operation by the portable terminal 2 is started or the processing operation is ended.

The heel movement sensor 59 detects the movement of the mobile terminal 2. As the motion sensor 59, a gyro sensor, an acceleration sensor, a geomagnetic sensor, or the like is used, and detects the angle and inclination of the portable terminal 2, and the head proximity body 4 in which the portable terminal 2 is accommodated, and also the speed. The motion sensor 59 may detect the position of the mobile terminal 2 itself. The motion sensor 59 may be replaced with a motion capture system using a camera (not shown) installed in the experience environment. Data on the movement of the user's head acquired by the motion sensor 59 is transmitted to the control application 20 via the peripheral I / F 57.

Microphone 60 collects ambient sounds and converts them into audio signals. The microphone 60 can transmit the converted audio signal to the headphones 43 via the peripheral I / F 57. In the process of transmission, some processing may be performed on the audio signal. In addition, while the portable terminal 2 is accommodated in the head proximity body 4 that is an attachment, the microphone 60 is also disposed therein. Accordingly, the user touches the head proximity body 4, hits the head proximity body 4, rubs it, deforms the head proximity body 4 by opening and closing the protruding

pieces

101 a and 101 b, etc. The sound resulting from the action is resonated by the head proximity body 4 and collected by the microphone 60. Therefore, by filtering or the like in the resonance frequency band unique to the head proximity body 4, it is possible to distinguish the external sound from the user's action on the head proximity body 4 that is an attachment.

The heel operation unit 65 is a so-called user interface for the user to perform input based on his / her own intention. The operation unit 65 includes a button or a touch screen that also serves as the display panel 66, and a user inputs via the touch screen. When any input is performed on the operation unit 65, the information is transmitted to the control application 20 via the peripheral I / F 57. However, while the portable terminal 2 is accommodated in the head proximity body 4 that is an attachment, the touch screen is also disposed therein. Therefore, as will be described later, a small opening that allows about one finger of the user to pass through the head proximity body 4 may be provided so that the touch screen can be touched through the opening. If the opening is enlarged, the area where the finger can touch increases, but external light is likely to enter, so the feeling of immersion is inferior. Therefore, although the area that can be touched by the finger is narrowed, a small opening is provided, and it is possible to touch an operation target such as an icon or a button displayed in the touch screen by a method described later.

The bag recording unit 69 is a storage for recording images, videos, and other various data. The writing and reading of data in the recording unit 69 are executed based on control by the control application 20.

The eyelid control application 20 is application software for controlling the entire head proximity image display system 1. The control application software itself is recorded in the recording unit 69, a memory (not shown), or the like.

FIG. 8 shows a step configuration of the control application 20 in the mobile terminal 2. The portable terminal 2 includes a video accumulation step S22 connected to the reproduction control step S28, a live video acquisition step S23, a video cutout direction update step S25, an audio data acquisition step S35, and a motion detection step S38. The video accumulation step S22 is continuous from the alternative video acquisition step S21, and the video cutout direction update step S25 is continuous from the head direction specifying step S24.

The substitute video acquisition step S21 acquires the video imaged by the panoramic video camera 31 as a substitute video. In this alternative video acquisition step S21, an alternative video processed based on various information and data acquired via the communication network 5 is acquired. Furthermore, in this alternative video acquisition step S21, information input via the flash memory or the recording medium may be acquired as an alternative video, or input via a user interface (not shown) in the portable terminal 2. It may be based on information.

Video accumulation step S22 accumulates the alternative video acquired in alternative video acquisition step S21. The storage of the substitute video in this video storage step S22 may be executed by the recording unit 69, for example. In this video accumulation step S22, not only the alternative video from the alternative video acquisition step S21 but also various contents and information may be accumulated in advance from the beginning and stored as alternative video. The substitute video recorded by the recording unit 69 in the video accumulation step S22 is read under the control of the reproduction control step S28.

The live video acquisition step S23 acquires the video signal captured by the imaging unit 44.

The head direction specifying step S24 acquires data relating to the movement of the user's head through the movement of the mobile terminal 2 detected by the motion sensor 59 in the mobile terminal 2. In this head direction specifying step S24, the direction of the user's head is actually specified from the acquired data relating to the movement of the user's head.

Video cutout direction update step S25 updates the cutout direction of the video to be displayed on the display panel 62 in the portable terminal 2 based on the direction of the user's head specified by the head direction specifying step S24.

The sound data acquisition step S35 acquires sound from the outside and accumulates it. As an external audio acquisition method, for example, it may be acquired from a public communication network via wire or wireless, or audio data recorded on a recording medium may be read and recorded. . Incidentally, the audio data acquisition step S35 is not limited to the case where the acquired audio data is accumulated, and is read out and used. The audio data acquired from the outside is directly output in real time as it is. Also good. The sound data acquired in the sound data acquisition step S35 or the sound data acquired from the outside is output by the headphones 43.

The reproduction control step S28 performs processing for reproducing the substitute video stored in the recording unit 69 in the video storage step S22 and the live video acquired in the live video acquisition step S23. In the reproduction control step S28, video reproduction is controlled using the information from the video cutout direction updating unit step S25 in its own reproduction operation. The playback video controlled in the playback control step S28 is displayed on the display panel 62 described above.

In the heel movement detection step S38, the movement of the portable terminal 2 is detected by the movement sensor 59 in the portable terminal 2. Hereinafter, the detailed operation by this motion detection step S38 will be described.

動き This motion detection step S38 detects the proximity movement of the head proximity body 4 as shown in FIGS. First, as shown in FIG. 9A, the user holds the head proximity body 4 in which the portable terminal 2 is accommodated in the accommodation portion 114. At this stage, the operation of the control application 20 is in a start state via the operation unit 65, but the display of video on the display panel 62 has not yet started.

Next, as shown in FIG. 9B, the head proximity body 4 is moved to the front of the user's head. The motion sensor 59 detects the movement of the head proximity body 4 via the mobile terminal 2 accommodated therein. At this stage, the user tries to bring the head proximity body 4 closer, but in fact, the distance between the

side plates

102a and 102b in the head frame 102 is approximately equal to or less than the general head width of the user. It is said that. Therefore, as shown in FIG. 9A, the user performs an operation of opening the

side plates

102a and 102b in accordance with the size of his / her head. As a result, the head proximity body 4 is decelerated in comparison with the speed at which it moves to the front of the user's head as shown in FIG. 9 (b) as the user opens the

side plates

102a and 102b. Become. The motion sensor 59 detects the deceleration of the moving speed of the head proximity body 4 via the mobile terminal 2 accommodated therein.
In this operation, generally, the orientation of the head proximity body 4 also changes. For example, even when the imaging unit 44 of the mobile terminal 2 faces the ground at the stage of gripping the head proximity body 4, the mobile terminal 2 is at the stage of moving the head proximity body 4 to the trouble of the user's head. The image pickup unit 44 may rise somewhat in the horizontal direction.
However, in FIGS. 9 (a) and 9 (b) and FIG. 10 (a) to be described later, the user's actions are expressed roughly, and it is as if the orientation of the head proximity body 4 has not changed. Show.
Now, after performing the operation of opening the

side plates

102a and 102b, the user brings the head frame 102 close to the head, as shown in FIG. Shield the sides. Incidentally, as an operation after this, for example, when the

side plates

102a and 102b are connected to each other via rubber or a string, they may be firmly attached by covering them on the user's head. However, the mounting as referred to in the present invention is used not only for firmly mounting such a rubber or string on the user's head, but also while being held by the user's hand as shown in FIG. 10 (b). Cases are also included.

The heel movement sensor 59 detects all the movements of the head proximity body 4 through the portable terminal 2. FIG. 11 shows a state in which the movement of the mobile terminal 2 accommodated in the head proximity body 4 in this proximity process is viewed from the side of the user. The initial gripping state of the head proximity body 4 as shown in FIG. 9A is E, and the

side plates

102a and 102b are opened as shown in FIGS. 9B and 10A. Let state be F. Further, the proximity state to the user's head is assumed to be G. At this time, the moving speed from E to F is high, while F to G are low. Moreover, since the operation of opening the

side plates

102a and 102b is essential when F reaches, the moving speed of the portable terminal 2 is always low in the vicinity of F. In addition, regarding the tilt of the mobile terminal 2, in many cases, the display panel 62 is almost upward in the state E, whereas in the states F and G, the orientation of the display panel 62 is almost horizontal. .

For this reason, it can be said that the direction and speed of the portable terminal 2 are specific movements that can occur when the mobile terminal 2 is mounted on the head proximity body 4 and used as the head proximity type video display system 1. . When the motion application 59 detects the movement of moving the head proximity body 4 to the front of the head and the deceleration accompanying the opening / closing operation of the protruding piece 101 before the head by the motion sensor 59 when the image is viewed. The user desires to start video display via the display panel 62, determines that the head proximity body 4 is close, and actually starts video display on the display panel 62. This makes it possible to smoothly link the start of the SR video experience with the user's action.

Thus, in the present invention, the mobile terminal 2 is used in a state of being accommodated in the head proximity body 4. For this reason, it is very complicated for the user to operate the mobile terminal 2 covered with the head proximity body 4, but in the present invention, the video is automatically started only by a natural proximity operation. Therefore, the operability is extremely excellent.

In addition, according to the present invention, since it can operate via the mobile terminal 2 that operates wirelessly, it is possible to realize complete wirelessization of the entire system.

On the other hand, when the movement of the mobile terminal 2 described above is not detected by the motion sensor 59, the control application 20 does not particularly start the video display via the display panel 62. Actually, the movement of the mobile terminal 2 as shown in FIG. 11 hardly occurs except when the present invention is used. Therefore, when the normal mobile terminal 2 is used, the above-described movement is caused by the motion sensor 59. It is hardly detected. For this reason, it is possible to prevent the control application 20 from erroneously starting the display of the video via the display panel 62 even though it is not intended when the normal mobile terminal 2 is used.

By the way, when the user ends video viewing,
Normally, the end operation is performed in the order of FIG. 10 (b), FIG. 10 (a), FIG. 9 (b), and FIG. 9 (a). The operations of G to F and F to E as shown in FIG. 11 are sequentially performed. In such a case, the latter is faster among the operation speeds G to F and the operation speeds F to E. For this reason, by detecting such a speed change via the motion sensor 59, it is determined that the user has moved away from the head proximity body 4 in the hope of stopping the video display via the display panel 62. The video display by the display panel 62 is terminated. As a result, the end of the SR video experience can be smoothly linked to the user's action.
Further, the stopping operation is not limited to the above-described method, and the user himself takes out the mobile terminal 2 from the head proximity body 4 and instructs to stop visual recognition through an operation by the normal operation unit 65. It may be.
Further, in the present embodiment, it takes a certain time until the reproduction of the video is stopped after the head proximity body 4 is removed from the user's head. For this reason, when the user is viewing video content such as a movie or video as an alternative video, the mobile terminal 2 starts to remove the head proximity object 4 from the user's head (step G in FIG. 11). The elapsed time from playback to the point when playback is stopped (step E in FIG. 11) is measured. Then, when the display is resumed with the head proximity body 4 mounted on the user's head, the mobile terminal 2 displays the video content only for the measured elapsed time or the time obtained by adding a certain grace time to the elapsed time. It is desirable to resume playback of the video content after rewinding. Thereby, the user can view the video content without interruption.

In addition, this invention is not limited to the case where it implement | achieves as the head proximity | contact proximity type | formula video display system 1 mentioned above, Information may be directly given to a brain using a brain machine interface technique, and this is also used. It may be embodied as an application program for realizing.

Next, the display operation of the display panel 62 by the portable terminal 2 in the head proximity type video display system 1 to which the present invention is applied will be described.

The mobile terminal 2 displays either live video display or alternative video display on the display panel 62, or displays the live video and alternative video in combination with each other on the display panel 62.

In the live video display, the imaging unit 44 performs imaging. The live video imaged by the imaging unit 44 corresponds to the direction and position of the mobile terminal 2 housed in the head proximity body 4. Actually, since the head proximity body 4 is close to the user, a live image corresponding to the orientation and position of the user's head is imaged through the imaging unit 44. The subject image picked up via the image pickup unit 44 is formed on the image pickup surface via the image pickup element and the lens, and a video signal is generated by photoelectric conversion.

The mobile terminal 2 receives such a video signal in a live video acquisition step S23, and a process for playing it as a live video is performed in a playback control step S28. The captured live video is transmitted to the display panel 62 via the mobile terminal 2.

The live video sent to the display panel 62 is emitted as light based on each RGB signal, which is an element for generating an image based on the video signal, and is scanned two-dimensionally. As a result, the live video imaged by the imaging unit 44 is projected onto the retina of the user's eye.

As described above, the imaging range by the imaging unit 44 corresponds to the movement of the user's head. For this reason, since the user can visually recognize the live video according to the movement of his / her head through the display panel 62, the user can view the real space from the display panel 62 with almost the same viewpoint as the user. It is possible to make the player feel the same feeling as he is visually recognizing.

で In the process of continuously executing such live video display, the mobile terminal 2 detects an alternative video display opportunity constantly or at intervals. The meaning of the detection of the alternative video display opportunity indicates that the user has expressed some intention regarding the display of the alternative video, or the case where the user has captured any opportunity to display the alternative video regardless of the user's intention. Is.

The detection of the substitute video display opportunity based on the user's intention may be based on, for example, the movement of the user's head detected by the motion sensor 59. Further, the proximity movement of the head proximity body 4 described above may be detected by the motion sensor 59, and an alternative image may be displayed accordingly. In such a case, instead of displaying a live video from the beginning, an alternative video is displayed. Furthermore, the detection of the substitute video display opportunity may be performed when some intention is displayed from the user via the operation unit 65 or detected via voice input via the microphone 60. It may also be based on any information acquired from the user, such as brain waves, vibrations, movements, position information, and the like. Alternatively, an alternative video display opportunity may be detected based on the surrounding environment including the user, for example, some smell, heat, or touch.

When there is an input through the operation unit 65 or an audio input through the microphone 60 as described above, in addition to displaying an alternative video at the same time as the input timing, between the display timing and the input timing. A time lag may be intentionally generated. As a result, the user can be made to recognize that the display of the substitute video is performed based on his / her feeling, and the display / playback of the substitute video can be realized without a sense of discomfort.

The detection of the substitute video display opportunity not based on the user's intention may be, for example, a forced transition to the display of the substitute video at intervals, or the substitute video based on the information received from the communication network 5 You may make it detect the display opportunity of an image | video. In addition, an event generated based on a predetermined program or algorithm incorporated in the mobile terminal 2 may be regarded as an alternative video display opportunity.

When the mode is shifted to the mode for displaying the substitute video, the substitute video is displayed on the display panel 62. A specific example of the alternative video display method will be described later.

後 After displaying the substitute video, the display may return to the normal live video only. When returning to the live video, the above-described various events to be detected as an opportunity to display the alternative video may be detected.

In the present invention, it is assumed that one or more alternative videos are acquired in the alternative video acquisition step S21. Incidentally, in this alternative video acquisition step S21, when two or more alternative videos are acquired, a so-called multi-layer alternative video is acquired in advance and recorded in the recording unit 69. deep. Then, under the control of the reproduction control step S28, one or more alternative videos are selected from the multi-layer alternative videos, and are reproduced in parallel with or superimposed on the live video.

In addition, according to the present invention, a live video and a substitute video may be displayed in combination. In such a case, when the live video and the substitute video are displayed so as to overlap each other, processing such as removing the background or extracting a person from at least one of the videos may be performed. . As a result, it is possible to display a target object such as a person who has been recognized to exist in reality or a person's own body on a past image, and it is possible to further improve the reality of an alternative image. .

Also, for example, the image quality of the substitute video and the live video may be set to be the same or similar. Thereby, even if a user superimposes and reproduces | regenerates a live image | video and an alternative image | video, it does not feel discomfort. These effects can be obtained not only when the substitute video and the live video are superimposed, but also when the video is switched to each other or displayed in parallel with each other.

In addition, in generating a superimposed image of the substitute image and the live image, the opacity of the substitute image and the opacity of the live image are adjusted in the reproduction control step S28, respectively. For example, only the transparency of the live video may be controlled, or the opacity of the substitute video as well as the live video may be controlled. As a result, it is possible to display the substitute video while keeping the video display of the live video thin. At this time, a configuration may be adopted in which the substitute video fades in or out with respect to the live video. As a result, even if a person, building, information, or the like that does not exist in the live image appears or disappears, the user does not feel uncomfortable.

Note that the opacity of the substitute video and the live video can be set for each pixel of the display panel 62. Even when the substitute video and the live video are always superposed, it is possible to actually superimpose only a part of the entire space. In addition, by freely setting the opacity of the substitute image and the live image in units of pixels, it is possible to freely change the range of the environment space where the substitute image and the live image are mixed with each other and the shape thereof. Further, the change in opacity for each pixel may be changed in time series and dynamically changed based on the user action detected by the motion sensor 59 or the like.

In addition, in the present invention, when an alternative image is displayed, a means for exerting at least one of smell, heat, vibration, touch, and sound on the user may be separately provided. For example, in the case where an emergency bulletin about natural disaster is played as an alternative video, it may be notified by vibration or voice to alert the user.

Furthermore, the present invention is not limited to the case where one substitute image is reproduced with a live image superimposed or in parallel. Since two or more alternative videos are stored in the video storage unit 22, the two or more alternative videos may be superimposed on the live video or reproduced in parallel.

As described above, according to the present invention, it is possible to multi-layer two or more alternative videos in advance and select one or more desired alternative videos and display them in combination with the live video. Further, according to the present invention, it is possible to sequentially switch a desired one of the alternative videos to be displayed.

Selection or switching of a desired substitute image, or switching between the substitute image and the live image may be executed based on the movement of the user's head, and consequently the movement of the head proximity body 4 close to the head. Good. In such a case, switching may be performed based on the movement of the head detected by the motion sensor 59. In addition, the present invention is not limited to this, and may be a case where some kind of intention is displayed from the user via the operation unit 65, or may be detected via voice input via the microphone 60. Further, it may be based on any information acquired from the user, such as brain waves, vibrations, and position information. Alternatively, an alternative video display opportunity may be detected based on the surrounding environment including the user, for example, some smell, heat, or touch.

In the above-described embodiment, one or more alternative videos are acquired in advance in the alternative video acquisition step S21 and recorded in the recording unit 69, and the alternative video is read from the recording unit 69 as necessary. However, the present invention is not limited to this. For example, when a new substitute image is required when displaying a live image, the newly required substitute image may be acquired from the communication network 5 each time.

によ According to the present invention, it is possible to expand the visual information space by using an alternative reality technology (a technology to experience the current and past images without any breaks as an extension of reality). Such an expansion of the visual information space allows the user to independently operate the information space and display a substitute video to access necessary information.

Furthermore, according to the present invention, the audio data acquired in the audio data acquisition step S35 may be reproduced together. In such a case, the mobile terminal 2 may reproduce the audio data in conjunction with the alternative video to be reproduced. For example, if the video to be played is map information, an announcement linked to this may be played back in conjunction with the audio data. If the video to be played is a content video related to a game, sound effects or music linked to the content may be played back as voice data.

In addition, according to the present invention, the electronic device may be operated in a wirelessly interlocked manner with another electronic device different from the portable terminal 2 or with another portable terminal. This also allows the SR video experience to be synchronized among multiple users. Actually, since the control application 20 can be controlled from the outside, sharing of the SR video experience can be realized by performing control for synchronization from another electronic device or another portable terminal.

In addition, the operation of the mobile terminal 2 at the time of visual recognition is not limited to the case where the operation is performed via the operation unit 65. By detecting that the user taps the protruding piece 101 by the motion sensor 59, the operation unit 65 may be used instead. In such a case, the motion sensor 59 detects the number of taps of the protruding piece 101, so that the video can be played back, stopped, fast forwarded, etc. freely.

In the following embodiments, examples of alternative videos will be described. However, this alternative video is not limited to the following embodiments, and any other content and data may be converted into the alternative video.

(Alternative video)
Hereinafter, an example of an alternative video will be described. However, this alternative video is not limited to the following embodiments, and any other content and data may be converted into the alternative video.

例 As an example of an alternative video, first, any information acquired from the communication network 5 can be used. In the above-described example, the mobile terminal 2 acquires user position information, and acquires a geographical display (map information) such as a building or a road corresponding to the position information from the communication network 5. And the map information is displayed according to the phenomenon currently displayed on the live image | video.

(Past video)
As this alternative image, a past image captured in the past may be applied. The past video is shot using the recording module 3. More specifically, the panoramic video camera 31 in the recording module 3 captures images in time series in all directions. The plan view of FIG. 12A shows a state in which a panoramic video camera 31 arranged at a position P is capturing an image in all directions. The imaging device 321 in the panoramic video camera 31 performs imaging without omission over 360 ° in the horizontal direction. Although this imaging is simultaneously performed even in the vertical direction, the following example will be described by taking the imaging in the horizontal direction as an example.

過去 In this way, when past images are captured, images are sequentially captured in time series in each direction by the plurality of imaging devices 321. Then, the captured image is sent to the mobile terminal 2. For this reason, at the stage where the imaging of the past video is finished, the video storage unit 22 in the mobile terminal 2 is in a state in which moving images captured in all horizontal directions around the position P are stored. . At this time, sound may be recorded simultaneously by a microphone (not shown).

Next, when taking this past video as an alternative video,
As shown in FIG. 12 (b), it is assumed that there is a user who has taken a past video and has brought the head proximity body 4 close to the same position P. When displaying the past video on the display panel 62, the portable terminal 2 reads the past video stored in the recording unit 69 and displays it on the display panel 62 in the same process as described above.

At this time, the line-of-sight direction detected by the movement of the head detected by the motion sensor 59 may be reflected in the display on the display panel 62. Information relating to the orientation of the user's head and the line-of-sight direction detected via the motion sensor 59 is sent to the mobile terminal 2. Information on the head direction by the motion sensor 59 specifies the actual head direction of the user in the head direction specifying step S24.

Next, the vision that the user actually wants to capture is specified through the head direction specifying step S24. If you specify that the user is facing the front,
It is only necessary to cut out an image reflected in the vision in the range indicated by the solid line in FIG. However, when the user recognizes the area indicated by the dotted line as a vision in the head direction specifying step S24, the range of the video to be extracted is shifted in the arrow direction by the video extraction direction updating step S25. The reproduction control step S28 cuts out the video in the range shifted in this way from the video recorded in the recording unit 69. At this time, since the past video is recorded in time series, it is desirable to cut out the cut-out timing along the time series. By displaying the image cut out in such a shifted range on the display panel 62 in the mobile terminal 2, the user can visually recognize the image visually according to the orientation of his / her head. It is possible to have a sense. Since the position P of the panoramic video camera 31 that has captured the past video is the same as the position P of the user's head where the mobile terminal 2 is brought close, such a feeling can be implanted.

のみ Only such substitute video as past video may be displayed on the display panel 62, or may be displayed in combination with live video. The user is initially viewing the live video, but is switched to this past video without realizing it. However, since the user visually recognizes the live video from the beginning, even if the user has switched to the past video, the user remains conscious of viewing the live video. In other words, by initially letting the user visually recognize the live video, it is difficult to notice the switch to the past video. At this time, after switching to the display of the substitute video, the sound recorded by the microphone (not shown) is also played through the headphones 43, so that the user can feel as if he / she is viewing the live video.

It should be noted that the display of the past video as the substitute video is not limited to the above-described embodiment. For example, it is not limited to the case where the position P of the panoramic video camera 31 that captured the past video and the position P of the user's head close to the head proximity body 4 are the same, and may be different positions. . Further, the present invention is not limited to the case where the past image is cut out by identifying both the head direction and the line-of-sight direction, and the past image may be cut out by identifying either the head direction or the line-of-sight direction. Good.

Note that when displaying the past video as an alternative video, various devices as described below may be incorporated.

For example, the image quality of the past video and the live video may be set to be the same or similar. In such a case, the mobile terminal 2 performs image quality adjustment via an adjustment value obtained from the characteristic data of the image sensor that captured each of the live video and the past video. Alternatively, the portable terminal 2 may determine an image quality standard in advance and automatically perform image quality adjustment so that the image quality of the live video and the past video is approximated. As a result, the user does not feel discomfort when switching from the live video to the past video, and can taste the past video as if viewing the live video. These effects can be obtained not only when the past video and the live video are switched and displayed, but also when the images are displayed in parallel with each other or when they are displayed in a superimposed manner.

In particular, when the substitute video is a past video, the target displayed on the past video and the target displayed on the live video may be mixed. Thereby, it becomes possible to make it more difficult to distinguish the past video from the current video.

(Game content video)
As an alternative video, for example, a content video related to a game may be applied. In recent years, games using a head-mounted display have been produced, but these are applied to alternative images, and actions from the user as a player (based on input from the motion sensor 59, the operation unit 65, etc.) are reflected in the game. Let Since the alternative video is layered in a plurality of layers, the video relating to the game content is also layered in a plurality of layers, and these are sequentially read and displayed according to the scene. At this time, layered game content videos are superimposed on each other, or these are faded out and faded in to each other, so that it is possible to represent the transition between the alternative images without any sense of incongruity to the user.

(Video content such as movies)
For example, video content such as a movie may be reproduced as an alternative video. In such a case, normal movie content may be played back as the first substitute video, and accompanying information regarding the movie content may be played back as the second substitute video. As this accompanying information, for example, a comment from a viewer regarding movie content may be acquired from a communication network and flowed, or a cast, a synopsis up to now, personal relationships, etc. may be displayed. Moreover, you may make it reproduce | regenerate the news, weather forecast, etc. which are unrelated to a movie to a 3rd alternative image | video.

(Real-time content such as investment information)
As an alternative video, investment information (stock, exchange rate, bond, futures transaction) may be reproduced in real time. In such a case, a chart of a 5-minute bar for a certain brand may be displayed in the first alternative video, and a daily chart of the brand may be displayed in the second alternative video. In addition, a chart of daily and monthly bars may be displayed side by side on the first alternative video, and a chart of exchange rates may be placed on the second alternative video, or news acquired in real time, and a quotation May be displayed. In any case, investment information is acquired from the communication network 5 in real time, and this is used as alternative information over a plurality of layers. The user designates investment information that the user wants to check among the alternative information composed of a plurality of layers, and these are read out and displayed on the display panel 62.

(Display by application)
Various applications applied to portable information terminals (mobile phones, smartphones), tablet terminals, PCs, and the like may be reproduced as alternative videos. In such a case, these applications are acquired from the communication network 5 and stored in the recording unit 69. If necessary, the application is read from the recording unit 69 and reproduced as a substitute video.

(Images taken elsewhere)
You may make it reproduce | regenerate this as a video image | photographed at the point different from the point where the user is actually viewing live video or a substitute image, and a substitute image. In such a case, a video is captured in advance or in real time by the recording module 3 installed in a different place, and this is transmitted to the mobile terminal 2. In the alternative video acquisition step S21, the transmitted video is acquired as an alternative video, and if it is a video captured in advance, this is once stored in the recording unit 69. By removing the background of the live video and displaying it on top of the above alternative video, it is possible to create a feeling as if you were at a remote location.

(Accompanying information)
When a theater is played as a live video or when a sporting event is performed at a stadium or the like, accompanying information regarding this may be reproduced as a substitute video. For example, after acquiring the location information of the mobile terminal 2, if the location is a stadium, information on the competition to be performed in that time zone, and if the location is a theater, about the play scheduled to be performed in that time zone Information is acquired from the communication network 5. Then, the information acquired from the communication network 5 is reproduced as a substitute video. Alternatively, videos taken from a plurality of panoramic cameras may be used as a plurality of alternative videos, which can be switched according to the user's intention.

(e-mail)
An e-mail screen may be displayed as an alternative video. The portable terminal 2 reproduces the mail received via the communication network 5 or information related thereto as an alternative video.

(Television broadcasting)
As an alternative video, a television broadcast may be applied. Thereby, the user can view the television broadcast from the alternative video in combination with the live video.

In the above embodiment, the present invention has been applied to provide an alternative video, but the present invention relates to a display system that provides an alternative such as a head-mounted display inexpensively and easily, The minimum configuration is realized by housing the mobile terminal 2 having a screen in an attachment that is the head proximity body 4.

For example, it is assumed that the mobile terminal 2 plays a video image and the user views it. When viewing video images, there are situations where you want to perform operations such as playback, pause, fast forward, and rewind.

状況 In a situation where the mobile terminal 2 is configured by a smartphone having a touch screen and is not accommodated in the attachment, such an operation can be performed by touching an icon, a button, or the like displayed on the touch screen.

On the other hand, in a situation where the mobile terminal 2 is accommodated in the attachment, the touch screen is also disposed inside the attachment. Therefore, in this embodiment, a small opening that allows one finger to pass through is provided on the bottom surface of the terminal accommodating unit 42 so that the user can touch the touch screen. The reason for making the opening small is to prevent external light from entering the attachment as much as possible.

The user can touch the touch screen through the opening of the attachment. Since the opening is small, the entire surface cannot be touched. For example, if a right-handed user has an opening on the right bottom surface for a right-handed user, the user can touch the right side of the touch screen but cannot touch the left side because his / her finger does not reach. . That is, the touchable area is limited to a part of the touch screen.

Therefore, in this embodiment, the inclination of the attachment housing the mobile terminal 2 is detected by the motion sensor 59 of the mobile terminal 2. And according to this inclination, the position of the operation target displayed in the touch screen is changed.

For example, when the operation objects are arranged in a line, the line is cyclically advanced in the direction of the detected inclination. Then, the path of travel passes through the touchable area.

In other words, even if a desired operation target is displayed outside the touchable area, the operation target can be moved into the touchable area by tilting the attachment housing the mobile terminal 2. After the desired operation target moves within the touchable area, the user's finger is touched on the desired operation target displayed on the touch screen through the opening at the bottom of the attachment, and is associated with the operation target. Processing is executed.

It is arbitrary how to arrange a plurality of operation objects and how to move them according to the detected inclination.

For example, if the operation target is arranged in multiple rows and multiple columns, each tilt and return, the row and the column are rotated one by one in the direction of the tilt, that is, the diagonal scroll is performed obliquely. Also good.

In addition, a physical simulation may be used in which a viscous liquid is put into a container, an object is put into the container, and the container is tilted. That is, the detected inclination is applied to the inclination of the container, and the operation target is associated with the object. The user can touch the operation target after adjusting the inclination of the attachment housing the mobile terminal 2 to bring the desired operation target close to the finger inserted into the attachment through the opening.

In the above embodiment, the image displayed on the mobile terminal 2 is controlled by opening / closing or tapping the protruding piece 101 of the attachment, touching the touch screen of the mobile terminal 2 through the opening of the attachment, or the like. Here, the attachment has a box-like shape. For this reason, when the user performs an action with contact with the attachment, such as hitting the surface, rubbing it, denting it a little, returning it, opening / closing or tapping the protruding piece 101, the action The sound caused by the sound is generated and resonates due to the box shape. The resonated sound can be collected from the microphone 60 of the mobile terminal 2.

In addition, when the above actions are performed, the position and orientation of the mobile terminal 2 accommodated in the attachment also change. This movement can be detected by the movement sensor 59.

Therefore, in the present embodiment, the video displayed by the mobile terminal 2 accommodated in the attachment is controlled by one or both of the voice and the movement generated by the action as described above by the user.

音声 As described above, the sound generated by the attachment due to the action accompanied by the contact with the attachment is collected by the microphone 60 after being resonated by the attachment. Therefore, it is possible to easily separate the collected sound from the external sound by a technique such as applying a bandpass filter that passes only the frequency band resonated by the attachment.

In addition, let multiple users perform the same operation (for example, tap operation on the same part) for the same attachment, record the voice and movement detected at that time, extract the voice and movement template in advance There is also a technique to leave.

For example, when the right side of the attachment is tapped once, a tap sound is detected via the microphone 60, and a movement of the mobile terminal 2 slightly moving left once and then returning to the right is detected. When the right side of the attachment is rubbed, a friction sound is detected via the microphone 60 and a movement of the mobile terminal 2 vibrating is detected, but the left side vibration is smaller than the right side vibration. It is possible to prepare a template for each action by collecting such voice and movement characteristics through experiments.

Then, the mobile terminal 2 compares the voice and movement detected during operation with a template prepared in advance, selects a template whose similarity is sufficiently high based on a predetermined comparison criterion, and associates the template with the template. Determine that the user performed the action.

For example, during video playback, you can associate actions with processes such as switching between playback and stop with one tap on the right, fast-forwarding with two taps on the right, rewinding with two taps on the left, and menu display when rubbing the right. it can. After the menu is displayed, move the cursor pointing to the item in the menu with the friction on the right side or the friction on the left side, and select the item pointed to by the cursor with a single tap on the right side, etc. be able to. Here, the association between the type and number of actions and the process can be arbitrarily changed.

According to the present embodiment, regardless of the presence or absence of the protruding piece 101, the mobile terminal 2 can be controlled without impairing the user's immersive feeling by identifying the action accompanied by the user touching the attachment.

The attachment according to the present embodiment is configured by a box that wraps the portable terminal 2 when the portable terminal 2 is transported or sold.
Here, packaging is the technology and state of applying appropriate materials, containers, etc. to an article to protect the value and state of the article during transportation, storage, etc., and is classified into individual packaging, interior decoration, and exterior packaging. Is done. Depending on the purpose of packaging, packaging is classified into industrial packaging intended for transportation, transportation packaging, delivery packaging, and other commercial packaging intended for sale. Called. Packaging in the present application is a concept including all of the above.
FIGS. 13 (a), (b), (c), and (d) are explanatory diagrams of a procedure for configuring an attachment with a packaging box of a portable terminal. Hereinafter, a description will be given with reference to FIG. In this figure, the cross section of the portable terminal 2 and the box which wraps this is illustrated. In this figure, the thickness, size, and gap of each part are exaggerated for easy understanding.

The box 501 for packaging the portable terminal 2 includes a lid 502, a main body 503, and a tray 504. The lid 502 covers the main body 503, and the tray 504 is housed inside the main body 503 with the portable terminal 2 placed on it (this figure (a)). The lid 502 and the main body 503 can be made of paper such as cardboard, or various resins, but are not limited to this, and may be made of any material such as ceramic material or metal. The material can be arbitrarily selected. Moreover, a pattern, a color, a decoration, etc. can be given to these according to the use and function.

In general, when the mobile terminal 2 is shipped, the mobile terminal 2 is placed on the tray 504 so that the bottom surface of the tray 504 is in contact with the surface (back surface) opposite to the screen of the mobile terminal 2. In addition, a protrusion called a nail is generated in a part of the tray 504 where the mobile terminal 2 is accommodated, and the mobile terminal 2 can be easily fixed by fitting the mobile terminal 2 therein. In this figure, the cross section of the place with a nail | claw is shown.

The bag tray 504 is made of a material such as a transparent plastic, and a Fresnel lens 505 is integrally formed as a thin lens at a central portion of a region where the mobile terminal 2 is placed. The user can obtain the Fresnel lens 505 by taking out the portable terminal 2 and the tray 502 from the main body 503 and then cutting the tray 504 along a perforation line or a cut line (this diagram (b)). The lid 502 is also provided with a perforation or a cut line, and an opening 506 for the user to get an eyepiece is obtained by cutting along the perforation or cutting line (this figure (b)).

When the user attaches the Fresnel lens 505 to the opening 506 of the lid 502 (this figure (c)), a member corresponding to the proximity unit 41 is completed. The Fresnel lens 505 can be fixed by, for example, making a cut at the edge of the opening 506 of the lid 502 and sandwiching it, or can be bonded using an adhesive tape or the like.

The user turns the tray 504 upside down to fix the mobile terminal 2 and inserts it into the main body 503 (this figure (c)), thereby fixing the mobile terminal 2 to the main body 503. Thereby, a member corresponding to the terminal accommodating unit 42 in which the portable terminal 2 is accommodated is completed.

After this, when the user covers the main body 503 with the lid 502 (this figure (d)), the head proximity body 4 (attachment) containing the portable terminal 2 is completed. The user can focus on the screen of the mobile terminal 2 by adjusting the position where the lid 502 is put on the main body 503.

The main body 503 is cut along a perforation or cut line prepared in advance, so that an opening for touching the touch screen of the mobile terminal 2 with a finger or a cord for headphones from the mobile terminal 2 is provided. An opening can be created for exit.

The Fresnel lens 505 need not be integrally formed with the tray 504, and may be attached separately or attached to the lid 502 from the beginning of shipment. Further, in the above description, the portable terminal 2 is fixed by inserting the tray 504 upside down. However, without using the tray 504, the bottom or wall surface of the main body 503 is provided with a notch or protrusion. The portable terminal 2 may be fixed as a stopper. In this aspect, the head proximity body 4 (attachment) can be created not only from a packing box (decorative box) used at the time of shipment of the mobile terminal 2 but also from various cardboard boxes for mail order sales.

In addition, when the Fresnel lens 505 is prepared separately from the tray 504 and the bottom surface of the tray 504 is made of a transparent material, the portable terminal 2 is turned over with respect to the tray 504 and carried on the bottom surface of the tray 504. After placing the terminal 2 so that the screen of the terminal 2 is in contact and fixing the portable terminal, the tray 504 is turned upside down in the main body 503 so that the screen of the portable terminal 2 can be seen outside through the bottom surface of the tray 504.

Then, the lid 502 may be covered with the main body 503 so that the screen of the portable terminal 2 can be seen through the Fresnel lens 505 installed in the opening 506 of the lid 502.

ほか There is another method that does not require the Fresnel lens 505 to be separated from the tray 504. FIGS. 14 (a), 14 (b), and 14 (c) are explanatory diagrams of a procedure for configuring an attachment with a packaging box of a portable terminal. Hereinafter, a description will be given with reference to FIG.

In this embodiment, the Fresnel lens 505 is formed on the bottom surface of the tray 504 as in the above embodiment (FIG. (A)).

Therefore, the portable terminal 2 is taken out and fixed to the bottom of the main body 503 with a stopper (not shown) using a notch or the like (this figure (b)). Then, the main body accommodation unit 42 is formed.

On the other hand, the tray 504 is turned over and inserted into the main body 502 (this figure (b)). Thereby, the simplest eyepiece unit 41 can be obtained.

That is, by turning the tray 504 upside down and inserting it into the main body 502, the simplest attachment can be obtained. In order to prevent intrusion of external light from the side of the tray 504, an opaque material may be used for the side of the tray 504. When a transparent material is used, a light shielding material is provided on the side of the tray 504. A sticker or tape may be attached.

In this aspect, if a flap (not shown) is prepared in the main body 503, the flap can be used as the protruding piece 101 of the housing unit.

In addition, the tray 504 is fixed to the lid 501 by a fixing method such as providing an opening 506 in the lid 501 and inserting the claws of the tray 504 into a groove provided in the lid 501 (not shown). A tray 504 may be inserted into 502 (this figure (c)). In this embodiment, the eyepiece unit 41 is obtained by turning the tray 504 upside down and fixing it to the lid 501. In this aspect, since both the inside and outside surfaces of the housing unit 42 are in contact with the eyepiece unit 41, the friction works sufficiently and the accommodation unit 42 and the eyepiece unit 41 are not easily displaced even after the focus is adjusted. Become.

In the embodiment in which the Fresnel lens 505 is provided on the tray 504, it is not always necessary to turn the tray 504 upside down. FIGS. 15 (a) and 15 (b) are explanatory views of an attachment constituted by a packaging box of a portable terminal. As shown in this figure (a), once the tray 504 is removed from the main body 502, the portable terminal 2 is fixed to the bottom surface of the main body 502, and then the tray 504 is inserted again into the main body 502 in the same direction, the bottom surface of the tray 504 The screen of the portable terminal 2 can be viewed through the Fresnel lens 505.

In addition, as shown in FIG. 5B, if the tray 502 is fixed to the lid 501 provided with the opening 506 by adhesion or the like and then covered with the main body 502, the tray 502 is effectively blocked while external light is blocked. The screen of the mobile terminal 2 can be visually recognized through the Fresnel lens 505 on the bottom surface of 504.

In the above example, one Fresnel lens 505 is used for the eyepiece, but two round convex lenses may be molded on the bottom surface of the tray 502. In this case, the lid 501 is provided with two round openings, or the portion other than the convex lens of the tray 502 is formed with an opaque member, or is shielded by an opaque member such as a seal or paper. By doing so, the attachment 4 can also be configured easily.

As described above, according to the present embodiment, it is possible to provide a user with a display system that is an alternative to a head mounted display and the like at low cost and easily.

(Summary)
As described above, the display system according to the present application is
A mobile terminal having a screen;
An attachment for accommodating the portable terminal so that the screen is visible by a user;
A display system comprising:
The portable terminal is
The sensor of the portable terminal detects an action by the user via the attachment,
The video to be displayed on the screen is controlled according to the detected action.

In the display system of the present invention,
The portable terminal is
Detecting voice emitted from the attachment and movement of the mobile terminal by the sensor,
Identifying whether the sensed voice and movement is caused by an action including contact to the attachment by the user, friction of the attachment, or deformation of the attachment;
The video to be displayed on the screen is controlled in accordance with the identified action.

In the display system of the present invention,
The portable terminal is
The sound emitted from the attachment is detected by the sensor,
Identifying whether the sensed sound is emitted due to an action including contact to the attachment by the user, friction of the attachment, or deformation of the attachment;
The video to be displayed on the screen can be controlled according to the identified action.

In the display system of the present invention,
The portable terminal is
The frequency component of the detected sound can be classified according to whether or not the frequency band in which the attachment resonates, and can be configured to identify whether or not the sound is generated due to the action.

In the display system of the present invention,
The portable terminal is
The movement of the mobile terminal is detected by the sensor,
Identifying whether the detected movement is caused by an action including contact to the attachment by the user, friction of the attachment, or deformation of the attachment;
The video to be displayed on the screen can be controlled according to the identified action.

In the display system of the present invention,
The screen is a touch screen;
The attachment has an opening for enabling the user to touch a partial area of the touch screen;
The portable terminal is
Based on the detected movement, the position of the operation target included in the video is moved in and out of the touchable area,
The touch screen is configured to control the video displayed on the screen according to the touch when the touch on the operation target that the user has moved into the touchable area is detected. Can do.

In the display system of the present invention,
The mobile terminal has a camera whose shooting direction is a direction in which the user visually recognizes the screen,
The attachment accommodates the portable terminal so that the camera can shoot a live video of a real space,
According to the identified action, the video displayed on the screen can be switched to the live video or the non-live video.

In the display system of the present invention,
The portable terminal is
While the non-live video is displayed on the screen, based on the movement detected by the sensor, a partial video is cut out from the alternative video including the panoramic video and the 3D video, and the cut-out partial video is removed from the non-live video. It can be configured to be live video.

In addition, in the attachment in the display system of the present invention,
The attachment is
In the box for packaging the portable terminal by fixing the portable terminal with a tray, storing the tray in the main body, and covering the main body with a lid,
By fixing a part of the bottom surface of the tray, the mobile terminal is fixed so that the screen of the mobile terminal is visible through an opening provided in the tray.
The opening is provided, the tray for fixing the mobile terminal is stored in the main body,
By covering a part of the upper surface of the lid and covering the main body with the lid so that the screen of the portable terminal can be visually recognized through an opening provided in the lid. be able to.

In addition, in the attachment in the display system of the present invention,
The tray is constituted by a transparent body having a lens at a part of the bottom surface,
A lens cut out from the tray can be attached to the opening provided in the lid.

In addition, in the attachment in the display system of the present invention,
The attachment is
In the box for packaging the portable terminal by fixing the portable terminal with a tray, storing the tray in the main body, and covering the main body with a lid,
The tray is constituted by a transparent body having a lens at a part of the bottom surface,
The mobile terminal is fixed to the bottom surface of the main body so that the screen of the mobile terminal is visible,
The tray can be housed in the main body so that the screen of the portable terminal can be visually recognized through the lens.

In addition, in the attachment in the display system of the present invention,
The attachment is
In the box for packaging the portable terminal by fixing the portable terminal with a tray, storing the tray in the main body, and covering the main body with a lid,
The tray is constituted by a transparent body having a lens at a part of the bottom surface,
The mobile terminal is fixed so that the screen of the mobile terminal is visible through an opening provided in the tray by cutting the lens from the bottom surface of the tray,
The opening is provided, the tray for fixing the mobile terminal is stored in the main body,
A lens excised from the tray is attached to the opening provided in the lid by excising a part of the upper surface of the lid,
The main body may be covered with the lid so that the screen of the portable terminal can be visually recognized through the lens attached to the opening.

The attachment of the present invention is
An attachment that houses a mobile terminal having a screen so that the screen can be viewed by a user,
Transmitting voice or movement generated due to an action accompanied by a contact with the attachment by the user to the mobile terminal;
The attachment is
In the box for packaging the portable terminal by fixing the portable terminal with a tray, storing the tray in the main body, and covering the main body with a lid,
The tray is constituted by a transparent body having a lens at a part of the bottom surface,
The mobile terminal is fixed to the bottom surface of the main body so that the screen of the mobile terminal is visible,
The tray is stored in the main body so that the screen of the portable terminal can be visually recognized through the lens.

The display method according to the present invention is as follows:
A display method executed by a mobile terminal, wherein the mobile terminal has a screen, and the mobile terminal is accommodated in an attachment so that the screen can be viewed by a user,
A detection step in which the mobile terminal detects an action by the user via the attachment by a sensor of the mobile terminal;
A control step of controlling an image displayed on the screen in accordance with the detected action.

The program according to the present invention is a program for controlling a mobile terminal, wherein the mobile terminal has a screen, and the mobile terminal is accommodated in an attachment such that the screen is visible by a user,
The program is stored in the mobile terminal.
The sensor of the portable terminal detects an action by the user via the attachment,
The video to be displayed on the screen is controlled according to the detected action.

The program can be recorded on a computer-readable non-transitory information recording medium. The information recording medium can be distributed and sold independently of the computer for realizing the portable terminal according to the present invention. In addition, the program can be distributed and sold from a distribution server to a computer for realizing the portable terminal according to the present invention via a temporary communication medium by a computer communication network such as the Internet.

によ According to the present invention, it is possible to provide a display system, a display method, an attachment, and a program that can be realized easily and inexpensively as an alternative to a head-mounted display or the like.

In this application, we shall claim priority based on Japanese Patent Application No. 2014-067716 filed in Japan on March 28, 2014, and the contents of the basic application as long as the laws of the designated country permit. Is incorporated into the present application.

1 Head proximity image display system 2 Mobile terminal 3 Recording module 4 Head proximity body (attachment)
5 Communication Network 20 Control Application 22 Video Storage Unit 31 Panoramic Video Camera 41 Proximity Unit 42 Terminal Housing Unit 43 Headphone 44 Imaging Unit 58 Power Switch 59 Motion Sensor 60 Microphone 62 Display Panel 65 Operation Unit 66 Display Panel 69 Recording Unit 101 Projection Piece 102 Side plate 102 Head frame 103 Hinge mechanism 110 Rear plate 111 Lens 112 Front plate 113 Holding piece 114 Groove 124 Groove 311 Base 312 Camera array 320 Main body 321 Each imaging device 321 Imaging device 330 Interface 501 Box 502 Lid 503 Main body 504 Tray 505 Fresnel lens 506 Opening S21 Alternative image acquisition step S22 Image accumulation step S23 Live image acquisition step S24 Head direction identification step S25 Direction update step S25 Direction updating unit step S28 the playback control step S35 speech data acquiring step S38 detection step

Claims

A mobile terminal having a screen;
An attachment for accommodating the portable terminal so that the screen is visible by a user;
A display system comprising:
The portable terminal is
The sensor of the portable terminal detects an action by the user via the attachment,
A display system that controls an image to be displayed on the screen in accordance with the detected action.
The portable terminal is
Detecting voice emitted from the attachment and movement of the mobile terminal by the sensor,
Identifying whether the sensed voice and movement is caused by an action including contact to the attachment by the user, friction of the attachment, or deformation of the attachment;
2. The display system according to claim 1, wherein the video to be displayed on the screen is controlled according to the identified action.
The portable terminal is
The sound emitted from the attachment is detected by the sensor,
Identifying whether the sensed sound is emitted due to an action including contact to the attachment by the user, friction of the attachment, or deformation of the attachment;
2. The display system according to claim 1, wherein the video to be displayed on the screen is controlled according to the identified action.
The portable terminal is
2. The frequency component of the detected sound is classified according to whether or not the frequency band in which an attachment resonates, and whether or not the sound component is emitted due to the action is identified. Display system described.
The portable terminal is
The movement of the mobile terminal is detected by the sensor,
Identifying whether the detected movement is caused by an action including contact to the attachment by the user, friction of the attachment, or deformation of the attachment;
2. The display system according to claim 1, wherein the video to be displayed on the screen is controlled according to the identified action.
The screen is a touch screen;
The attachment has an opening for enabling the user to touch a partial area of the touch screen;
The portable terminal is
Based on the detected movement, the position of the operation target included in the video is moved in and out of the touchable area,
When the touch on the operation target that the user has moved within the touchable area is detected by the touch screen, the video displayed on the screen is controlled according to the touch. The display system according to claim 2 or 5.
The mobile terminal has a camera whose shooting direction is a direction in which the user visually recognizes the screen,
The attachment accommodates the portable terminal so that the camera can shoot a live video of a real space,
6. The display system according to claim 2, wherein the video to be displayed on the screen is switched to the live video or the non-live video in accordance with the identified action.
The portable terminal is
While the non-live video is displayed on the screen, based on the movement detected by the sensor, a partial video is cut out from the alternative video including the panoramic video and the 3D video, and the cut-out partial video is removed from the non-live video. The display system according to claim 6, wherein the display system is a live video.
The attachment is
In the box for packaging the portable terminal by fixing the portable terminal with a tray, storing the tray in the main body, and covering the main body with a lid,
By fixing a part of the bottom surface of the tray, the mobile terminal is fixed so that the screen of the mobile terminal is visible through an opening provided in the tray.
The opening is provided, the tray for fixing the mobile terminal is stored in the main body,
The cover is configured to cover the main body with the lid so that the screen of the mobile terminal can be visually recognized through an opening provided in the lid by cutting off a part of the upper surface of the lid. The attachment in the display system according to any one of claims 1 to 8.
The tray is constituted by a transparent body having a lens at a part of the bottom surface,
10. The attachment according to claim 9, wherein a lens cut from the tray is attached to an opening provided in the lid.
The attachment is
In the box for packaging the portable terminal by fixing the portable terminal with a tray, storing the tray in the main body, and covering the main body with a lid,
The tray is constituted by a transparent body having a lens at a part of the bottom surface,
The mobile terminal is fixed to the bottom surface of the main body so that the screen of the mobile terminal is visible,
9. The display system according to claim 1, wherein the display system is configured by storing the tray in the main body so that the screen of the portable terminal can be visually recognized through the lens. Attachment in.
The attachment is
In the box for packaging the portable terminal by fixing the portable terminal with a tray, storing the tray in the main body, and covering the main body with a lid,
The tray is constituted by a transparent body having a lens at a part of the bottom surface,
The mobile terminal is fixed so that the screen of the mobile terminal is visible through an opening provided in the tray by cutting the lens from the bottom surface of the tray,
The opening is provided, the tray for fixing the mobile terminal is stored in the main body,
A lens excised from the tray is attached to the opening provided in the lid by excising a part of the upper surface of the lid,
12. The attachment according to claim 11, wherein the attachment is configured to cover the main body with the lid so that the screen of the mobile terminal can be visually recognized through the lens attached to the opening.
An attachment that houses a mobile terminal having a screen so that the screen can be viewed by a user,
Transmitting voice or movement generated due to an action accompanied by a contact with the attachment by the user to the mobile terminal;
The attachment is
In the box for packaging the portable terminal by fixing the portable terminal with a tray, storing the tray in the main body, and covering the main body with a lid,
The tray is constituted by a transparent body having a lens at a part of the bottom surface,
The mobile terminal is fixed to the bottom surface of the main body so that the screen of the mobile terminal is visible,
The attachment is configured by storing the tray in the main body so that the screen of the portable terminal can be visually recognized through the lens.
A display method executed by a mobile terminal, wherein the mobile terminal has a screen, and the mobile terminal is accommodated in an attachment so that the screen can be viewed by a user,
A detection step in which the mobile terminal detects an action by the user via the attachment by a sensor of the mobile terminal;
A display method comprising: a control step of controlling an image displayed on the screen in accordance with the detected action.
A program for controlling a mobile terminal, wherein the mobile terminal has a screen, and the mobile terminal is accommodated in an attachment so that the screen can be viewed by a user,
The program is stored in the mobile terminal.
The sensor of the portable terminal detects an action by the user via the attachment,
A program for executing a process of controlling an image displayed on the screen in accordance with the detected action.
A portable terminal having recording means for recording one or more alternative videos, display means for displaying at least the alternative videos recorded in the recording means, and a motion sensor for detecting its own movement;
A head proximity body having an accommodating means for accommodating the portable terminal so that the display means is positioned in front of the user's eyes at the time of visual recognition,
The head proximity image display system, wherein the display means starts or stops displaying the substitute image in accordance with the motion detected by the motion sensor.
The portable terminal further includes an imaging unit that captures the real space as a live video from a viewpoint substantially the same as the user,
The display means displays and stops the alternative video recorded in the recording means or the live video imaged by the imaging means according to the motion detected by the motion sensor, or the alternative video and the above 17. The head proximity type video display system according to claim 16, wherein the live video display is switched.
The head proximity body includes a head frame that is close to the user's head when visually recognizing an image, and an openable / closable protruding piece that protrudes from both sides of the head frame at a narrower interval than the user's head. And
17. The head proximity image display system according to claim 16, wherein the movement sensor detects a movement according to an opening / closing operation of the protruding piece by a user.
The motion sensor
19. The head according to claim 18, wherein an operation of moving the head proximity object to the front of the head at the start of video viewing and a deceleration accompanying the opening / closing operation of the protruding piece in front of the head are detected. Close proximity type image display system.
17. The head proximity image display system according to claim 16, wherein the head proximity body further includes a lens for enlarging an image displayed by the display means of the mobile terminal accommodated in the accommodation means. .
Alternative video acquisition means for acquiring, as one alternative video, a video captured in time series in advance in all directions about the user's head or a virtual video such as computer graphics created in real time In addition,
When the mobile terminal reproduces the one alternative video, the video control means for cutting out the video to be displayed from the one alternative video along the time series based on the line of sight of the user or the movement of the head 17. The head proximity image display system according to claim 16, further comprising:
22. The head proximity type image display system according to claim 21, wherein the substitute image acquisition means processes information acquired by accessing the public communication network as needed or information stored in advance into one substitute image. .
Voice data acquisition means for acquiring voice data,
23. The head proximity type video display system according to claim 22, wherein the video control unit reproduces the audio data acquired by the audio data acquisition unit in conjunction with the substitute video to be reproduced.
17. A head used in the head proximity type video display system according to claim 16, further comprising a storage unit in which the mobile terminal is stored so that the display unit is positioned in front of the user's eyes when viewing the video. Close proximity body.
A head frame that is close to the user's head when the image is viewed; and an openable and closable protruding piece that protrudes from both sides of the head frame at a narrower interval than the user's head. The head proximity body according to claim 24.
In a head proximity type video display program that causes a mobile terminal to display a video for a user who has a head proximity body close to the head,
An alternative video recording step of recording one or more alternative videos;
A motion detection step for detecting the motion of the mobile terminal;
Displaying one or more alternative videos recorded in the alternative video recording step is started by a portable terminal in accordance with the motion detected in the motion detection step or stopped. Head proximity type video display program.