JP6294054B2 - Video display device, video presentation method, and program - Google Patents

Description

  The present invention relates to a technique for superimposing video content on a real environment and presenting it to a user.

  A technique for presenting an image using augmented reality (AR) is known. Patent Document 1 and Non-Patent Document 1 disclose a technique for presenting an image such as an advertisement without performing pre-processing such as placing a marker in a real environment. Patent Document 1 estimates a region having a surface in a background image from a photographed image, and a pixel value of each pixel in the estimated region indicates a single color or a monochrome gradation, and a region having a predetermined size or more is represented as a presentation image. Is selected as a region to be superimposed. Non-Patent Document 1 detects a plane group that is a target of advertisement presentation from an urban area image that is an image of an urban area, and selects a region suitable for display of a presentation image based on the result of calculating the height of a wall surface. In addition, superimposing a CG image on an urban area image is disclosed.

JP 2013-109469 A

Hiroyuki Uchiyama, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase, Takahito Kawanishi, Kunio Kanno, “Augmented Reality Advertising on Urban Structures”, [online], 2011, Proceedings of Workshop on Real Use of ViEW Vision Technology, [October 25, 2013 search] Internet <URL: http://www.murase.nuie.nagoya-u.ac.jp/publications/887-pdf.pdf>

In the techniques described in Patent Document 1 and Non-Patent Document 1, a display area of an image to be superimposed on the actual environment is determined according to the characteristics of the image obtained by capturing the actual environment.
In contrast, an object of the present invention is to enable a user to place video content in an actual environment observed by the user.

  In order to solve the above-described problems, a video display device according to the present invention includes a shooting unit that captures an image observed by a user and generates shooting data, and a display unit that displays video content superimposed on the observed image. And a gesture recognition unit for recognizing a gesture performed in a spatial region by the user based on shooting data generated by the shooting unit, and a gesture from the position of the user based on the gesture recognized by the gesture recognition unit. A setting unit for setting a video presentation area on an extension line of the spatial area, and displaying the video content on the display unit so that the user can observe the video content in the video presentation area set by the setting unit. A display control unit.

In the video display device according to the aspect of the invention, the gesture recognition unit may recognize a gesture that surrounds the space area planarly performed by the user.
The video display device of the present invention further includes a plane area detection unit that detects a plane area existing on the extension line, and the setting unit displays the video presentation area on the plane area detected by the plane area detection unit. It may be set.

  In the video display device of the present invention, when the video presentation area is no longer observed by the user, the display control unit interrupts reproduction of video content in the video presentation area, and the video presentation area is observed again. The playback may be resumed when done.

In the video display device according to the aspect of the invention, the display control unit may prevent the user from observing video content in one video presentation area when the user observes the plurality of video presentation areas. Or you may perform control which makes it harder to observe than the video content of the said other video presentation area | region .

  The video display device of the present invention includes a setting information acquisition unit that acquires setting information indicating the setting of the video presentation area from another video display device that is not its own device, and the setting unit is acquired by the setting information acquisition unit The video presentation area may be set based on setting information, and the display control unit may display the same video content as that of the other video display device.

  In the video display device of the present invention, the video display device includes a correction unit that corrects the video presentation area set by the setting unit based on the gesture recognized by the gesture recognition unit, and the gesture recognition unit includes the set video presentation region. When displayed on the display unit, a gesture for correcting the video presentation area may be recognized.

  The video presentation method of the present invention is a video of a video display device comprising: a photographing unit that captures an image observed by a user to generate captured data; and a display unit that displays video content superimposed on the observed image. A method of presentation, comprising: recognizing a gesture made in a spatial region by the user based on photographing data generated by the photographing unit; and viewing from the user's position based on the recognized gesture A step of setting a video presentation area on an extension line of the space area, and a step of displaying the video content on the display unit so that the user can observe the video content in the set video presentation area.

  The program of the present invention is provided in a computer of a video display device including a photographing unit that shoots an image observed by a user and generates photographing data, and a display unit that displays a video content on the observed image. The step of recognizing a gesture made in the spatial region by the user based on the photographing data generated by the photographing unit, and on the extension line of the spatial region as seen from the position of the user based on the recognized gesture A program for executing a step of setting a video presentation area and a step of displaying the video content on the display unit so that the user can observe the video content in the set video presentation area.

  According to the present invention, the user can place video content in an actual environment that the user observes.

The figure which shows the external appearance structure of the spectacles type terminal which concerns on one Embodiment of this invention. Explanatory drawing of the outline | summary of the video presentation function which the spectacles type terminal has. The block diagram which shows the hardware constitutions of the spectacles type terminal. The figure which shows the structural example of the setting table of the spectacles type terminal. The functional block diagram which shows the function structure of the control part of the spectacles type terminal. Explanatory drawing of the gesture which the user of the spectacles type terminal performs. The flowchart which shows the flow of the process which the spectacles type terminal performs. Explanatory drawing of the user's gesture which the spectacles type terminal recognizes. Explanatory drawing of the setting method of the image | video presentation area | region in the spectacles type terminal. Explanatory drawing of the other setting method of the image | video presentation area | region in the spectacles type terminal. The flowchart which shows the flow of the process which the spectacles type terminal performs. The figure which shows the example of a display of the video content in the spectacles type terminal. 10 is a flowchart showing a flow of processing performed by a glasses-type terminal according to Modification 3; 10 is a flowchart showing a flow of processing performed by a glasses-type terminal according to Modification 4;

[Embodiment]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing an external configuration of a glasses-type terminal 1 according to an embodiment of the present invention. FIG. 1 shows the eyeglass-type terminal 1 when worn by the user 2. The glasses-type terminal 1 is a video display device that is used while being worn on the head of the user 2. The glasses-type terminal 1 is worn so as to sandwich the head of the user 2. At this time, in the glasses-type terminal 1, a part called an arm part is in contact with the user's both ears 2 a. A user 2 wearing the glasses-type terminal 1 observes an image through a lens of the glasses unit 10. The user 2 described below is assumed to be in a state of wearing the glasses-type terminal 1 unless otherwise specified.

  The glasses unit 10 of the glasses-type terminal 1 includes an imaging unit 11, a display unit 12, a distance sensor 13, and a position / direction sensor 14 as an electrical configuration. The imaging unit 11 includes an imaging element such as a CCD (Charge Coupled Device) and can capture moving images and still images. The imaging unit 11 captures an image observed by the user through the glasses unit 10. The photographing unit 11 is disposed at a position where the position (range) of the image to be photographed can be regarded as coincident with the position (range) of the image observed by the user 2 through the glasses unit 10. When shooting a moving image or a still image, the shooting unit 11 generates shooting data representing the shot image (that is, the shot image).

  The display unit 12 includes an optical see-through type head mounted display (HMD), and displays video content superimposed on an image observed by the user through the glasses unit 10. Here, the display unit 12 displays the video content so as to overlap the lens on the left eye side of the glasses unit 10 so that the video is visually recognized by the user's left eye.

The distance sensor 13 is, for example, an optical or ultrasonic distance sensor, and is a distance detection unit that detects a distance to a target in front of the user 2 (for example, the direction of the line of sight of the user 2).
The position / direction sensor 14 is a nine-axis sensor including, for example, three-axis acceleration, three-axis angular velocity, and three-axis geomagnetism, and is a position / direction detection unit that detects the position of the glasses-type terminal 1 and the direction in which the glasses-type terminal 1 faces.

FIG. 2 is a diagram for explaining an overview of the video presentation function of the glasses-type terminal 1.
The glasses-type terminal 1 sets (defines) a virtual area (hereinafter referred to as “video presentation area”) for presenting video content in the actual environment observed by the user 2. Then, the glasses-type terminal 1 displays the video content on the display unit 12 so that the video content is observed by the user 2 in the set video presentation area. Only the user 2 wearing the glasses-type terminal 1 can observe the video content presented in the video presentation area. FIG. 2 shows an example in which the video presentation areas SC1 to SC4 are set in the room space 100 where the user 2 is present. The room space 100 is formed by being surrounded by the ceiling C, the side wall W, and the floor F. For example, the user 2 can observe the video content in the video presentation area SC1, SC2, or SC3 when the user looks at the side wall W, and the user 2 looks at the ceiling C toward the video presentation area SC4. Video content can be observed. In addition to this, when a video presentation area is set on a desk or other furniture installed on the floor F or the room space 100, the user 2 observes video content by directing his / her line of sight to the video presentation area. Can do. As can be seen from the video presentation areas SC1 to SC4 shown in FIG. 2, the video presentation area of the present embodiment is a rectangular area (including a square).

FIG. 3 is a block diagram illustrating a hardware configuration of the glasses-type terminal 1. As shown in FIG. 3, the glasses-type terminal 1 includes a glasses unit 10, a control unit 20, a storage unit 30, and a communication unit 40.
When the photographing unit 11 of the eyeglass unit 10 shoots and generates shooting data, the generated shooting data is supplied to the control unit 20. The display unit 12 displays video content according to display control performed by the control unit 20. For example, the distance sensor 13 detects the distance to the target when the control unit 20 sets the video presentation area, and supplies the detection result to the control unit 20. The position / direction sensor 14 repeatedly detects the position of the eyeglass-type terminal 1 and the direction in which the eyeglass-type terminal 1 faces during operation of the eyeglass-type terminal 1, for example, and supplies the detection result to the control unit 20.

  The control unit 20 includes a microcomputer having a central processing unit (CPU), a read only memory (ROM), and a random access memory (RAM). The CPU controls each unit of the glasses-type terminal 1 by reading a program stored in the ROM or the storage unit 30 into a RAM as a work area and executing the program. The storage unit 30 is a storage unit having, for example, an EEPROM (Electrically Erasable Programmable ROM), and stores an application program executed by the control unit 20 and a setting table 31. The application program stored in the storage unit 30 is, for example, an application program, a browser, or a mailer for reproducing moving image content or still image content. These application programs are application programs for displaying video. The communication unit 40 is an interface for communicating with an external device (typically wireless communication).

FIG. 4 is a diagram illustrating a configuration example of the setting table 31. The setting table 31 is a data table that stores information related to the set video presentation area.
As shown in FIG. 4, in the setting table 31, a “record number” field, a “function” field, and a “setting information” field are associated with each other. In the “record number” field, a number for identifying a record in the setting table 31 is stored. Each record of record numbers “1” to “4” shown in FIG. 4 stores information related to the video presentation area having the same code end value among the video presentation areas SC1 to SC4 described in FIG. The “function” field stores information for identifying a function assigned to each video presentation area. In the present embodiment, the “function” field stores an identifier (hereinafter referred to as “application identifier”) that identifies an application program to be executed in order to use the function. In order to make it easy to understand the functions assigned to each video presentation area, the “function” field in FIG. 4 shows a character string describing the function.

  In the “setting information” field, setting information indicating the setting of the video presentation area is stored. Specifically, the “setting information” field includes a “direction information” field, a “size information” field, and a “tag information” field. The “direction information” field stores direction information indicating the direction in which the glasses-type terminal 1 is facing when the video presentation area is set. In the “size information” field, size information indicating the vertical and horizontal sizes of the video presentation area is stored. In the “tag information” field, tag information associated with the video presentation area as a tag is stored. Specifically, the tag information is photographing data representing a photographed image (here, a still image) in the video presentation area.

FIG. 5 is a functional block diagram showing a functional configuration of the control unit 20 of the glasses-type terminal 1. As illustrated in FIG. 5, the control unit 20 realizes functions corresponding to the imaging control unit 21, the gesture recognition unit 22, the setting unit 23, and the display control unit 24. The setting information acquisition unit 25 and the correction unit 26 illustrated in FIG. 5 are functions related to a modified example described later, and are not related to the present embodiment.
The shooting control unit 21 controls the shooting of the shooting unit 11. For example, the shooting control unit 21 causes the shooting unit 11 to shot a moving image when setting a video presentation area. In addition, the shooting control unit 21 causes the shooting unit 11 to take a still image of the video presentation area set by the setting unit 23 and generates tag information.

  The gesture recognition unit 22 recognizes a gesture that surrounds the spatial area made by the user 2 in a plane based on the shooting data generated by the shooting unit 11. FIG. 6 is a diagram illustrating a gesture performed by the user 2. As shown in FIG. 6, the user 2 creates an “L” shape with the thumb FR1 and the index finger FR2 of the right hand, creates an “L” shape with the thumb FL1 and the index finger FL2 of the left hand, and combines them into a planar shape. Create a space area that is approximately rectangular. The gesture recognizing unit 22 recognizes the gesture of the user 2 described with reference to FIG. 6 and detects a spatial region specified by the gesture. The spatial area detected by the gesture recognition unit 22 is hereinafter referred to as “gesture area”.

  Based on the gesture recognized by the gesture recognition unit 22, the setting unit 23 sets the video presentation area on the extension line of the gesture area as viewed from the position of the user 2. The position of the user 2 is, for example, the position of the viewpoint of the user 2, but is regarded as the position of the photographing lens of the photographing unit 11 unless otherwise specified. When setting the video presentation area, the setting unit 23 stores setting information indicating the setting of the video presentation area in the setting table 31 in association with the function assigned to the video presentation area. Further, the setting unit 23 stores the direction information specified based on the detection result of the position / direction sensor 14 in the setting table 31.

  The setting unit 23 includes a planar area detection unit 23a. The plane area detection unit 23a detects a plane area existing (existing) in front of the user 2 based on the detection result of the distance sensor 13 or the shooting data generated by the shooting unit 11. This planar region is, for example, the ceiling C, the side wall portion W, or the floor F that constitutes the room space 100. The setting unit 23 sets a video presentation area on the plane area detected by the plane area detection unit 23a.

  Based on the setting table 31, the display control unit 24 causes the display unit 12 to display the video content so that the video content can be observed by the user 2 in the video presentation area set by the setting unit 23. The display control unit 24 displays the video content on the display unit 12 so as to give the user 2 a feeling (for example, perspective) that the video content is projected (projected) in the video presentation area.

FIG. 7 is a flowchart showing a flow of processing relating to the setting of the video presentation area performed by the glasses-type terminal 1.
First, the control unit 20 of the glasses-type terminal 1 executes an application program (step S1). Here, the control unit 20 executes an application program for displaying video content on the display unit 12 in accordance with an instruction from the user 2.

Next, the control unit 20 recognizes the three-dimensional space around the glasses-type terminal 1 (step S2). In order to recognize the three-dimensional space in front of the user 2, the control unit 20 acquires a three-dimensional point group by SfM (Structure from Motion) based on, for example, shooting data indicating a moving image generated by the shooting unit 11. Specifically, the control unit 20 detects a feature point in the captured image represented by the captured data according to, for example, a known Harris operator. Next, the control unit 20 tracks the feature points according to, for example, a KLT (Kanade Lucas Tomasi) tracker based on the imaging data of a plurality of frames continuous on the time axis. And the control part 20 estimates the three-dimensional position of a feature point by SfM from the tracked feature point.
As another method, for example, the control unit 20 may recognize the three-dimensional space based on the detection result of the distance detected by the distance sensor 13, or recognize the three-dimensional space by a method using a stereo camera. May be.

  Next, the control part 20 recognizes the gesture performed by the user 2 (step S3). The control unit 20 recognizes the gesture described with reference to FIG. 6 performed by the user 2 at a certain time (instant) based on, for example, shooting data of one frame. As shown in FIG. 8A, the user 2 performs this gesture in front of his / her eyes and beyond the lens of the spectacle unit 10. The user 2 designates the position and size of the video presentation area by performing a gesture for creating the area indicated by the hatched portion in FIG.

In the process of step S <b> 3, the control unit 20 may recognize a gesture made by the user 2 using a known gesture recognition technique. For example, there is a gesture recognition technology called SixSense (For SixSense, see http://www.pranavmistry.com/projects/sixthsense/ (See 08 / tedmitpattie-maessixth-sense.html.) In this gesture recognition technology, the user attaches a marker to each of the thumb and index finger of both hands and performs the gesture described with reference to FIG. The camera held by the user recognizes a gesture made by the user by detecting each marker from the captured image. Apart from this, there is a gesture recognition technology using Ubi-Camera (refer to http://japanese.engadget.com/2012/03/30/ubi-camera/ for Ubi-Camera). In this gesture recognition technology, the user attaches a small camera to any one finger (typically the index finger of the right hand) and performs the gesture described with reference to FIG. When the camera attached to the user's finger detects that the pressure sensor of its own device has been pressed by another finger (typically, the thumb of the left hand), the camera recognizes the gesture made by the user. When this technique is applied to the eyeglass-type terminal 1, the control unit 20 recognizes the gesture by acquiring information indicating that the photographing data or the pressure sensor is pressed from the camera, for example.
Note that the control unit 20 may recognize the gesture by another method. For example, the control unit 20 may recognize a gesture by detecting an image region of a skin color or a color close to the skin color from a captured image represented by the captured data and performing image analysis.

Returning to the description of FIG.
The control unit 20 recognizes the direction in which the glasses-type terminal 1 faces and the size of the gesture area (step S4). First, the control unit 20 recognizes the direction in which the glasses-type terminal 1 is directed based on the direction detected by the position / direction sensor 14 when the gesture is recognized. Further, as shown in FIG. 8B, the control unit 20 crosses the point P1 at the position where the right thumb FR1 and the left index finger FL2 intersect, and the right index finger FR2 and the left thumb FL1. The point P2 at the position is recognized. Then, the control unit 20 detects a rectangle (a hatched portion in FIG. 8B) having a line segment connecting the points P1 and P2 as a diagonal line as the gesture region T. Here, the control unit 20 recognizes the length “A” of the line segment connecting the points P1 and P2 as the size of the gesture region T.
As another method, for example, the control unit 20 may detect the gesture region T by specifying a circumscribed rectangle or an inscribed rectangle in the shaded region (graphic) shown in FIG.

  Next, the control unit 20 detects a plane area in front of the user 2 (step S5). For example, the control unit 20 detects a planar area based on the recognition result (feature point group) of the three-dimensional space in front of the user 2 recognized in the process of step S2 (see the description of Non-Patent Document 1). As another method, the control unit 20 may detect the planar area based on the distance detection result by the distance sensor 13. For example, when the distance to the target in front of the user 2 is equal to or less than the threshold value, the control unit 20 detects the planar area at a position that is further forward by that distance.

  Next, the control unit 20 determines whether or not a planar area has been detected (step S6). When the control unit 20 determines that the plane area is not detected in the process of step S5 (step S6; NO), the control unit 20 returns to the process of step S2. On the other hand, if the control unit 20 determines that a plane area has been detected in the process of step S5 (step S6; YES), it sets a video presentation area on the detected plane area (step S7).

FIG. 9 is a diagram illustrating a method for setting a video presentation area. FIG. 9A shows a diagram for explaining the positional relationship among the imaging unit 11, the gesture region T, and the side wall part W when the video presentation area SC is set on the side wall part W. FIG. 9B shows the positional relationship between the imaging unit 11, the gesture region T, and the side wall W when viewed in a plan view in the direction of arrow I in FIG. 9A, that is, in the horizontal direction.
In setting the video presentation area described below, the following three items are assumed.
(Assumption 1) The position of the user 2 (that is, the position of the viewpoint) is the same as the position of the photographing unit 11 (the position of the lens).
(Assumption 2) The position of the user 2 (that is, the position of the viewpoint) and the center of gravity of the gesture area are the same height.
(Assumption 3) The side wall W and the plane direction of the gesture region are parallel.

  As shown in FIG. 9A, the length of the gesture region T in the horizontal direction (here, the horizontal direction) is “Ax” and the length of the vertical direction (here, the vertical direction) is “Ay”. Further, as shown in FIG. 9B, the length between the position of the photographing unit 11 and the center of gravity of the gesture region T is “B”. The length B may be measured using a distance sensor (not shown), or may be determined in advance based on the likelihood. The length between the center of gravity of the gesture region T and the side wall W is “C”. For the length C, the detection result of the distance sensor 13 is used.

The control unit 20 sets the video presentation area SC on the extension line of the gesture area T when viewed from the position of the user 2. Specifically, the control unit 20 sets a point Pe1 at a point that intersects the side wall portion W on the extension line of the line connecting the position of the imaging unit 11 and the point P1, and sets the position of the imaging unit 11 and the point P2. A point Pe2 is set at a point intersecting the side wall W on the extended line connecting the two. And the control part 20 sets the rectangular area | region which makes the line segment which connects the point Pe1 and the point Pe2 a diagonal line as video presentation area | region SC. Here, when the horizontal length of the video presentation area SC is “Dx”, the vertical length is “Dy”, and the length of the line segment connecting the points Pe1 and Pe2 is “D” The control unit 20 sets the video presentation area SC so as to satisfy the relationship of the following formula (1).
Dx = Ax * C / B, Dy = Ay * C / B, D = A * C / B (1)

  According to the method for setting the video presentation area SC described with reference to FIG. 9, a square weight composed of the position of the user 2 and the gesture area T and a square weight composed of the position of the user 2 and the video presentation area SC are obtained. It will be similar. Further, as can be seen from the equation (1), the control unit 20 enlarges the size of the video presentation area SC by multiplying the gesture area T vertically and horizontally by a magnification according to the ratio of the length B to the length C. It will be decided.

  By the way, in the video presentation area setting method described with reference to FIG. 9, the difference between the position of the photographing unit 11 and the position of the viewpoint of the user 2 is ignored by providing (Assumption 1). However, the position of the photographing unit 11 and the position of the viewpoint of the user 2 do not exactly match. Therefore, the control unit 20 sets parameters obtained from the relative positional relationship between the position of the photographing unit 11 when the user 2 wears the eyeglass-type terminal 1 and the position of the viewpoint of the user 2 (for example, the It is desirable to perform processing for converting the captured image of the imaging unit 11 on the captured data based on the external parameter. In this case, the control unit 20 may perform the conversion process so that the captured image of the imaging unit 11 is the same as the image observed from the viewpoint of the user 2.

Regarding (Assumption 2) and (Assumption 3), the position of the photographing unit 11 and the center of gravity of the gesture region T are not at the same height, and the photographing unit 11 may be tilted up and down. In this case, the plane area (side wall W) where the video presentation area is set and the plane direction of the gesture area T are not parallel. In this case, (Assumption 2) and (Assumption 3) do not hold. Therefore, as shown in FIG. 10, when the photographing unit 11 is tilted downward at an angle θ from the horizontal direction, the control unit 20 sets the video presentation area SC ′ so as to satisfy the relationship of the following formula (2).
Dx ′ = Dx, Dy ′ = Dy * cos θ (2)

In Expression (2), the horizontal length of the video presentation area SC ′ is “Dx ′”, and the vertical length is “Dy ′”. The value of θ is calculated using, for example, a sensor that is provided in the glasses-type terminal 1 and detects the posture of the photographing unit 11 or the glasses-type terminal 1. The sensor that detects the posture of the photographing unit 11 or the glasses-type terminal 1 is, for example, an acceleration sensor or a gyro sensor.
Even when the photographing unit 11 is tilted upward, the control unit 20 may set the video presentation area SC ′ by the method described in FIG.

  By the way, the viewing angle of the display unit 12 is generally smaller than the human field of view and the angle of view of the photographing unit 11. For this reason, the viewing angle required in the quadrangular pyramid connecting the gesture region T described in FIG. 8B and the line of sight of the user 2 (in this case, horizontal viewing angle = 2 arctan (Ax / 2C), vertical viewing angle = 2 arctan (Ay / 2C)) may exceed the viewing angle of the display unit 12 in some cases. In this case, the control unit 20 may set the video presentation area with the viewing angle of the display unit 12 as an upper limit. In this setting method, the video content is observed from the user 2 on a screen smaller than the requested size, but the video content on the entire screen can be observed at a time. Alternatively, the control unit 20 may set the video presentation area to be larger than the viewing angle of the display unit 12 as requested by the user, and the video content displayed on the display unit 12 may be a part of the screen instead of the entire screen. In the case of this setting method, the video content is observed from the user 2 in the requested size, and only a part of the video content on the screen is observed from a fixed location. In order to observe the video content on the entire screen, the user 2 moves the range of the video presentation area in which the user 2 moves and enters the field of view.

Returning to FIG. 7, next, the control unit 20 causes the photographing unit 11 to photograph the set video presentation area. Here, the control unit 20 causes the photographing unit 11 to photograph a portion corresponding to the video presentation area SC in the side wall portion W so that the user's 2 hand and the like are not reflected. And the control part 20 stores the imaging | photography data showing the still image produced | generated by the imaging | photography part 11 as tag information in the same record as the process of step S7 of the setting table 31 (step S8).
And the control part 20 stores the direction information which shows the direction which the position direction sensor 14 detected by the process of step S4 in the same record as the process of step S7, S8 of the setting table 31 (step S9).
The above is the description of the operation of the glasses-type terminal 1 when setting the video presentation area.

FIG. 11 is a flowchart showing a flow of processing performed when the glasses-type terminal 1 presents video content to the user 2. Hereinafter, the operation of the glasses-type terminal 1 when presenting video content in the video presentation area SC1 described in FIG. 2 will be described. During the following operations, the control unit 20 causes the imaging unit 11 to capture a moving image.
The control unit 20 determines whether the user 2 is observing the video presentation area (step S11). The control unit 20 collates the captured image of one frame represented by the photographing data generated by the photographing unit 11 with the photographed image represented by each tag information stored in the setting table 31 using, for example, a known pattern matching technique. To do. And the control part 20 judges whether the video presentation area | region corresponding to which tag information is observed by the user 2 based on both identity (for example, coincidence degree).
In the process of step S <b> 11, the control unit 20 may set the tag information of all records in the setting table 31 as a search target. You may narrow down. For example, the control unit 20 searches for a record including direction information indicating a predetermined range with reference to the direction in which the glasses-type terminal 1 is currently facing. By narrowing down the search target, the processing amount of the control unit 20 relating to the processing of step S1 is reduced.

  When the user 2 determines that the video presentation area is being observed (step S11; YES), the control unit 20 specifies the display position of the video content in the video presentation area being observed (step S12). This display position is a position on the display unit 12. The control unit 20 determines the positional relationship between the image indicated by the tag information stored in the setting table 31 and the video presentation area corresponding to the tag information, and the positional relationship between the current captured image of the imaging unit 11 and the video presentation area. The display position of the video content is specified so that is the same (maintained). That is, the control unit 20 maintains a positional relationship that gives the user 2 who observes the video presentation area a feeling that the video presentation area is fixed to a planar area such as the side wall W.

  And the control part 20 displays a video content in the display position of the display part 12 specified by the process of step S12 so that a video content can be observed by the user 2 in a video presentation area (step S13). The control unit 20 specifies the function assigned to the video presentation area being observed based on the setting table 31, and displays the video content based on the application program for using the specified function. The control unit 20 may execute an application program used for displaying video content in advance or may be executed in the process of step S13. As shown in FIG. 12A, when the user 2 is observing the entire video presentation area SC1, the user 2 displays the entire video content of the function assigned to the video presentation area SC1 in the video presentation area SC1. It can be observed as if it is displayed on the (side wall W).

  Next, the control unit 20 determines whether or not the user 2 is observing the video presentation area (step S14). When the control unit 20 determines that the user 2 is observing the video presentation area (step S14; YES), the control unit 20 returns to the process of step S12. During observation of the video presentation area, the control unit 20 repeatedly executes the processing steps of steps S12 to S14.

Here, the operation of the glasses-type terminal 1 when the user 2 changes the direction of the line of sight will be described.
It is assumed that the user 2 who has been viewing the direction described in FIG. 12A changes his / her line of sight in the upper right direction, for example. In this case, the control unit 20 changes the display position of the video content in the video presentation area SC1. As shown in FIG. 12B, when the user 2 observes only the upper right part of the video presentation area SC1, the control unit 20 observes only the video content of the part corresponding to the observed part. As described above, the display position of the video content is changed. It is assumed that the user 2 who was looking at the direction shown in FIG. 12B changed the line of sight for example and did not observe the video presentation area SC1 at all. In this case, the control unit 20 does not display the video content in the video presentation area SC1 as shown in FIG. At this time, the user 2 is in a state of observing only the real space.
Thereafter, when the control unit 20 determines in the process of step S14 that the video presentation area is not being observed (step S14; NO), the display of the video content is terminated (step S15).

The user 2 wearing the glasses-type terminal 1 of the embodiment described above can place an image in the real environment observed by the user 2 by a gesture performed in front of the eyes. For this reason, the user 2 arranges the virtual video presentation area with an intuitive and simple gesture, and further, this video is displayed as if the video content is displayed on the display in the video presentation area. Content can be observed.
Further, as described with reference to FIG. 2, the glasses-type terminal 1 can simultaneously set a plurality of video presentation areas and present different video contents. For this reason, the user 2 can observe the video content of a plurality of functions executed by multitasking while changing the direction of the line of sight and the position of the user.
Moreover, in a general-purpose HMD, an image is always displayed in front of the eyes when worn by the user, and for example, it may cause eyestrain for the user 2. Further, when the user 2 is temporarily prevented from viewing the video, the user needs to interrupt the playback of the video or to remove the attached HMD once. On the other hand, according to the glasses-type terminal 1, the video content is not observed unless the user 2 observes the place where the video presentation area is set (see FIG. 12C). For this reason, the effect of reducing the use burden for the user 2 can also be expected.

[Modification]
The present invention can be implemented in a form different from the above-described embodiment. The present invention can also be implemented in the following forms, for example. Further, the following modifications may be combined as appropriate.
(Modification 1)
When displaying the moving image content, the control unit 20 may reproduce the moving image content only during the period during which the user 2 is observing. That is, if the entire video presentation area is no longer observed by the user 2 during playback of the video content, the control unit 20 of the glasses-type terminal 1 interrupts playback of the video content observed in the video presentation area. (Pause). Thereafter, when the user 2 observes at least a part of the video presentation area again by the user 2, the control unit 20 resumes the reproduction of the video content that has been interrupted. As another method, when at least a part of the video presentation area is no longer observed by the user 2, the control unit 20 interrupts the reproduction of the moving image content, and when the entire video presentation area is observed again, The playback of the video content may be resumed. According to the glasses-type terminal 1 of this modification, it is possible to reduce the possibility that the user 2 partially misses the moving image content.
In this modification, the control unit 20 may play back video content other than video content only during a period during which the user 2 is observing. For example, the control unit 20 interrupts the processing (work) using the application program during the period when the user 2 does not observe the video presentation area associated with the designated application program, and the video content The display of is interrupted. Whether the process is interrupted when the user 2 is not observing the video presentation area may be set in advance for each application program, or may be explicitly specified by the user 2 sequentially. Examples of the designation method when the user 2 designates include a method using a gesture using a hand or a finger and a method using voice input.

(Modification 2)
When a plurality of video presentation areas are observed simultaneously by the user 2, the control unit 20 displays the video content so that the video content is observed in any one of the video presentation areas, and displays the video content in the other video presentation areas. You may make it not display. The control unit 20 may determine the video presentation area in which the video content is displayed based on any condition. For example, the video presentation with the largest area among the plurality of video presentation areas observed by the user 2 is provided. The video content is observed in the area, the video presentation area arranged at the position closest to the center in the captured image of the imaging unit 11, or the video presentation area having the highest priority function.

In addition, when a plurality of video presentation areas are observed simultaneously by the user 2, the control unit 20 displays video content in any one video presentation area, and the video contents in the other video presentation areas are observed by the user 2. You may perform control which makes it difficult to do. For example, the control unit 20 performs video processing for increasing the transparency of video content that is difficult to be observed by the user 2. In addition to this, the control unit 20 performs video processing that lowers the resolution, temporarily reduces the size of the video presentation area, or changes the color for video content that is difficult to be observed by the user 2. You may give it.
According to the glasses-type terminal 1 of this modification, it is possible to suppress a reduction in attention to the video content being presented due to the user 2 observing a plurality of video content at the same time.

(Modification 3)
The glasses-type terminal 1 may have a function of sharing a video presentation area and video content with another video display device (here, another glasses-type terminal; hereinafter referred to as “other device”) that is not its own device. Good. Other glasses-type terminals have the same configuration as the glasses-type terminal 1.
In addition to the functional configuration described with reference to FIG. 5, the control unit 20 of the glasses-type terminal 1 according to this modification further realizes a function corresponding to the setting information acquisition unit 25. The setting information acquisition unit 25 acquires setting information indicating the setting of the video presentation area set by another device via the communication unit 40. The setting information acquired by the setting information acquisition unit 25 may be the same as the information stored in the “setting information” field described with reference to FIG. The setting information acquisition unit 25 stores the acquired setting information in association with the application identifier in the same record of the setting table 31.
The display control unit 24 sets a video presentation area based on the setting information acquired by the setting information acquisition unit 25, and displays the same video content as that of other devices in the video presentation area.

FIG. 13 is a flowchart showing a flow of processing performed when the glasses-type terminal 1 sets a video presentation area based on setting information acquired from another device.
The control unit 20 acquires setting information and an application identifier from another device, stores them in the setting table 31 in association with each other (step S21). Next, the control unit 20 sets a video presentation area based on the setting information acquired in step S21, and displays the same video content as that of the other device in the video presentation area (step S22). When the same video content as that of the other device is stored in the storage unit 30, the control unit 20 causes the display unit 12 to display the video content read from the storage unit 30. Further, the control unit 20 may cause the display unit 12 to display video content acquired from another device or the like via the communication unit 40. Thereafter, the control unit 20 executes the processing steps of Steps S11 to S15 in the same procedure as the above-described embodiment.

  In this modification, it is desirable that the video content observed by the user is synchronized between the glasses-type terminal 1 and the other device. Therefore, when reproducing moving image content, the control unit 20 may reproduce the moving image content so as to synchronize with other devices. In order to realize synchronized playback, for example, the display control unit 24 acquires a synchronization signal shared with another device via the communication unit 40, and plays back video content based on the acquired synchronization signal. According to the glasses-type terminal 1, it is possible to give a feeling of use as if a plurality of users are observing the same image at the same time.

(Modification 4)
The control unit 20 may have a function of correcting (for example, fine adjustment) the set video presentation area based on a gesture performed by the user 2.
In addition to the functional configuration described with reference to FIG. 5, the control unit 20 of the glasses-type terminal 1 according to this modification further implements a function corresponding to the correction unit 26. The correction unit 26 corrects the video presentation area set by the setting unit 23 based on the gesture recognized by the gesture recognition unit 22. When correcting the video presentation area, the correction unit 26 rewrites the setting information stored in the setting table 31. The gesture recognition unit 22 recognizes a gesture for correcting the video presentation area when the video presentation area set by the setting unit 23 is displayed on the display unit 12.

FIG. 14 is a flowchart showing a flow of processing performed when the glasses-type terminal 1 corrects the video presentation area.
When the control unit 20 executes the processing steps of Steps S1 to S7 and sets the video presentation area, the control unit 20 displays the set video presentation area on the display unit 12 (Step S101). The control unit 20 may display video content in the video presentation area or display an image indicating the range of the video presentation area. And the control part 20 correct | amends this video presentation area based on the gesture of the user 2 recognized during the display of a video presentation area (step S102). For example, when the user 2 performs the gesture described with reference to FIGS. 6 and 8 on the video presentation area that has been set, the control unit 20 recognizes this gesture and corrects the video presentation area. At this time, the control unit 20 may reset the video presentation area by the same method as in the above-described embodiment. And the control part 20 rewrites the information of the setting table 31 according to the correction result of an image | video presentation area | region.
The control unit 20 may correct the video presentation area based on another gesture by the user 2. For example, when the finger is moved in a predetermined direction with the finger placed on the correction target video presentation area, the control unit 20 performs correction to move the video presentation area in the movement direction of the finger.

(Modification 5)
In the embodiment described above, the control unit 20 detects a plane area in front of the user 2 and sets a video presentation area on the detected plane area. However, when the user 2 is in a large room space, there may be no plane area in the vicinity. In this case, the control unit 20 may set the video presentation area at a predetermined distance in front of the user 2. The method for setting the video presentation area of the glasses-type terminal 1 in this case may be the same as that of the above-described embodiment, and the video presentation area is assumed on the assumption that the planar area exists at a position separated by a predetermined distance in the real space. Should be set. For example, the control unit 20 sets the video presentation area at the position of the infinite edge of the lens of the spectacles unit 10 or a predetermined fixed distance.
In this modification, the configuration related to the detection of the planar area in the glasses-type terminal 1 (for example, the function of the planar area detection unit 23a) may be omitted.

(Modification 6)
The glasses-type terminal 1 may not include the distance sensor 13. In this case, the control unit 20 may analyze the shooting data generated by the shooting unit 11 to recognize a three-dimensional space, detect a plane area, or detect a distance to the plane area.
The glasses-type terminal 1 may be configured not to store tag information. In this case, the glasses-type terminal 1 specifies the position and direction when the video presentation area is set based on the detection result of the position / direction sensor 14 and stores the position information and the direction information in the setting table 31. Then, the glasses-type terminal 1 collates the current detection result of the position / direction sensor 14 with the position information and the direction information stored in the setting table 31, and determines the position of the video presentation area based on the identity of both. Identify.
The glasses-type terminal 1 may include a direction sensor that detects the direction in which the glasses-type terminal 1 faces instead of the position / direction sensor 14. This direction sensor is, for example, an acceleration sensor, a geomagnetic sensor, or a gyro sensor.
The glasses-type terminal 1 may not include the position / direction sensor 14. In this case, the glasses-type terminal 1 specifies the position of the video presentation area using the tag information without using the direction information.

  The function (application program) assignment method for the video presentation area performed by the control unit 20 of the glasses-type terminal 1 described in the above-described embodiment is merely an example. For example, the control unit 20 may assign a function according to a predetermined operation performed by the user 2 after setting the video presentation area. The control unit 20 may change the function to be assigned after setting the video presentation area and storing the information in the setting table 31.

  Moreover, the control part 20 may determine a video presentation area | region by relationships other than the relationship demonstrated by Formula (1) and (2). For example, the control unit 20 recognizes two points (P1, P2) where fingers of both hands of the user 2 intersect to detect a gesture region that is a rectangular region, but recognizes four points that are vertices of the rectangular region. Then, the gesture area may be detected. The control unit 20 may set a video presentation area having the same size as the gesture area. In this case, the user 2 designates the desired size of the video presentation area with the gesture using the right hand and the left hand, and the control unit 20 recognizes this gesture and detects the gesture area.

(Modification 7)
The control unit 20 of the eyeglass-type terminal 1 according to the above-described embodiment recognizes a gesture surrounding the space area performed by the user 2 and sets the video presentation area. The control unit 20 may recognize another gesture and set the video presentation area. For example, when the user 2 performs a gesture for designating the presentation position of the video content by a point in the space area, the control unit 20 calculates the size of the video presentation area that the user 2 can visually recognize. And the control part 20 sets the image | video presentation area | region of the calculated size on the extension line of the presentation position which the user 2 designated seeing from the user 2 position. At this time, the control unit 20 may calculate the size of the video presentation area using the distance to the planar area such as the side face W.
That is, the control unit 20 may set the video presentation area in a size that allows the user 2 to observe the video content on the extension line of the spatial area specified by the user 2 gesture as viewed from the position of the user 2. Various modifications can be made to the specific gesture and the relationship between the gesture and the video presentation area.

(Modification 8)
The specific shape of the glasses-type terminal 1 (glasses unit 10) is not limited to the shape shown in FIG.
The video display device of the present invention is not limited to a glasses-type video display device, and may be any video display device having a function of displaying a video image superimposed on an image observed by a user. For example, the video display device of the present invention may be a video display device that is held and used by a user, or a video display device that is used by being attached to an attachment such as a helmet worn by the user. There may be.

(Modification 9)
The function of the control unit 20 of the glasses-type terminal 1 described above may be realized by any of hardware resources, software resources, or a combination thereof. When the function of the control unit 20 is realized using a program, the program includes a magnetic recording medium (magnetic tape, magnetic disk (HDD (Hard Disk Drive), FD (Flexible Disk)), etc.), optical recording medium (optical disk). Etc.), may be provided in a state of being stored in a computer-readable recording medium such as a magneto-optical recording medium or a semiconductor memory, or may be distributed via a network. The present invention can also be implemented as a video presentation method performed by a computer.

DESCRIPTION OF SYMBOLS 1 ... Eyeglass-type terminal, 2 ... User, 10 ... Glasses part, 11 ... Imaging | photography part, 12 ... Display part, 13 ... Distance sensor, 14 ... Position direction sensor, 20 ... Control part, 21 ... Shooting control part, 22 ... Gesture Recognizing unit, 23 ... setting unit, 23a ... plane area detecting unit, 24 ... display control unit, 25 ... setting information acquisition unit, 26 ... correction unit, 30 ... storage unit, 31 ... setting table, 40 ... communication unit.

Claims (12)

An imaging unit that captures an image observed by the user and generates imaging data;
A display unit that displays video content superimposed on the observed image;
A gesture recognition unit for recognizing a gesture made in a spatial region by the user based on shooting data generated by the shooting unit;
Based on the gesture recognized by the gesture recognition unit, a setting unit that sets a video presentation region on an extension line of the spatial region specified by the gesture as viewed from the user position;
A display control unit for displaying the video content on the display unit so that the video content is observed by the user in the video presentation area set by the setting unit;
With
The display control unit
When a plurality of the video presentation areas are observed by the user, the video content of one video presentation area is not observed by the user or is observed more than the video contents of the other video presentation areas. Film image display apparatus you and performs control to Nikuku. An imaging unit that captures an image observed by the user and generates imaging data;
A display unit that displays video content superimposed on the observed image;
A gesture recognition unit for recognizing a gesture made in a spatial region by the user based on shooting data generated by the shooting unit;
Based on the gesture recognized by the gesture recognition unit, a setting unit that sets a video presentation region on an extension line of the spatial region specified by the gesture as viewed from the user position;
A display control unit for displaying the video content on the display unit so that the video content is observed by the user in the video presentation area set by the setting unit;
And a setting information acquisition unit that acquires setting information indicating settings of the video presentation area from another image display device is not a self apparatus,
The setting unit
Set the video presentation area based on the setting information acquired by the setting information acquisition unit,
The display control unit
The other movies image display device you characterized in that display the same said video contents and the video display device. An imaging unit that captures an image observed by the user and generates imaging data;
A display unit that displays video content superimposed on the observed image;
A gesture recognition unit for recognizing a gesture made in a spatial region by the user based on shooting data generated by the shooting unit;
Based on the gesture recognized by the gesture recognition unit, a setting unit that sets a video presentation region on an extension line of the spatial region specified by the gesture as viewed from the user position;
A display control unit for displaying the video content on the display unit so that the video content is observed by the user in the video presentation area set by the setting unit;
The image presentation area where the setting unit has set, and a correcting section that corrects, based on the gesture the gesture recognition unit recognizes,
The gesture recognition unit
When the video presenting regions the setting is displayed on the display unit, film image display device you characterized by recognizing a gesture for correcting the video presentation area. The gesture recognition unit
Video display according to any one of claims 1 to 3, characterized in that recognize gestures that surrounds the space region made by the user in a plane. A plane area detecting unit for detecting a plane area existing on the extension line;
The setting unit
Wherein the plane area detection unit the flat region in which the detected image display device according to any one of claims 1 to 4 and sets the video presentation area. The display control unit
When the video presentation area is no longer observed by the user, the reproduction of the video content in the video presentation area is interrupted, and the reproduction is resumed when the video presentation area is observed again. video display according to any one of claims 1 to 5. An imaging unit that captures an image observed by the user and generates imaging data;
A display unit for displaying video content superimposed on the observed image;
A video presentation method for a video display device comprising:
A first step of recognizing a gesture made in a spatial region by the user based on photographing data generated by the photographing unit;
A second step of setting a video presentation area on an extension line of a spatial area specified by the gesture as viewed from the user's position based on the recognized gesture;
A third step of displaying the video content on the display unit so that the video content is observed by the user in the set video presentation area;
Have
In the third step, when a plurality of the video presentation areas are observed by the user, the video content of one video presentation area is not observed by the user, or other video presentation areas A video presentation method, characterized by being a step of performing control to make observation more difficult than video content. An imaging unit that captures an image observed by the user and generates imaging data;
A display unit for displaying video content superimposed on the observed image;
A video presentation method for a video display device comprising:
A first step of recognizing a gesture made in a spatial region by the user based on photographing data generated by the photographing unit;
A second step of setting a video presentation area on an extension line of a spatial area specified by the gesture as viewed from the user's position based on the recognized gesture;
A third step of displaying the video content on the display unit so that the video content is observed by the user in the set video presentation area;
A fourth step of acquiring setting information indicating the setting of the video presentation area from another video display device that is not its own device;
Have
The second step is a step of setting the video presentation area based on the acquired setting information,
The third step is a step of displaying the same video content as that of the other video display device.
A video presentation method characterized by the above. An imaging unit that captures an image observed by the user and generates imaging data;
A display unit for displaying video content superimposed on the observed image;
A video presentation method for a video display device comprising:
A first step of recognizing a gesture made in a spatial region by the user based on photographing data generated by the photographing unit;
A second step of setting a video presentation area on an extension line of a spatial area specified by the gesture as viewed from the user's position based on the recognized gesture;
A third step of displaying the video content on the display unit so that the video content is observed by the user in the set video presentation area;
A fourth step of correcting the set video presentation area based on the recognized gesture;
With
The first step is a step of recognizing a gesture for correcting the video presentation area when the set video presentation area is displayed on the display unit. An imaging unit that captures an image observed by the user and generates imaging data;
A display unit for displaying video content superimposed on the observed image;
In a computer of a video display device comprising
A first step of recognizing a gesture made in a spatial region by the user based on photographing data generated by the photographing unit;
A second step of setting a video presentation area on an extension line of a spatial area specified by the gesture as viewed from the user's position based on the recognized gesture;
A third step of displaying the video content on the display unit so that the video content is observed by the user in the set video presentation area;
A program for executing
In the third step, when a plurality of the video presentation areas are observed by the user, the video content of one video presentation area is not observed by the user, or other video presentation areas A program characterized in that it is a step of performing control to make it less observable than video content. An imaging unit that captures an image observed by the user and generates imaging data;
A display unit for displaying video content superimposed on the observed image;
In a computer of a video display device comprising
A first step of recognizing a gesture made in a spatial region by the user based on photographing data generated by the photographing unit;
A second step of setting a video presentation area on an extension line of a spatial area specified by the gesture as viewed from the user's position based on the recognized gesture;
A third step of displaying the video content on the display unit so that the video content is observed by the user in the set video presentation area;
A fourth step of acquiring setting information indicating the setting of the video presentation area from another video display device that is not its own device;
A program for executing
The second step is a step of setting the video presentation area based on the acquired setting information,
The third step is a step of displaying the same video content as that of the other video display device.
A program characterized by that. An imaging unit that captures an image observed by the user and generates imaging data;
A display unit for displaying video content superimposed on the observed image;
In a computer of a video display device comprising
A first step of recognizing a gesture made in a spatial region by the user based on photographing data generated by the photographing unit;
A second step of setting a video presentation area on an extension line of a spatial area specified by the gesture as viewed from the user's position based on the recognized gesture;
A third step of displaying the video content on the display unit so that the video content is observed by the user in the set video presentation area;
A fourth step of correcting the set video presentation area based on the recognized gesture;
A program for executing
The first step is a step of recognizing a gesture for correcting the video presentation area when the set video presentation area is displayed on the display unit.

Images