JP2017123006A - Portable information terminal and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology for enabling a user to select a partial area displayed at a display part without touching the display part.SOLUTION: In a portable information terminal 1, an azimuth inclination detection part 20 detects a direction and inclination of a display part 50. A video reproduction part 15 reproduces an omnidirectional video at least partially including a meta data area in which meta data are set. A video determination part 14 determines a display video to be displayed at the display part 50 among the omnidirectional video on the basis of the direction and inclination. An area selection part 16 selects the meta data area when at least a part of the meta data area included in the display video is displayed at a predetermined display position determined at the display part 50. An execution part 11 executes an operation specified by the meta data set in the meta data area selected by the area selection part 16.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a portable information terminal and a program executable by the portable information terminal.

  A housing for providing a simple head mounted display (HMD) that uses a display unit of a portable information terminal such as a smartphone as a video presentation unit is known. By inserting a hand-held smartphone into this housing, the user can easily enjoy virtual reality (VR) without purchasing a dedicated HMD.

US Patent Application Publication No. 2010/0079356

  Generally, the display screen of a portable information terminal such as a smartphone is a touch screen, and the display screen itself is an interface for user operation. In the technique as described in the above-mentioned document, once the portable information terminal is attached to the housing, it becomes difficult for the user to operate the portable information terminal. Although the above document also discloses providing a device for assisting operation different from the portable information terminal, it is difficult for the user to see the hand while using the portable information terminal as the HMD. For this reason, even if there is a device for auxiliary operation, it does not change that the user has difficulty operating the portable information terminal.

  The present invention has been made in view of these points, and an object of the present invention is to provide a technique that allows a user to select a partial area displayed on a display unit without touching the display unit.

  The program according to the first aspect of the present invention provides at least one azimuth / inclination detection function for detecting the orientation and inclination of the portable information terminal and a metadata area in which metadata is set to the portable information terminal that executes the program. A video playback function for playing back an omnidirectional video included in the display unit, a video determination function for determining a display video to be displayed on the display unit of the portable information terminal based on the orientation and tilt, and the display video When at least a part of the metadata area included in is displayed at a predetermined display position defined on the display unit, an area selection function for selecting the metadata area and the selected metadata area are set. And an execution function that executes an operation according to the metadata that has been performed.

  A marker display control function for displaying a marker for selecting a metadata area at a predetermined display position defined on the display unit on the portable information terminal; a metadata area included in the marker and the display video; An icon display control function for causing the display unit to display an icon indicating that the icons overlap each other may be further realized.

  The icon display control function displays an icon including information indicating a time elapsed since at least a part of a metadata area included in the display video is displayed at a predetermined display position defined in the display unit. May be displayed on the screen.

  The area selection function has a predetermined selection determination time for determining whether or not an area is selected at a predetermined display position determined in the display unit at least a part of the metadata area included in the display video. In the case where the display is continued during the elapse of time, a metadata area included in the display video may be selected.

  The metadata set in the metadata area may be data indicating content related to the subject of the omnidirectional video in the metadata area, and the execution function is the metadata area selected by the area selection function An operation of displaying content related to the subject on the display unit may be executed.

  The execution function may execute an operation of recognizing a subject in the metadata region selected by the region selection function and displaying content related to the recognized subject on the display unit.

  The area selection function may select the content when at least a part of the content displayed on the display unit is displayed at a predetermined display position defined on the display unit.

  The portable information terminal further implements a time measurement function for measuring the time at which the partial area is displayed at a predetermined display position defined on the display unit for each of the plurality of partial areas included in the omnidirectional video. The video playback function may play back the omnidirectional video in which information indicating the length of the time measured by the time measurement function is superimposed for each partial region to be measured.

  The portable information terminal according to the second aspect of the present invention includes at least a part of a display unit, an azimuth / inclination detection unit that detects the orientation and inclination of the display unit, and a metadata area in which metadata is set. A video playback unit that plays back a azimuth video, a video determination unit that determines a display video to be displayed on the display unit out of the omnidirectional video based on the orientation and tilt, and at least a metadata area included in the display video When a part is displayed at a predetermined display position defined on the display unit, an area selection unit for selecting the metadata area, and metadata set in the metadata area selected by the area selection unit And an execution unit that executes an operation specified by.

It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium, and the like are also effective as an aspect of the present invention.

  In addition, a program that causes a computer to realize each step when the above components are converted into a method is also effective as an aspect of the present invention. This program may be provided as a part of firmware incorporated in a device in order to perform basic control of hardware resources such as a portable information terminal. This firmware is stored in a semiconductor memory such as a ROM (Read Only Memory) or a flash memory in the portable information terminal, for example. In order to provide the firmware or to update a part of the firmware, a computer-readable recording medium storing the program may be provided, and the program may be transmitted through a communication line.

  ADVANTAGE OF THE INVENTION According to this invention, the technique which can select the partial area | region displayed on the display part without a user touching a display part can be provided.

It is a figure which shows the external appearance and usage example of the portable information terminal which concerns on embodiment. It is a figure which shows typically the function structure of the portable information terminal which concerns on embodiment. It is a figure for demonstrating the omnidirectional video which the video reproduction part which concerns on embodiment reproduce | regenerates. It is a figure for demonstrating selection of the metadata area | region by the area | region selection part which concerns on embodiment. It is a figure for demonstrating the metadata of the omnidirectional video which the portable information terminal which concerns on embodiment handles. It is a figure for demonstrating the virtual store which the portable information terminal which concerns on embodiment provides. It is a schematic diagram which shows the heat map which concerns on embodiment. It is a flowchart for demonstrating the flow of the area | region selection process which the portable information terminal which concerns on embodiment performs.

<Outline of the embodiment>
Fig.1 (a)-(b) is a figure which shows the external appearance and usage example of the portable information terminal 1 which concerns on embodiment. Specifically, FIG. 1A is a diagram illustrating a state in which the display unit 50 of the portable information terminal 1 according to the embodiment is displaying an image, and FIG. 1B is related to the embodiment. It is a figure which shows the suitable usage example of the portable information terminal. Hereinafter, the outline of the portable information terminal 1 according to the embodiment will be described with reference to FIG.

  The portable information terminal 1 according to the embodiment is a device that can execute a program for displaying an omnidirectional video. Specifically, the mobile information terminal 1 is a smartphone or a tablet PC (Personal Computer) as shown in FIG. It is. Here, the “omnidirectional image” is an image having an amount of information sufficient to be projected onto a spherical screen or virtual screen. An omnidirectional video can be acquired by photographing using a known omnidirectional camera.

  As shown to Fig.1 (a), the display part 50 with which the portable information terminal 1 which concerns on embodiment is provided is a plane, and it is difficult to display all the omnidirectional videos which are spherical shapes at once. For this reason, the portable information terminal 1 displays a part of the omnidirectional video on the display unit 50. At this time, the portable information terminal 1 changes the image displayed on the display unit 50 in conjunction with the orientation and tilt of the display unit 50.

  As shown in FIG. 1B, the portable information terminal 1 according to the embodiment can be housed and used in a housing 70 for housing the portable information terminal 1. The casing 70 has a substantially rectangular parallelepiped shape made of paper or resin, and an opening is provided on one surface. The housing 70 is configured such that when the user P looks into the opening, a space for shielding external light is formed in front of the user P's eyes. A holding unit (not shown) for inserting and holding the portable information terminal 1 is provided on the opposite side of the opening of the housing 70. Therefore, when the portable information terminal 1 is inserted into the holding unit so that the display unit 50 faces the inside of the housing 70, an image displayed on the display unit 50 in front of the eyes of the user P looking through the housing 70 is presented. That is, the housing 70 can realize a simple HMD by using the portable information terminal 1.

  By using the housing 70, a virtual reality experience with a deep immersion feeling can be provided to the user P. When the user P changes the orientation and inclination of the face while looking through the housing 70, the video displayed on the display unit 50 is also changed in conjunction with the change. Thereby, the user P can experience as if it is in the image | video which the display part 50 displays.

  On the other hand, when the housing 70 is used, it becomes difficult for the user P to touch the display unit 50 of the portable information terminal 1 or to press a button or the like provided in the portable information terminal 1. Generally, the display unit 50 of the portable information terminal 1 is a touch screen, and the display unit 50 is also a user interface for the user P to operate the portable information terminal 1. When the user P taps and selects an operation icon or the like displayed on the touch screen, the portable information terminal executes an operation associated with the icon. However, in a state where the portable information terminal 1 is housed in the housing 70 and used, it becomes difficult for the user P to operate the portable information terminal 1 via the touch screen.

For this reason, in the portable information terminal 1 according to the embodiment, when a selectable area such as an icon in the video displayed on the display unit 50 is displayed at a predetermined location set in the display area of the display unit 50, The selectable area is selected. Accordingly, the user P can select an icon or the like without touching the display unit 50 by changing the orientation and inclination of the display unit 50 and moving the display position of the selectable region.
Hereinafter, the portable information terminal 1 according to the embodiment will be described in more detail.

<Functional configuration of portable information terminal>
FIG. 2 is a diagram schematically illustrating a functional configuration of the portable information terminal 1 according to the embodiment. The portable information terminal 1 according to the embodiment includes a control unit 10, an azimuth / inclination detection unit 20, a storage unit 30, a temporary storage unit 40, a display unit 50, and a communication unit 60. The control unit 10 includes an execution unit 11, an icon display control unit 12, a marker display control unit 13, a video determination unit 14, a video playback unit 15, a region selection unit 16, and a time measurement unit 17.

  FIG. 2 shows a functional configuration for realizing the portable information terminal 1 according to the embodiment, and other configurations are omitted. In FIG. 2, each element described as a functional block for performing various processes is configured by a CPU (Central Processing Unit), a main memory, and other LSI (Large Scale Integration) of the portable information terminal 1 in terms of hardware. be able to. In terms of software, it is realized by a program loaded in the main memory. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms depending on the calculation resource of the portable information terminal 1, and is not limited to any one.

  The communication unit 60 exchanges information with other devices via a network such as the Internet or a mobile phone communication network. The communication unit 60 can be realized using a known communication module such as WiFi (registered trademark), LTE (Long Term Evolution) module, or 3G (3rd Generation) module.

  The storage unit 30 includes an operating system (hereinafter referred to as “OS”) for controlling the operation of the portable information terminal 1 and an application for causing the portable information terminal 1 to realize the functions of the respective units of the control unit 10. A program (hereinafter simply referred to as “application”) is stored. The storage unit 30 also stores electronic information such as omnidirectional video to be displayed on the display unit 50 and metadata associated with the omnidirectional video. The storage unit 30 can be realized by a large-capacity storage such as an SSD (Solid State Drive). Note that the storage unit 30 acquires the omnidirectional video from, for example, an imaging device for capturing the omnidirectional video, or acquires the omnidirectional video from another device such as a server via the communication unit 60 and the network. Store the video.

The control unit 10 is the CPU and GPU (Graphic Processing Unit) of the portable information terminal 1. The control unit 10 controls the operation of the portable information terminal 1 by executing the OS read from the storage unit 30. The control unit 10 also executes an application read from the storage unit 30, thereby executing an execution function corresponding to the function of each unit in the control unit 10, an icon display control function, a marker display control function, a video determination function, and a video playback function. Realize an area selection function and a time measurement function.
The temporary storage unit 40 is a volatile memory that provides a work area for the control unit 10. The temporary storage unit 40 can be realized using, for example, a DRAM (Dynamic Random Access Memory).

The azimuth detecting unit 20 detects the azimuth and inclination of the portable information terminal 1, that is, the azimuth and inclination of the display unit 50 provided in the portable information terminal 1. The azimuth inclination detecting unit 20 can be realized by using, for example, a known azimuth angle sensor and an acceleration sensor, or using a six-axis sensor.
The video playback unit 15 reads out the omnidirectional video stored in the storage unit 30 and plays back the omnidirectional video. The omnidirectional video reproduced by the video reproduction unit 15 is a moving image, and the video reproduction unit 15 reproduces all or at least a part of the omnidirectional video and stores it in the temporary storage unit 40 serving as a frame buffer. Note that the video playback unit 15 may play back the omnidirectional video as a normal two-dimensional video, or by playing back each of the parallax image for the right eye and the parallax image for the left eye of the user P. You may play as.

  FIGS. 3A to 3B are diagrams for explaining the omnidirectional video 80 reproduced by the video reproduction unit 15 according to the embodiment. Specifically, FIG. 3A is a diagram for explaining an omnidirectional video 80 projected on a virtual spherical screen, and FIG. 3B is an omnidirectional video 80 reproduced by the video reproduction unit 15. It is a figure which shows the relationship with the display image 51 which the display part 50 of the portable information terminal 1 displays.

  As shown in FIG. 3A, the video playback unit 15 plays back the omnidirectional video 80 on the assumption that a spherical surface with a predetermined radius centered on the user P is projected onto a virtual spherical screen. In FIG. 3A, an area denoted by reference numeral 51 is a display video 51 determined as a video to be displayed on the display unit 50 of the portable information terminal 1 by the video determination unit 14. FIG. 3A shows an example in which the azimuth of the display unit 50 detected by the azimuth inclination detecting unit 20 is an azimuth slightly north from true east. Further, as shown in FIG. 3B, the azimuth inclination detection unit 20 indicates that the inclination of the display unit 50, that is, the elevation angle is α. The video determination unit 14 determines the display video 51 based on the azimuth and tilt detected by the azimuth tilt detection unit 20. The video determination unit 14 cuts out the display video 51 from the omnidirectional video 80 and stores it in the frame buffer. The display unit 50 displays the display video 51 stored in the frame buffer.

  When the user P moves the portable information terminal 1 to change the azimuth and inclination of the display unit 50, the display video 51 cut out from the omnidirectional video 80 by the video determination unit 14 also changes accordingly. Thereby, the user P who observes the display image 51 can obtain an experience as if he is in a scene where the omnidirectional image 80 is projected.

  The omnidirectional video 80 reproduced by the video reproduction unit 15 according to the embodiment includes a metadata region in which metadata is set at least in part. The area selection unit 16 selects a metadata area when at least a part of the metadata area included in the display video 51 is displayed at a predetermined display position defined on the display unit 50. The execution unit 11 executes an operation corresponding to the metadata set in the metadata area selected by the area selection unit 16.

  The metadata set in the metadata area includes a command that instructs the execution unit 11 to perform some operation. For example, it is assumed that the metadata set in the metadata area selected by the area selection unit 16 includes a command instructing to end the reproduction of the omnidirectional video 80 by the video reproduction unit 15. In this case, the execution unit 11 ends the reproduction of the omnidirectional video 80 by the video reproduction unit 15. Similarly, when the selected metadata includes an instruction to switch from the omnidirectional video 80 reproduced by the video reproduction unit 15 to another omnidirectional video 80, the execution unit 11 performs the video reproduction unit 15. Then, another omnidirectional video 80 is reproduced. Thereby, the user P of the portable information terminal 1 can operate the portable information terminal 1 without touching the display unit 50 by changing the orientation and inclination of the display unit 50 and selecting the metadata area.

  FIGS. 4A and 4B are diagrams for explaining selection of a metadata area by the area selection unit 16 according to the embodiment. Specifically, FIG. 4A illustrates the metadata area 53 in the display video 51, and FIG. 4B illustrates the selected metadata area 53. As illustrated in FIG. FIG. 4A and FIG. 4B show an example in which three metadata areas 53 of the first metadata area 53a, the second metadata area 53b, and the third metadata area 53c are displayed. Show. 4A to 4B, parallelograms or rectangles denoted by reference numerals 53a, 53b, and 53c are respectively a first metadata area 53a, a second metadata area 53b, and a third metadata. This is a data area 53c. Note that the number of metadata areas 53 included in the display video 51 is not limited to three, and may be one or two, or four or more. In this specification, when a plurality of metadata areas 53 are not distinguished, they are collectively referred to as “metadata areas 53”.

  In the three metadata areas 53 shown in FIGS. 4A and 4B, metadata including a command indicating that a new video content is to be reproduced is set. Although not shown in FIGS. 4A and 4B, the video playback unit 15 displays thumbnail videos of video content to be played back in association with the metadata area 53 in each metadata area 53. Also good.

  The marker display control unit 13 displays a marker 52 for selecting the metadata area 53 at a predetermined display position determined on the display unit 50. In the example shown in FIGS. 4A and 4B, the marker display control unit 13 displays a marker 52 that is sufficiently small with respect to the size of the display video 51 so as to be positioned at the center of the display unit 50. The position, size, and shape of the marker 52 displayed on the display unit 50 by the marker display control unit 13 are tested according to the size of the display unit 50 of the portable information terminal 1, the type of video reproduced by the portable information terminal 1, and the like. It may be determined by As an example, as shown in FIGS. 4A and 4B, the marker display control unit 13 displays a rectangle of 5 pixels vertically and 5 pixels horizontally as a marker 52 at the center of the display unit 50.

  FIG. 4A shows an example in which the marker 52 does not overlap any metadata area 53. The user P who observes the display image 51 changes the display image 51 displayed on the display unit 50 by changing the azimuth and inclination of the display unit 50. For example, when the user P changes the tilt of the display unit 50 so that the marker 52 overlaps the second metadata area 53b with respect to the display video 51 shown in FIG. 4A, the marker is shown in FIG. 4B. 52 overlaps the second metadata area 53b. As shown in FIG. 1B, when the user P uses the portable information terminal 1 as an HMD together with the housing 70, the user P moves the face upward to move the marker 52 to the second position. It can be overlapped with the metadata area 53b.

  The icon display control unit 12 causes the display unit 50 to display an icon 54 indicating that the marker 52 and the metadata area 53 included in the display video 51 overlap. In the example shown in FIG. 4B, the icon 54 includes a cross centered on the marker 52 and a circle surrounding the cross. Thus, since the icon 54 indicates that the marker 52 overlaps the metadata area 53, the user P can confirm that the metadata area 53 has been properly selected. The icon display control unit 12 displays the icon 54 on the display unit 50 when the marker 52 overlaps at least a part of the metadata area 53.

  As shown in FIG. 4B, the icon 54 that the icon display control unit 12 displays on the display unit 50 has elapsed since at least a part of the metadata area 53 included in the display video 51 is displayed on the marker 52. Contains information indicating time. More specifically, in the example shown in FIG. 4B, a circle that is one of the components of the icon 54 indicates the time that has elapsed since the marker 52 overlapped the second metadata area 53b. In other words, this circle functions as a graphical user interface (GUI) called a wait bar or progress bar.

  When the marker 52 overlaps the second metadata area 53b, the icon display control unit 12 causes the display unit 50 to display the icon 54. At this time, the circle of the icon 54 is white. Thereafter, the white portion changes to black with the passage of time when the marker 52 overlaps the second metadata area 53b. When all the circles become black, the area selection unit 16 determines the selection of the second metadata area 53b.

  As described above, the area selection unit 16 allows at least a part of the metadata area 53 included in the display video 51 and the marker 52 while the predetermined selection determination time elapses to determine whether or not the area is selected. If the images are displayed continuously overlapping, the metadata area 53 is selected. By providing a time difference from when the marker 52 overlaps the metadata area 53 until the selection of the metadata area 53 by the area selection unit 16 is confirmed, the metadata area is unintentionally moved when the user P moves the display unit 50. The selection of 53 can be reduced. Further, the elapse of time when the marker 52 overlaps the second metadata area 53b is clearly indicated to the user P using the icon 54, thereby improving usability when the user P intentionally selects the metadata area 53. can do. Note that the specific length of the “selection determination time” may be determined by experiments according to the content of metadata associated with the metadata area 53, the content displayed on the display unit 50, and the like. Seconds.

<Metadata structure>
FIGS. 5A to 5B are diagrams for explaining metadata of an omnidirectional video handled by the portable information terminal 1 according to the embodiment. Specifically, FIG. 5A is a diagram illustrating a boundary line of the metadata area 53 defined in the display video 51, and FIG. 5B is a diagram of the metadata area 53 defined in FIG. 4 is a schematic diagram showing a data structure of metadata set in one metadata area 53. FIG.

  A display image 51 shown in FIG. 5A is an image of a fruit store. In the display video 51, a fourth metadata area 53d indicating a fruit shop owner, a fifth metadata area 53e indicating a customer, and a sixth metadata area 53f indicating a banana as a product are defined. FIG. 5B is a schematic diagram showing the data structure of metadata set in the sixth metadata area 53f. Each metadata includes a metadata ID (IDentifier) that uniquely identifies each metadata, an outline of the metadata, absolute coordinates that indicate the position of the metadata area 53 in the omnidirectional video 80, and the display position of the metadata area 53 in the display video 51. Relative coordinates indicating information, information for specifying one or more related contents associated with the metadata area 53, whether or not the metadata area 53 is active, that is, presenting the metadata area 53 to the user P It contains information such as whether or not it can be selected.

  The absolute coordinates are coordinates determined in advance for specifying the display position in the omnidirectional video 80. The absolute coordinates may be any as long as the position of the omnidirectional video 80 can be uniquely specified. For example, the azimuth angle and elevation angle (or depression angle) when the line segment connecting the east and west in FIG. 3A is the X axis, the line segment connecting the north and south is the Y axis, and the line segment from the user P to the zenith is the Z axis. Can be used to represent the global coordinates. The azimuth angle is the polar coordinate angle on the XY plane, the elevation angle is the angle above the XY plane, and the depression angle is the angle below the XY plane.

  In FIG. 5A, the seventh metadata area 53g, the eighth metadata area 53h, and the ninth metadata area 53i become active when the user P selects the sixth metadata area 53f, and the video reproduction unit Reference numeral 15 denotes a metadata area 53 to be displayed on the display unit 50. More specifically, this is a metadata area for presenting related content associated with the sixth metadata area 53f to the user P. In the example shown in FIGS. 5A and 5B, when the user P selects the sixth metadata area 53f indicating a banana, a URL (Uniform Resource Locator) that introduces the production area, an image of a similar product, the product Related content such as a recipe for cooking using is activated and displayed on the display unit 50.

  When the user P puts the marker 52 on the related content that has been activated, the region selection unit 16 selects the related content. The execution unit 11 performs an operation of causing the display unit 50 to display content related to the subject in the metadata area 53 selected by the area selection unit 16. As described above, the execution unit 11 sets the content related to the subject in the metadata region 53 selected by the region selection unit 16 by setting data indicating the content related to the subject in the omnidirectional video 80 in advance as metadata. An operation to be displayed on the display unit 50 can be executed. As a result, the user P can obtain information related to the subject of the omnidirectional video 80.

<Shopping using omnidirectional video>
By using the technique described with reference to FIGS. 5A and 5B, for example, a shopping place using the omnidirectional video 80 can be provided to the user P.
FIGS. 6A to 6B are diagrams for explaining a virtual store provided by the portable information terminal 1 according to the embodiment. Specifically, the display video 51 in FIGS. 6A to 6B shows an example in the case of a jacket section in a clothing store.

  Each jacket photographed in the display video 51 shown in FIG. 6A is set in the metadata area 53 by the producer of the omnidirectional video 80 in advance. Therefore, as shown in FIG. 6A, when the user P overlaps the marker 52 with a desired jacket in the display video 51, the region selection unit 16 selects a metadata region 53 corresponding to the jacket. In the example shown in FIGS. 6A and 6B, in each metadata area 53 that is a jacket, information such as similar products to the jacket and other clothing items that match the jacket is set as metadata. ing. For example, FIG. 6B shows a state in which a tenth metadata area 53j and an eleventh metadata area 53k indicating jackets similar to the jacket selected by the user P are displayed. In FIG. 6B, a twelfth metadata area 53l indicating shoes recommended to the user P as shoes matching the jacket selected by the user P is also active.

In addition, although not shown in FIGS. 6A and 6B, when the user P selects the metadata area 53 that is a jacket, the metadata area 53 for moving to a settlement page for purchasing the jacket. Is displayed. Thereby, it is possible to provide the user P with an experience as if he / she purchased a jacket at a real shop.
Note that the following description is based on the assumption that the portable information terminal 1 according to the embodiment acquires and reproduces the omnidirectional video 80 from a server (not shown).

<Heat map>
In this way, the user P selects the desired metadata area 53 by overlapping the marker 52 with the metadata area 53 in the omnidirectional video 80. That is, the marker 52 can be an index for measuring which point of the omnidirectional video 80 the user P is paying attention to. For this reason, the time measuring unit 17 measures the time when the marker 52 is displayed in the partial area for each of the plurality of partial areas defined in the omnidirectional video 80.

  The communication unit 60 transmits the time measured by the time measurement unit 17 and the overall coordinates indicating the measurement position in the omnidirectional video 80 to the server that provides the omnidirectional video 80. The server aggregates time information and position information from a plurality of portable information terminals 1 communicating via a network, and creates a heat map. Here, the “heat map” is an omnidirectional video in which information indicating the length of time measured by the time measuring unit 17 of one or more portable information terminals 1 is superimposed for each partial region to be measured. The heat map is information that visually indicates the length of time when the user P of the portable information terminal 1 pays attention to a partial area of the omnidirectional video 80, that is, the act of overlapping the marker 52 with the partial area. The video reproduction unit 15 acquires a heat map from the server and causes the display unit 50 to display the heat map.

  FIG. 7 is a schematic diagram showing a heat map according to the embodiment. In FIG. 7, a broken-line rectangle indicated by reference numeral 55 indicates one partial region 55. In order to complicate, not all symbols are attached, but in FIG. 7, rectangular regions surrounded by broken lines are different partial regions 55. In the example shown in FIG. 7, the region where the pixel luminance value is small, that is, the portion observed in black has a longer time for the user P to overlap the marker 52 than the portion where the pixel luminance value is large and observed white. Is shown.

  The heat map can be a useful index for estimating the attention area of the user P as a customer in shopping using the omnidirectional video 80 as shown in FIGS. 6A to 6B, for example. For example, a shopping provider using the omnidirectional video 80 can prepare a plurality of omnidirectional videos 80 with different designs and measure the attention area of the user P in each design. Thereby, the AB test using the omnidirectional video 80 can be realized. A shopping provider using the omnidirectional video 80 can easily find out what kind of omnidirectional video 80 should be prepared so that the attention time of the user P for the product increases. It is possible to provide a shopping place.

<Processing flow>
FIG. 8 is a flowchart for explaining the flow of area selection processing executed by portable information terminal 1 according to the embodiment. The process in this flowchart is started when the portable information terminal 1 is activated, for example.

  The video playback unit 15 plays back the omnidirectional video 80 read from the storage unit 30 (S2). The azimuth detecting unit 20 detects the azimuth indicating the direction in which the display unit 50 of the portable information terminal 1 is facing and the inclination of the display unit 50 (S4). The video determination unit 14 determines the display video 51 to be displayed on the display unit 50 among the omnidirectional video 80 based on the azimuth and tilt detected by the azimuth tilt detection unit 20 (S6).

  The area selection unit 16 stands by until the user P selects the metadata area 53 by moving the display section 50 and overlaying the marker 52 on the metadata area 53 (S8). While there is no candidate for selecting the metadata area 53 by the user P (NO in S10), the process returns to step S2 and repeats the process from step S2 to step S8. If there is a selection candidate for the metadata area 53 by the user P (YES in S10), the icon display control unit 12 controls the display unit 50 so that the icon 54 is displayed over the marker 52 (S12).

  The icon display control unit 12 measures an elapsed time after the marker 52 is overlapped with the metadata area 53 (S14). Until the time measured by the icon display control unit 12 passes a predetermined selection determination time (NO in S16), the icon display control unit 12 continues the time measurement. When the time measured by the icon display control unit 12 has passed the predetermined selection determination time (YES in S16), the region selection unit 16 confirms the selection of the metadata region 53 on which the marker 52 overlaps (S18). The execution unit 11 executes an operation corresponding to the metadata set in the metadata area 53 whose selection is confirmed (S20). While the region selection process is not finished (NO in S22), the process returns to step S2 and executes the process from step S2 to step S20. When the area selection process is ended (YES in S22), the process in this flowchart is ended.

As described above, according to the portable information terminal 1 according to the embodiment, the user P can select the metadata area 53 displayed on the display unit 50 without touching the display unit 50 of the portable information terminal 1. it can.
In particular, since the icon 54 is displayed on the display unit 50 when the user P puts the marker 52 on the metadata area 53, it can be understood at a glance that the user P has selected the metadata area 53. In addition, since the operation executed by the execution unit 11 is set as metadata in each metadata area 53, the user P can cause the portable information terminal 1 to execute various operations without touching the display unit 50. it can. This is advantageous, for example, in that the accuracy of the instruction is higher than the instruction by voice recognition. Further, it is advantageous in that the variation of the operation that can be instructed can be increased as compared with the case where the inclination of the portable information terminal 1 is simply used as an instruction.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention. Such modifications will be described below.

  The product of the omnidirectional video 80 is manually set by the creator of the omnidirectional video 80 and associated metadata for products such as a banana at a fruit shop storefront or a clothing store jacket, which is the subject of the omnidirectional video 80 taken. On the other hand, in recent years, a subject recognition technique for recognizing a subject in a video using machine learning has been rapidly developed. Among these, in particular, a person recognition technique for detecting a person in a video and specifying who the person is is in a practical stage.

  Therefore, the metadata area 53 may be automatically created by recognizing a person photographed in the omnidirectional video 80 using a known person recognition technique. This can be realized, for example, by the execution unit 11 recognizing a person in the display video 51 and executing contour extraction to generate the metadata area 53. The execution unit 11 can identify metadata indicating person information in the metadata area 53 by making an inquiry to a SNS (Social Networking System) server (not shown) based on the recognized person name. Accordingly, the execution unit 11 can execute an operation of causing the display unit 50 to display content related to the subject, such as the face image and name of the recognized subject.

  By using the subject recognition function by the execution unit 11, for example, assuming an SNS service that posts a short movie of an omnidirectional video 80 taken by the photographer himself, a user using the SNS service is posted. It is assumed that a subject such as a photographer photographed in the omnidirectional video 80 is selected. In response to the selection of the subject, the user can “follow” the subject. In other words, it is an omnidirectional video version of a short posting SNS. Thereby, the user who uses this SNS service only changes the direction and inclination of the display unit 50 and selects a subject shown in the omnidirectional video 80, and the other omnidirectional video 80 posted by the subject in the past can be obtained. In addition, the omnidirectional video 80 to be posted can be easily followed.

DESCRIPTION OF SYMBOLS 1 ... Portable information terminal 10 ... Control part 11 ... Execution part 12 ... Icon display control part 13 ... Marker display control part 14 ... Video determination part 15 ... Video reproduction part 16 ... area selection part 17 ... time measurement part 20 ... azimuth inclination detection part 30 ... storage part 40 ... temporary storage part 50 ... display part 51 ... display video 53 ... Metadata area 54 ... Icon 55 ... Partial area 60 ... Communication unit 70 ... Housing 80 ... Omni-directional video

Claims (9)

On your mobile device,
An azimuth tilt detection function for detecting the orientation and tilt of the portable information terminal;
A video playback function for playing back an omnidirectional video including at least a part of a metadata area in which metadata is set;
A video determination function for determining a display video to be displayed on the display unit of the portable information terminal out of the omnidirectional video based on the orientation and the inclination;
An area selection function for selecting the metadata area when at least a part of the metadata area included in the display video is displayed at a predetermined display position defined in the display unit;
An execution function for executing an operation according to the metadata set in the selected metadata area;
A program characterized by realizing. In the portable information terminal,
A marker display control function for displaying a marker for selecting a metadata area at a predetermined display position determined in the display unit;
An icon display control function for causing the display unit to display an icon indicating that the marker and a metadata area included in the display video overlap;
The program according to claim 1, further comprising:   The icon display control function displays an icon including information indicating a time elapsed since at least a part of a metadata area included in the display video is displayed at a predetermined display position defined in the display unit. The program according to claim 2, wherein the program is displayed on a screen.   The area selection function has a predetermined selection determination time for determining whether or not an area is selected at a predetermined display position determined in the display unit at least a part of the metadata area included in the display video. 4. The program according to claim 1, wherein a metadata area included in the display video is selected when the image is continuously displayed for a while. The metadata set in the metadata area is data indicating content related to the subject of the omnidirectional video in the metadata area,
The execution function executes an operation of causing the display unit to display content related to a subject in the metadata area selected by the area selection function.
The program according to any one of claims 1 to 4, wherein: The execution function executes an operation of recognizing a subject in the metadata region selected by the region selection function and displaying content related to the recognized subject on the display unit.
The program according to any one of claims 1 to 5, wherein:   6. The area selection function selects content when at least a part of the content displayed on the display unit is displayed at a predetermined display position defined on the display unit. Or the program of 6. The portable information terminal further implements a time measurement function for measuring the time at which the partial area is displayed at a predetermined display position defined on the display unit for each of the plurality of partial areas included in the omnidirectional video. Let
The video playback function plays back the omnidirectional video that is superimposed for each partial area that is the measurement target information indicating the length of time measured by the time measurement function.
The program according to any one of claims 1 to 7, characterized by: A display unit;
An azimuth inclination detection unit for detecting the orientation and inclination of the display unit;
A video playback unit that plays back an omnidirectional video including at least part of a metadata area in which metadata is set;
A video determination unit for determining a display video to be displayed on the display unit among the omnidirectional video based on the orientation and the inclination;
An area selection unit that selects the metadata area when at least a part of the metadata area included in the display video is displayed at a predetermined display position defined in the display unit;
An execution unit that executes an operation defined by the metadata set in the metadata region selected by the region selection unit;
A portable information terminal comprising: