WO2017029918A1

WO2017029918A1 - System, method and program for displaying moving image with specific field of view

Info

Publication number: WO2017029918A1
Application number: PCT/JP2016/071040
Authority: WO
Inventors: 豪放小倉
Original assignee: 株式会社ディー・エヌ・エー
Priority date: 2015-08-20
Filing date: 2016-07-15
Publication date: 2017-02-23
Also published as: JP2017041780A

Abstract

The purpose of the present invention is to appropriately display information such as a comment inputted in a moving image with a field of view that may differ between users. A system according to one embodiment displays, on a terminal device of each of a plurality of users, a moving image which is constructed as a moving image having a wide-angle field of view and the total field of view of which is associated with a virtual space with the respective field of views of the plurality of users, and when accepting a comment from a user, specifies a position on the virtual space included in the field of view of the user, places the comment at the position, and according to the placement of the comment, displays the comment on the terminal device of a user including the position at which the comment is placed in the field of view thereof.

Description

System, method, and program for displaying moving image with specific visual field

This cross-referenced application claims priority based on Japanese Patent Application No. 2015-162720 (filed Aug. 20, 2015), the contents of which are hereby incorporated by reference in their entirety.
The present invention relates to a system, method, and program for displaying a moving image with a specific field of view.

Conventionally, in order to activate communication between users who browse the same video, a system that displays a comment given by a user together with a video has been proposed (for example, see Patent Document 1). In such a system, for example, a comment is displayed together with a moving image during a moving image reproduction time when the comment is given.

In recent years, a moving image called “360-degree moving image” shot using a camera that can shoot almost all directions at once has been distributed. In such videos, the display direction (field of view) changes continuously, for example, by changing the orientation of the user terminal (changing direction / tilting) or flicking / dragging the screen. The user can view the moving image while changing the field of view. And in such a moving image, it is desirable that information such as a comment given by the user can be shared in order to activate communication between users who view the same moving image. Therefore, it is also conceivable to display the comment during the moving image reproduction time to which the comment is given as in the conventional system described above.

Japanese Patent Laying-Open No. 2015-57896

However, in a video that can have different fields of view among users, such as a 360-degree video, even if the user is viewing the same video, the target being viewed may vary depending on the field of view. Therefore, for example, it is difficult for another user who is viewing a video with a field of view different from the field of view of the user who provided the comment to sympathize with the content of the comment because the target viewed is different from the user who provided the comment. . If the content of the comment cannot be sympathized, the activation of communication is also limited. Therefore, it is desirable to appropriately display information such as a comment in a moving image that may have different fields of view among users.

An embodiment of the present invention has an object to appropriately display information such as comments input in moving images that may have different fields of view among users. Other objects of the embodiments of the present invention will become apparent by referring to the entire specification.

A system according to an embodiment of the present invention is a system that displays a moving image in a specific field of view, and includes one or more computer processors, and the one or more computer processors execute readable instructions. Accordingly, a specific video that is configured as a video having a wide-angle field of view and a virtual space is associated with the entire field of view in each terminal device of a plurality of users including the first and second users, A step of displaying each field of view of the plurality of users, and a first position in the virtual space included in the field of view of the first user when first input information is received from the first user. And specifying the first input information at the first position, and according to the arrangement of the first input information, the second user's end including the first position in the field of view The device executes a step of displaying the first input information.

In the system according to the embodiment described above, the specific moving image may be configured as a moving image having a visual field of 360 degrees in at least the horizontal direction, and the virtual space may be configured as an inner surface of a virtual sphere. Such a moving image may be referred to as a “360-degree moving image”, and the vertical visual field is, for example, in the range of 180-360 degrees.

A method according to an embodiment of the present invention is a method for displaying a moving image with a specific field of view, which is executed by one or a plurality of computers, and each terminal device of a plurality of users including a first user and a second user. And displaying a specific moving image having a wide-angle visual field and a virtual space associated with the entire visual field in each visual field of the plurality of users; Identifying a first position in the virtual space included in the field of view of the first user when receiving the first input information, and arranging the first input information at the first position; Displaying the first input information on the terminal device of the second user including the first position in the field of view according to the arrangement of the first input information.

A program according to an embodiment of the present invention is a program for displaying a moving image in a specific field of view, and is executed on the one or more computers in accordance with the first and the first computers. Each terminal device of a plurality of users including the second user is provided with a specific moving image that is configured as a moving image having a wide-angle field of view and has a virtual space associated with the entire field of view of each of the plurality of users. A step of displaying in a field of view, and when receiving first input information from the first user, a first position in the virtual space included in the field of view of the first user is specified, and the first Arranging the first input information at the position of the first user, and in response to the arrangement of the first input information, the terminal device of the second user including the first position in the field of view, Display input information A step that causes the execution.

Various embodiments of the present invention can appropriately display information such as comments input in moving images that can have different fields of view among users.

1 is a configuration diagram schematically showing a configuration of a network including a system 1 according to an embodiment of the present invention. The block diagram which shows roughly the function which the system 1 (server 10 and the terminal device 30) in one Embodiment has. The figure which shows an example of the information managed in the user management table 51a in one Embodiment. The figure which shows an example of the information managed in the comment management table 51b in one Embodiment. The figure for demonstrating the virtual space and user's visual field in one Embodiment. The figure for demonstrating the user's visual field in one Embodiment. The figure which shows an example of the 1st moving image reproduction screen 60 in one Embodiment. The figure which shows an example of the 2nd moving image reproduction screen 70 in one Embodiment. The flowchart which shows an example of the comment arrangement | positioning process in one Embodiment. The figure for demonstrating the user's gaze point FP and gaze area | region FR in one Embodiment. The figure which shows an example of the 1st moving image reproduction screen 60 and the comment object 114 in one Embodiment. The figure which shows an example of the 1st moving image reproduction screen 60 in one Embodiment. The figure which shows an example of the 2nd moving image reproduction screen 70 in one Embodiment. The figure for demonstrating the sound output according to arrangement | positioning of the comment in one Embodiment. The figure which shows an example of the 1st moving image reproduction screen 60 in one Embodiment. The figure which shows an example of the display of the comment object 114 in one Embodiment. The figure which shows an example of the 1st moving image reproduction screen 160 in other embodiment.

Hereinafter, various embodiments of the present invention will be described with reference to the drawings as appropriate. In addition, the same referential mark is attached | subjected to the component which is common in drawing.

FIG. 1 is a configuration diagram schematically showing the configuration of a network including a system 1 according to an embodiment of the present invention. The system 1 in one embodiment includes a server 10 and a plurality of terminal devices 30 that are communicably connected to the server 10 via a communication network 40 such as the Internet, as illustrated. The server 10 provides a moving image distribution service that distributes various moving images to the terminal device 30. The moving image distributed in the moving image distribution service in the embodiment includes a real-time moving image (live moving image) provided by the moving image providing apparatus 20.

The server 10 in one embodiment is configured as a general computer, and as illustrated, a CPU (computer processor) 11, a main memory 12, a user I / F 13, a communication I / F 14, and storage (storage). Device) 15, and these components are electrically connected to each other via a bus. The CPU 11 loads an operating system and various other programs from the storage 15 into the main memory 12 and executes instructions included in the loaded programs. The main memory 12 is used for storing a program executed by the CPU 11, and is configured by a DRAM or the like, for example. In addition, the server 10 in one embodiment may be configured using a plurality of computers each having a hardware configuration as described above.

The user I / F 13 includes, for example, an information input device such as a keyboard and a mouse that accepts an operator's input, and an information output device such as a liquid crystal display that outputs a calculation result of the CPU 11. The communication I / F 14 is implemented as hardware, firmware, communication software such as a TCP / IP driver or a PPP driver, or a combination thereof, and can communicate with the video providing device 20 and the terminal device 30 via the communication network 40. Composed.

The storage 15 is composed of, for example, a magnetic disk drive, and stores various programs such as a control program for providing a moving image distribution service. The storage 15 can also store various data for providing the moving image distribution service. Various data that can be stored in the storage 15 may be stored in a database server or the like that is physically separate from the server 10 that is communicably connected to the server 10.

In one embodiment, the server 10 also functions as a web server that manages a website composed of a plurality of web pages in a hierarchical structure, and provides a video distribution service to the user of the terminal device 30 through such a website. obtain. The storage 15 can also store HTML data corresponding to this web page. HTML data is associated with various image data, and various programs described in a script language such as JavaScript (registered trademark) can be embedded.

In one embodiment, the server 10 can provide a video distribution service via an application executed in an execution environment other than the web browser in the terminal device 30. Such applications can also be stored in the storage 15. This application is created using a programming language such as Objective-C or Java (registered trademark). The application stored in the storage 15 is distributed to the terminal device 30 in response to the distribution request. Note that the terminal device 30 can also download such an application from a server other than the server 10 (such as a server providing an application market).

In this way, the server 10 can manage the website for providing the video distribution service and distribute the web page (HTML data) constituting the website in response to a request from the terminal device 30. . In addition, as described above, the server 10 may be an application executed in the terminal device 30 in place of or in addition to the provision of the moving image distribution service using such a web page (web browser). The video distribution service can be provided based on the communication. Regardless of which aspect of the service is provided, the server 10 can transmit and receive various data (including data necessary for screen display) necessary for providing the video distribution service to and from the terminal device 30. . In addition, the server 10 can store various data for each identification information (for example, user ID) for identifying each user, and can manage the provision status of the video distribution service for each user. Although detailed description is omitted, the server 10 may have a function of performing user authentication processing, billing processing, and the like.

The moving image providing apparatus 20 according to the embodiment is configured as a general computer, and as illustrated in FIG. 1, a CPU (computer processor) 21, a main memory 22, a user I / F 23, a communication I / F 24, and the like. , A storage (storage device) 25 and an ultra-wide-angle camera 26, and these components are electrically connected to each other via a bus.

The CPU 21 loads an operating system and various other programs from the storage 25 to the main memory 22 and executes instructions included in the loaded programs. The main memory 22 is used for storing a program executed by the CPU 21 and is configured by, for example, a DRAM or the like.

The user I / F 23 includes, for example, an information input device that receives an operator input and an information output device that outputs a calculation result of the CPU 21. The communication I / F 24 is implemented as hardware, firmware, communication software such as a TCP / IP driver or a PPP driver, or a combination thereof, and is configured to be able to communicate with the server 10 and the terminal device 30 via the communication network 40. The

The ultra-wide-angle camera 26 has a built-in microphone and is configured to capture an ultra-wide-angle image via an ultra-wide-angle lens or a plurality of lenses. In one embodiment, the super wide-angle camera 26 is configured as a 360-degree camera having a 360-degree field of view in the horizontal direction and a field-of-view range of 180-360 degrees in the vertical direction. 20 is configured to transmit a 360-degree moving image having a substantially omnidirectional field of view photographed through the super wide-angle camera 26 to the server 10 in real time.

The terminal device 30 according to the embodiment is an arbitrary information processing device that displays a web page of a website provided by the server 10 on a web browser and implements an execution environment for executing an application. , Wearable devices (eg, head-mounted displays), personal computers, gaming terminals, and the like.

The terminal device 30 is configured as a general computer, and as shown in FIG. 1, a CPU (computer processor) 31, a main memory 32, a user I / F 33, a communication I / F 34, a storage (storage device) ) 35 and various sensors 36, each of which is electrically connected to each other via a bus.

The CPU 31 loads an operating system and various other programs from the storage 35 to the main memory 32 and executes instructions included in the loaded programs. The main memory 32 is used for storing a program executed by the CPU 31, and is configured by, for example, a DRAM or the like.

The user I / F 33 includes, for example, an information input device such as a touch panel that accepts user input, a keyboard, a button, and a mouse, and an information output device such as a liquid crystal display that outputs a calculation result of the CPU 31. The communication I / F 34 is implemented as hardware, firmware, communication software such as a TCP / IP driver or a PPP driver, or a combination thereof, and can communicate with the server 10 and the moving image providing apparatus 20 via the communication network 40. Composed.

The storage 35 is composed of, for example, a magnetic disk drive, a flash memory, or the like, and stores various programs such as an operating system. The storage 35 can store various applications received from the server 10 or the like.

The various sensors 36 include, for example, an acceleration sensor, a gyro sensor (angular velocity sensor), a geomagnetic sensor, and the like. Based on information detected by these sensors, the terminal device 30 can specify the posture, inclination, direction, and the like of the terminal device 30 itself.

The terminal device 30 includes, for example, a web browser for interpreting an HTML file (HTML data) and displaying the screen, and interprets and receives the HTML data acquired from the server 10 by the function of the web browser. A web page corresponding to the HTML data thus displayed can be displayed. The web browser of the terminal device 30 can incorporate plug-in software that can execute various types of files associated with HTML data.

When the user of the terminal device 30 uses the video distribution service provided by the server 10, for example, HTML data, an animation instructed by an application, an operation icon, and the like are displayed on the terminal device 30 on the screen. The user can input various instructions using the touch panel of the terminal device 30 or the like. The instruction input from the user is transmitted to the server 10 via the function of the application execution environment such as the web browser of the terminal device 30 and NgCore (trademark).

Next, functions of the system 1 (the server 10 and the terminal device 30) in the embodiment configured as described above will be described. In the moving image distribution service according to the embodiment, various types of moving images can be distributed to the terminal device 30, but here, a real-time 360 degree moving image provided mainly from the moving image providing device 20 or the like is mainly transmitted to the terminal device 30. The function to deliver will be described.

FIG. 2 is a block diagram schematically illustrating functions of the server 10 and the terminal device 30 according to an embodiment. As illustrated, the server 10 according to the embodiment includes an information storage unit 51 that stores information, a moving image distribution control unit 52 that controls distribution of moving images, and a virtual that manages a virtual space associated with the entire field of view of moving images. A space management unit 53. These functions are realized by the cooperative operation of hardware such as the CPU 11 and the main memory 12 and various programs and tables stored in the storage 15, for example, instructions included in the loaded program This is realized by the CPU 11 executing. 2 may be realized by the cooperation of the server 10 and the terminal device 30 or may be realized by the terminal device 30.

The information storage unit 51 according to the embodiment is realized by the storage 15 or the like, and as illustrated in FIG. 2, a user management table 51a for managing information related to a user of the video distribution service, and comments (input information) input by the user A comment management table 51b for managing information related to

FIG. 3 shows an example of information managed in the user management table 51a in the embodiment. As shown in the figure, the user management table 51a is associated with a “user ID” that identifies an individual user, and includes information such as “nickname” of this user and “avatar information” that is information related to the user's avatar. to manage. These pieces of information can be provided from the user at a timing such as new user registration of the moving image distribution service, and can be updated as appropriate thereafter.

FIG. 4 shows an example of information managed in the comment management table 51b in one embodiment. As shown in the figure, the comment management table 51b is associated with a combination of “moving image ID” that identifies an individual moving image and “comment ID” that identifies an individual comment, and identifies the user who has input this comment. "Input user ID", "comment content" indicating the content of this comment, "arrangement position" indicating the position in the virtual space where this comment is placed, "arrangement time" indicating the time when this comment is placed, this comment Information such as “erase time” indicating the time of deleting (releasing the arrangement) and “Like number” indicating the number of “Like” input for this comment is managed. As described above, the comment management table 51b according to the embodiment manages information about comments for each moving image to be distributed.

Here, the 360-degree moving image in one embodiment is associated with a virtual space in which the entire field of view is configured as an inner surface of a virtual sphere, and the comment management table 51b has an arrangement position on the virtual space. A value specifying the position (coordinate) of is set.

In addition, in the video distribution service according to the embodiment, the user may input support (favorable emotion) for a comment input by another user who is viewing (playing back) the same video as “Like”. The number of Like in the comment management table 51b is set to the number of “Like” input for the comment. These pieces of information managed in the comment management table 51b are appropriately updated according to the input of a comment by the user, the input of “Like” for the comment, and the like.

The moving image distribution control unit 52 in one embodiment executes various controls related to moving image distribution. For example, the moving image distribution control unit 52 converts a real-time 360 degree moving image received from the moving image providing device 20 or the like into a streaming format and distributes it to the terminal device 30 or a streaming format real time received from the moving image providing device 20 or the like. The 360 degree moving image is distributed to the terminal device 30. In one embodiment, the moving image distributed by the moving image distribution control unit 52 may include a three-dimensional moving image configured to be viewed stereoscopically by the user.

The virtual space management unit 53 in one embodiment executes various processes related to management of a virtual space associated with the entire field of view of the moving image. For example, when the user inputs a comment, the virtual space management unit 53 specifies a position in the virtual space included in the user's field of view, and places the comment at the specified position. Here, the virtual space and the user's field of view associated with the entire field of view of the 360-degree moving image in one embodiment will be described with reference to FIG.

As shown in FIG. 5, the 360-degree moving image is configured as a moving image having the entire field of view on the inner surface (all or a part) of the virtual sphere S, and the line of sight of the user located at the center C of the sphere S When the direction A is specified, the user's visual field V is specified based on a predetermined viewing angle θ. That is, in the 360 moving image, when the direction of the line of sight of the user located at the center C of the virtual sphere S is specified, the moving image of the portion included in the field of view based on the direction of the line of sight is displayed. In FIG. 5, the visual field V is expressed as a curve, but the visual field V is a partial region of the inner surface of the sphere S as illustrated in FIG. 6. The virtual space in one embodiment is configured as the inner surface of the sphere S, that is, associated with the entire field of view of the 360 moving images.

The virtual space management unit 53 in one embodiment transmits various pieces of information regarding the virtual space to the terminal device 30. For example, the virtual space management unit 53 can transmit information related to the comment object based on the comments arranged in the virtual space to the terminal device 30.

Also, the virtual space management unit 53 receives an input of “Like” for the comment. For example, when the state in which the comment object is displayed at the user's point of interest continues for a predetermined effective time (for example, 10 seconds), the virtual space management unit 53 displays “ The input of “Like” is received via the terminal device 30.

As shown in FIG. 2, the terminal device 30 according to an embodiment includes a reproduction control unit 55 that controls reproduction of a moving image, and an input management unit 56 that manages input by a user. These functions are realized by the cooperation of hardware such as the CPU 31 and the main memory 32, and various programs and tables stored in the storage 35. For example, instructions included in the loaded program This is realized by the CPU 31 executing. In addition, part or all of the functions of the terminal device 30 illustrated in FIG. 2 can be realized by the cooperation of the server 10 and the terminal device 30, or can be realized by the server 10.

The playback control unit 55 in one embodiment executes various controls related to playback of moving images. For example, the reproduction control unit 55 displays the 360 degree moving image received from the server 10 on the terminal device 30 with a field of view specified by the user. For example, the playback control unit 55 identifies the direction of the user's line of sight according to the user's operation to change the attitude, tilt, orientation, and the like of the terminal device 30 or the flick / drag operation of the screen. The moving image of the part included in the visual field determined based on the direction of the user's line of sight is displayed among the moving images of the entire visual field of.

Further, the reproduction control unit 55 displays information related to the virtual space on the terminal device 30 based on various information related to the virtual space received from the server 10. For example, the reproduction control unit 55 displays information related to the comment on the terminal device 30 based on information related to the comment arranged in the virtual space. In addition, the reproduction control unit 55 outputs a sound based on various information related to the virtual space, for example, outputs a sound corresponding to the comment being placed in the virtual space.

The input management unit 56 in one embodiment executes various processes related to management of input by the user. For example, when the input management unit 56 detects that the state where the comment is displayed on the user's gaze point has continued for a predetermined effective time, the input management unit 56 recognizes the input of “Like” for the comment, Information indicating the input of “Like” is transmitted to the server 10.

Next, the operation of the system 1 in one embodiment having such a function will be described. A user who uses the video distribution service in one embodiment can select a desired video from a plurality of videos provided in the video distribution service via the terminal device 30 and reproduce it on the terminal device 30. . When the user selects a real-time 360-degree video, the server 10 that has received the video distribution request from the terminal device 30 sends the 360-degree video received in real time from the video providing device 20 or the like to the terminal device 30 in a streaming format. To deliver. In one embodiment, two video playback screens having different screen configurations are provided as screens for playback of 360-degree video, and the user can select one of the screens to play the video. .

FIG. 7 is an example of a first video playback screen 60 that is one of the video playback screens. As shown in the figure, the first moving image playback screen 60 in one embodiment is configured as a display area 61 that displays a 360-degree moving image with a specific field of view. As described above, the display area 61 displays a moving image of a portion included in the visual field specified by the user among the entire visual field of the 360-degree moving image. For example, when the user's field of view (the direction of the line of sight) changes according to the user's operation to change the posture, tilt, orientation, and the like of the terminal device 30 or the flick / drag operation of the display area 61, Is displayed in the display area 61.

In addition, the display area 61 of the first moving image playback screen 60 is configured to display comments (comment objects) arranged in a virtual space included in the visual field so as to overlap the moving image of the portion included in the visual field of the user. Has been. Details will be described later.

In one embodiment, the first moving image playback screen 60 is a moving image included in a specific field of view of a 360-degree moving image using, for example, a VR (Virtual Reality) glass or a VR headset that is equipped with a smartphone or the like. Can be configured as a screen for a user who displays the image three-dimensionally. In this case, the first moving image playback screen 60 can be configured to be divided into a screen for the right eye and a screen for the left eye. Since it is considered that the user who uses the VR glasses or the VR headset cannot perform the flick / drag operation of the screen, the attitude, inclination, and orientation of the terminal device 30 (the VR glasses, the smartphone, etc.) Change the field of view by changing etc.

FIG. 8 is an example of a second video playback screen 70 that is one of a plurality of video playback screens. As shown in the drawing, the second moving image playback screen 70 in one embodiment is configured as a space including a virtual stage, and a display area 71 for displaying a 360-degree moving image with a specific field of view is virtual. Is placed on the stage. Similar to the display area 61 of the first moving image playback screen 60, the display area 71 displays a portion of the moving image included in the visual field specified by the user out of the entire visual field of the 360-degree moving image. When the user's field of view changes in response to an operation for changing the orientation, tilt, orientation, etc. of the terminal device 30 or a flick / drag operation of the display area 71, a moving image of a part included in the changed field of view is displayed. It is displayed in area 71.

Further, on the second video playback screen 70, as shown in the figure, the avatar 110 of the user who is viewing (playing back) the same video is arranged in the avatar display area 76 corresponding to the space before the virtual stage. ing.

Furthermore, as shown in the drawing, the second moving image playback screen 70 has a comment input area 72 and a comment transmission button 74 displayed as “Send” at the bottom of the screen. When the user inputs a desired character string or the like as a comment in the comment input area 72 and selects the comment transmission button 74, the input comment is transmitted to the server 10. Here, a comment placement process executed by the server 10 in response to receiving a comment will be described.

FIG. 9 is a flowchart showing an example of comment placement processing in an embodiment. In the comment placement process, first, as shown in the figure, the position on the virtual space where the input comment is placed is specified (step S110). In one embodiment, the position where the comment is arranged is specified to be a position on the virtual space included in the visual field (when the comment is input) of the user who input the comment. Information regarding the visual field of the user who input the comment (for example, the direction of the line of sight) can be received from the terminal device 30 together with the input comment, for example.

In various embodiments of the present invention, by applying various criteria, an arbitrary position included in the user's field of view can be specified as a position where a comment is placed. In one embodiment, the position where the comment is arranged is specified to be outside the range of the gaze area including the user's gaze point. FIG. 10A is a diagram for describing a user's gaze point FP and a gaze area FR according to an embodiment. In one embodiment, as illustrated, the center of the user's field of view V (the intersection of the direction of the user's line of sight and the virtual space) is defined as the gazing point FP that the user is gazing at, and the gazing point FP is the center. In addition, a circular area having a predetermined radius is defined as a gaze area FR. For example, the position where the comment is arranged is specified as a position in which the direction of moving (separating from) the gazing point FP is specified and moved in the specified direction by a distance corresponding to the radius of the gazing area FR. The direction of moving from the gazing point FP is specified at random, for example. As shown in FIG. 10B, for example, a building that appears in the moving image displayed in the display area 71 (view V) of the second moving image playback screen 70 is being viewed (the building is displayed at the gazing point FP). When the user inputs a comment, the position outside the range of the gaze area FR is a position moved in a direction away from the building, and is difficult to overlap the building.

When the position where the comment is arranged is specified in this way, next, the comment is arranged at the specified position in the virtual space (step S120), and this comment arrangement processing is ended. Specifically, information related to the comment is registered in the comment management table 51b. Here, of the information registered in the comment management table 51b, a time obtained by adding a predetermined arrangement duration (for example, 30 seconds) to the current time is set as the deletion time.

Here, the operation of the terminal device 30 when comments are arranged in the virtual space will be described. First, display of the first moving image playback screen 60 will be described. As described above, the display area 61 of the first moving image playback screen 60 according to the embodiment displays a 360-degree moving image with a specific field of view and comments (comment objects) arranged in the virtual space included in the field of view. ) Overlaid on the video. FIG. 11A illustrates the first moving image playback screen 60 displayed on the user terminal device 30 including the position in the virtual space in which the comment is arranged in the field of view. As shown in the drawing, in the display area 61 of the first moving image playback screen 60, the comment object 114 is displayed so as to overlap the moving image at a position on the virtual space where the comment is arranged.

FIG. 11B illustrates details of the comment object 114. In one embodiment, the comment object 114 includes a user's avatar 110 that has input a comment, and a balloon object 112. In the balloon object 112, the content of the comment (“Excellent!” In the example of FIG. 11B), the nickname of the user who input the comment (“by XXX” in the example of FIG. 11B), and the comment are input. The number of Likes is displayed.

As illustrated in FIG. 11A, when a user who is looking at a building that appears in a moving image (the building is displayed at the gazing point FP) inputs a comment, the comment is arranged at a position away from the gazing point FP. The corresponding comment object 114 is displayed at a position away from the building.

FIG. 12 illustrates the first moving image playback screen 60 in which a plurality of comment objects 114 based on comments input by each of a plurality of users are displayed in the display area 61. For example, when a plurality of users who are looking at the same building input a comment at the same timing, the direction away from the gazing point FP when specifying the position where the comment is arranged is for each comment (for example, randomly) Since it is specified, it is suppressed that the plurality of comment objects 114 are displayed in an overlapping manner at a position moved in the same direction from the gazing point FP.

FIG. 13 exemplifies a second moving image reproduction screen 70 displayed on the user terminal device 30 including the position where the comment is arranged in the field of view. On the second video playback screen 70, as described above, the avatar 110 of the user who views the same video is displayed in the avatar area 76, and the avatar 110 displayed in the avatar area 76 is arranged in the virtual space. The above-described balloon object 112 is added to the avatar 110 of the user who has input the comment being displayed.

Thus, in one embodiment, the comment placed in the virtual space is displayed on the terminal device 30 of the user including the position where the comment is placed in the field of view. Specifically, in the case of the first moving image playback screen 60, the comment object 114 is displayed in the display area 61 so as to be superimposed on the moving image at the position in the virtual space where the comment is arranged. In the case of 70, the balloon object 112 is added to the avatar 110 of the user who has input the comment in the avatar display area 76 and displayed.

In one embodiment, when a comment is arranged in a virtual space, the position of the comment and the position of the user's field of view (gaze point) in each terminal device 30 of a plurality of users viewing the same video Sound effects based on the relationship are output. For example, the sound effect output from the terminal device 30 may be configured such that the sound volume increases as the user's field of view (gaze point) is closer to the position where the comment is placed.

FIG. 14 illustrates the volume of the sound effect set based on the positional relationship between the position where the comment is placed and the user's field of view. For example, as shown in the figure, when the comment C1 is arranged at a position included in the user's visual field V, the sound effect S1 output from the terminal device 30 of the user is commented at a position not included in the user's visual field V. The volume is larger than the sound effects S2 and S3 when C2 and C3 are arranged. For example, the sound effect S2 when the comment C2 is arranged is the sound effect S3 when the comment C3 (the distance between the position where the comment is arranged and the user's visual field V is farther than the comment C2) is arranged. Is louder than. Here, the distance between the position where the comment is arranged and the visual field of the user is the angle between the direction in which the comment is arranged and the direction of the user's line of sight with respect to the center of the sphere S (θ1, θ2 in FIG. 14). Can be identified based on Thus, the user can feel a sense of realism by configuring the volume so that the closer the user's field of view (gaze point) and the position where the comment is placed, the greater the volume.

Further, for example, the sound effect is output as a sound in a direction (for example, right direction or left direction) where the comment is arranged with reference to the user's visual field (gaze point). For example, in the example of FIG. The sound effect S2 corresponding to the arrangement of the comment C2 arranged on the left side of the visual field is output as a sound audible from the left direction, and corresponds to the arrangement of the comment C3 arranged on the right side of the user's visual field. The sound effect S3 is output as a sound that can be heard from the right direction.

Thus, in one embodiment, when a comment is placed in a virtual space, a sound effect is output from each terminal device 30 of a plurality of users who view the same video, so the user is not included in the field of view. It is possible to know the input of a comment including a comment arranged at a position.

In one embodiment, a comment placed in the virtual space is deleted (arranged) when an erasing time set for each comment is reached. When the comment is erased, the comment object 114 displayed on the moving image in the display area 61 of the first moving image reproduction screen 60 is also erased (cannot be displayed).

In one embodiment, the balloon object 112 displayed in the avatar display area 76 of the second video playback screen 70 may be configured to be deleted in response to the deletion of the comment. It may be erased at different timings. That is, in one embodiment, the deletion of the comment object 114 on the first moving image playback screen 60 and the deletion of the balloon object 112 on the second moving image playback screen 70 can be controlled independently.

Here, in one embodiment, a comment having a larger number of accepted Likes is configured to have a longer time until the comment is deleted. Specifically, for example, every time the number of Likes reaches a predetermined number (for example, 10), a predetermined additional time (for example, 10 seconds) is added to the erasing time. Here, the operation related to the input of Like for the comment in the embodiment will be described.

First, the operation related to the input of Like via the first moving image playback screen 60 will be described. In one embodiment, if the state where the comment object 114 is located at the user's point of interest continues for a predetermined effective time (for example, 10 seconds), a Like input for the comment corresponding to the comment object 114 is accepted. .

For example, as illustrated in FIG. 15, when the state in which the comment object 114a is located at the point of interest (the center of the display area 61) continues for a valid time, “Like” for the comment corresponding to the comment object 114a is accepted. FIG. 16 exemplifies a change in the display of the comment object 114 until the state where the comment object 114 is positioned at the point of gaze continues for an effective time. In one embodiment, when a certain time (for example, 3 seconds) shorter than the valid time elapses, the background color of the balloon object 112 changes and the progress gauge 113 is added (i). Thereafter, the display of the progress gauge 113 changes with the passage of time, and when the valid time is reached (ii), the progress gauge 113 is erased and the background color of the balloon object 112 is restored to the original, and the input of “Like” is accepted. And the display of the number of Likes is updated (iii).

Upon accepting “Like” via the terminal device 30, the server 10 updates the number of Likes in the comment management table 51b. Further, as described above, the erase time can be updated (added) as the number of Likes increases.

Next, an operation related to Like input via the second moving image playback screen 70 will be described. In one embodiment, in response to the user selecting the balloon object 112 displayed in the avatar display area 76 of the second video playback screen 70, the input of “Like” for the comment corresponding to the balloon object 112 Is accepted. The operation of accepting the input of “Like” and updating the display of the number of Likes is the same as in the case of the first moving image playback screen 60 described above.

In the embodiment described above, a 360-degree moving image distributed in a streaming format in real time (live streaming distribution) is stored in the information storage unit 51 of the server 10, and the stored moving image is used as a request from the terminal device 30. Accordingly, it can be configured so that it can be played back later. In this case, the comment input at the time of live streaming distribution is arranged in the virtual space according to the information managed in the comment management table 51b. Specifically, corresponding comments are arranged in the virtual space according to the arrangement time, arrangement position, and deletion time of the comment management table 51b, and then deleted.

In the above-described embodiment, the comment is input via the second video playback screen 70. However, a comment corresponding to the first video playback screen 60 can be input. It can also be configured. The first moving image playback screen 160 in another embodiment illustrated in FIG. 17 has a display area 61 similar to the first moving image playback screen 60 described above, and the second moving image is displayed at the lower end of the display area 61. A comment input area 72 and a comment transmission button 74 similar to those on the reproduction screen 70 are arranged. In various embodiments of the present invention, comment input can also be realized by applying a voice input technique.

In the above-described embodiment, a comment is exemplified as input information that is input from a user and arranged in the virtual space. However, in the embodiment of the present invention, the input information is not limited to a comment. For example, various information that can be input from the user, such as a stamp and an icon, can be included in the input information.

In the above-described embodiment, the sound effect output according to the arrangement of the comment is configured such that the sound volume increases as the user's field of view and the position where the comment is arranged are closer to each other. Is not limited to those whose volume changes based on the positional relationship between the position where the comment is placed and the user's field of view, but instead of or in addition to this, the timbre and pitch of the sound effect Etc. can vary.

In the various embodiments of the present invention described above, each terminal device 30 of a plurality of users is configured with a plurality of moving images that are configured as a moving image having a wide-angle visual field and a virtual space is associated with the entire visual field. When the user's field of view is displayed and a comment is received from the user, the position in the virtual space included in the user's field of view is specified and arranged, and the arranged position is viewed according to the arrangement of the comment. The comment is displayed on the terminal device 30 of the user included. Therefore, the user of the terminal device 30 in which the comment is displayed has a field of view close to the field of view of the user who has input the comment. As described above, the embodiment of the present invention can appropriately display information such as a comment input in a moving image that may have different fields of view among users.

The processes and procedures described in this specification are realized by software, hardware, or any combination thereof other than those explicitly described in the embodiment. More specifically, the processes and procedures described in this specification are performed by mounting logic corresponding to the processes on a medium such as an integrated circuit, a volatile memory, a nonvolatile memory, a magnetic disk, or an optical storage. Realized. Further, the processes and procedures described in this specification can be implemented as a computer program and executed by various computers.

Even if the processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple Component and / or multiple modules. In addition, even though the data, tables, or databases described herein are described as being stored in a single memory, such data, tables, or databases are provided on a single device. Alternatively, the data can be distributed and stored in a plurality of memories or a plurality of memories arranged in a plurality of devices. Further, the software and hardware elements described herein may be implemented by integrating them into fewer components or by decomposing them into more components.

In the present specification, when the constituent elements of the invention are described as one or a plurality, or when they are described without being limited to one or a plurality of cases, they should be understood separately in context. The component may be either singular or plural.

DESCRIPTION OF SYMBOLS 1 System 10 Server 20 Movie providing device 30 Terminal device 40 Communication network 51 Information storage unit 52 Movie distribution control unit 53 Virtual space management unit 55 Playback control unit 56

Input management unit

60, 160 First movie playback screen 70 Second movie Playback screen

Claims

A system that displays video with a specific field of view,
Comprising one or more computer processors;
In response to executing the readable instructions, the one or more computer processors
A specific video that is configured as a video having a wide-angle field of view and a virtual space is associated with the entire field of view on each of the terminal devices of the plurality of users including the first and second users. Displaying in each field of view;
When the first input information is received from the first user, the first position on the virtual space included in the visual field of the first user is specified, and the first position is set to the first position. Placing input information; and
Displaying the first input information on the terminal device of the second user including the first position in the field of view in accordance with the arrangement of the first input information.
system.
The system of claim 1, comprising:
The specific moving image is configured as a moving image having a visual field of 360 degrees in at least the horizontal direction,
The virtual space is configured as an inner surface of a virtual sphere.
system.
The system according to claim 1 or 2, wherein the input information is a comment.
The system according to any one of claims 1 to 3, wherein in the step of displaying the input information, the first input information is displayed so as to overlap the specific moving image at the first position in the virtual space.
5. The system according to claim 1, wherein the placing step includes specifying the first position so that the position is outside a first range including a gazing point of the first user. .
The arranging step specifies the direction of movement from the gazing point of the first user, the position moved in the specified direction, and the position outside the first range. 6. A system according to any of claims 1 to 5, comprising identifying the location of one.
The one or more computer processors further support the first input information when the first input information is displayed on the second user's gaze point for a first time. The system according to claim 4, wherein the receiving step is executed.
The system according to any one of claims 5 to 7, wherein the gazing point is a substantial center of a user's visual field.
The one or more computer processors further execute a step of releasing the arrangement of the first input information when a second time elapses after being arranged at the first position. 8. The system according to any one of 8.
10. The system according to claim 9, wherein
The one or more computer processors further execute a step of accepting support for the first input information;
The step of releasing includes releasing the arrangement of the first input information so that the second time becomes longer as the number of received supports increases.
system.
The one or more computer processors may further cause each terminal device of the plurality of users to receive the visual field and the first position of the plurality of users according to the arrangement of the first input information. The system according to claim 1, wherein a step of outputting a sound based on the positional relationship is executed.
The system according to claim 11, wherein the outputting step includes outputting a sound so that the volume of sound increases as the visual field of the user is closer to the first position.
The step of outputting causes the plurality of users to output a first sound to the terminal device of the second user including the first position in the field of view according to the arrangement of the first input information. The system according to claim 11, further comprising: causing a terminal device of a third user who does not include the first position included in the visual field to output a second sound different from the first sound.
14. The system according to claim 1, wherein the step of displaying the specific moving image includes delivering the specific moving image to each terminal device of the plurality of users by live streaming.
A method executed by one or more computers to display a moving image with a specific field of view,
A specific video that is configured as a video having a wide-angle field of view and a virtual space is associated with the entire field of view on each of the terminal devices of the plurality of users including the first and second users. Displaying in each field of view;
When the first input information is received from the first user, the first position on the virtual space included in the visual field of the first user is specified, and the first position is set to the first position. Placing input information; and
Displaying the first input information on the terminal device of the second user including the first position in the field of view in accordance with the arrangement of the first input information.
Method.
A program that displays a video with a specific field of view,
In response to being executed on one or more computers, the one or more computers,
A specific video that is configured as a video having a wide-angle field of view and a virtual space is associated with the entire field of view on each of the terminal devices of the plurality of users including the first and second users. Displaying in each field of view;
When the first input information is received from the first user, the first position on the virtual space included in the visual field of the first user is specified, and the first position is set to the first position. Placing input information; and
Displaying the first input information on the terminal device of the second user including the first position in the field of view according to the arrangement of the first input information.
program.