JP6566698B2 - Display control apparatus and display control method - Google Patents

Description

  The present invention relates to a display control device, and more particularly, to a display control device and a display control method for reproducing a moving image whose viewpoint can be moved, such as a panoramic moving image and a omnidirectional moving image.

  In recent years, various video distribution services represented by YouTube (registered trademark) have become widespread. In addition, there is a moving image distribution service in which a user can freely add a comment to a moving image. In such a service, the comment added to the user can be displayed superimposed on the moving image when the moving image is played back.

  In addition, with the development of moving image distribution technology, moving image distribution technology that enables moving viewpoints such as panoramic moving images and omnidirectional moving images has been realized. By using a moving image distribution technology that enables movement of viewpoints such as panoramic moving images and omnidirectional moving images, a user can view moving images from various angles arbitrarily. As a technique related to such a moving image distribution technique, Patent Document 1 discloses a moving image distribution technique that enables user viewpoint information to be collected, logged, and used.

  On the other hand, like a car navigation system, a display device that displays auxiliary information such as traffic jam information to a user is widespread. As a technique related to such a car navigation system, Patent Document 2 discloses a car navigation system that can easily display traffic jam information outside the display range of a display device.

JP 2003-132362 A JP 2001-304895 A

  However, with the technologies disclosed in Patent Literature 1 and Patent Literature 2, when viewing a moving image such as a panoramic video or a omnidirectional video, it is difficult for the user to select and view an appropriate viewpoint. .

  The present invention has been made in view of the above circumstances, and automatically selects and reproduces an appropriate viewpoint in a display control apparatus that reproduces a moving image such as a panoramic video or a omnidirectional video. It is an object to provide a display control device.

In order to achieve the above object, the display control device of the present invention provides:
Display control means for controlling to display a part of all directions that can be displayed in the video as a display range;
Switching means for switching between the first display mode and the second display mode;
In the case of the first display mode, when there is a drag operation on the display area displaying the moving image, which is a display range change operation from the user without automatically changing the display range. , Change the display range of the video displayed by the display control means,
Wherein when the second display mode, even without the change operation, have a control means for controlling so as to automatically change the display range of the video being displayed by said display control means,
In the second display mode, the switching unit switches to the first display mode in response to the drag operation on the display area .

  According to the display control apparatus of the present invention, it is possible to automatically select and reproduce an appropriate viewpoint when reproducing a moving image whose viewpoint can be moved, such as a panoramic moving image and an omnidirectional moving image.

It is a block diagram for demonstrating the structure of the Example which concerns on this invention. It is the figure which showed the network structure of PC and the moving image delivery server which concerns on this invention. It is the figure which showed the external appearance of the moving image player which concerns on this invention. It is a figure explaining the switching operation | movement of the viewing position and viewing position of the moving image concerning this invention. It is a figure which shows the structural example of the user operation information which concerns on this invention, and viewing assistance information. It is a figure explaining the totalization process performed with the moving image delivery server which concerns on this invention. It is a figure explaining the viewing direction automatic switching process flow of the moving image player which concerns on this invention. It is a figure explaining the instruction | indication method of the viewing direction of the moving image player which concerns on this invention. It is a figure explaining operation of the priority setting of the user performed with the video player concerning the present invention.

  Preferred embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram showing an example of the configuration of a display control apparatus 100 to which each embodiment of the present invention can be applied. As shown in FIG. 1, the display control apparatus 100 can be configured using a general personal computer (hereinafter, PC).

  In FIG. 1, a CPU 101, a hard disk (HD) 102, a memory 103, a display control unit 104, an input unit 105, and a communication I / F 106 are connected to an internal bus 111. Each unit connected to the internal bus 111 can exchange data with each other via the internal bus 111.

  The hard disk 102 stores image data, other data, various programs for operating the CPU 101, and the like. For example, the CPU 103 including a RAM controls each unit of the display control apparatus 100 using the memory 103 as a work memory in accordance with a program stored in the hard disk 102, for example. The program for operating the CPU 101 is not limited to being stored in the hard disk 102 but may be stored in advance in a ROM (not shown), for example.

  The input unit 105 receives a user operation, generates a control signal corresponding to the operation, and supplies the control signal to the CPU 101. For example, the input unit 105 includes character information input devices such as a keyboard and pointing devices such as a mouse and a touch panel as input devices that accept user operations. The touch panel is an input device that outputs coordinate information according to a position touched with respect to, for example, a planar input unit. The CPU 101 controls each unit of the display control apparatus 100 according to a program based on a control signal generated and supplied by the input unit 105 in accordance with a user operation performed on the input device. Thereby, it is possible to cause the display control apparatus 100 to perform an operation according to the user operation.

  The display control unit 104 outputs a display signal for displaying an image on the display 110. For example, a display control signal generated by the CPU 101 according to a program is supplied to the display control unit 104. The display control unit 104 generates a display signal based on the display control signal and outputs the display signal to the display 110. For example, the display control unit 104 causes the display 110 to display a GUI screen constituting a GUI (Graphical User Interface) based on a display control signal generated by the CPU 101.

  When a touch panel is used as the input unit 105, the input unit 105 and the display 110 can be integrally configured. For example, the touch panel is configured such that the light transmittance does not hinder the display of the display 110 and is attached to the upper layer of the display surface of the display 110. Then, the input coordinates on the touch panel are associated with the display coordinates on the display 110. Thereby, it is possible to configure the GUI as if the user can directly operate the screen displayed on the display 110.

  A communication interface (I / F) 106 communicates with a network 120 such as a LAN or the Internet based on the control of the CPU 101.

  FIG. 2 is a diagram showing an example of connection between the PC 100 and the moving image distribution server 10 of the present invention. Similar to the PC 100, the moving image distribution server 10 shown in FIG. 2 includes a CPU, an HD, a memory, a display control unit, a display, an input unit, and a communication I / F. The moving image distribution server 10 and the PC 100 are connected via a network 120. The moving image distribution server 10 is also connected to a plurality of other PCs.

  The operation of each embodiment of the present invention will be described below with reference to FIGS.

<Example 1>
In the first embodiment, an example will be described in which a moving image that can be arbitrarily switched between the viewing range and the viewing direction distributed from the moving image distribution server 10 is played back by the application on the PC 100 while automatically switching the viewing direction.

  FIG. 3A is a diagram illustrating an appearance of the moving image player 200 which is an application on the PC 100 according to the present embodiment. The moving image player 200 includes a hatched moving image display area 201, a playback stop button 202, a progress bar 203, a time display area 204, and a viewing position automatic switching button 205. The viewing direction will be described in detail later.

  A hatched moving image display area 201 is an area for displaying a moving image, and a reproduction stop button 202 is a button for reproducing and stopping the moving image. The progress bar 203 displays the progress of the moving image playback. The time display area 204 displays the total time of the moving image and the time of the moving image that has been played so far. The viewing position automatic switching button 205 is a button for turning on / off the automatic switching function of the viewing position. By pressing the viewing position automatic switching button 205, the viewing position switching function is turned on. When the automatic viewing position switching function is ON, the video player 200 automatically performs the viewing position switching operation based on the data transmitted from the video distribution server 10. The data transmitted from the moving image distribution server 10 and the automatic switching process of the viewing position of the moving image player 200 will be described in detail later.

  By pressing the viewing position automatic switching button 205 again while the viewing position switching function is ON, the viewing position switching function is turned OFF.

  When the viewing position switching function is in the OFF state, the user can arbitrarily switch the viewing position of the moving image by performing a drag operation using the mouse pointer 300 within the moving image display area 201 as shown in FIG. Can do. At this time, information operated by the user is transmitted to the motion distribution server 10. The data transmitted to the moving image distribution server 10 will be described in detail later.

  Next, the switching operation between the viewing position and the viewing position by the user's mouse operation will be described with reference to FIGS.

  FIG. 4A shows an example in which hemispherical moving image data is transmitted from the moving image distribution server 10. In FIG. 4A, 401 illustrated in a hemisphere indicates moving image data distributed from the moving image distribution server 10. Reference numeral 403 indicated by a dotted line indicates a viewing area displayed on the video player 200. The viewing position of the viewing area 402 is represented using the viewing angle θ and the viewing elevation angle φ. The viewing angle θ is an angle formed by the center of the viewing area 403 and the reference line 402a on the reference plane 404. The viewing elevation angle φ is an angle formed by the center of the viewing area 403 and the reference plane 401. The user can move the viewing area 403 freely by changing the viewing angle θ and the viewing elevation angle φ by performing a drag operation up, down, left and right using the mouse pointer 300.

  FIG. 4B shows an example when cylindrical moving image data is transmitted from the moving image distribution server 10. In the case of moving image data on a cylinder, the viewing position is represented using the viewing angle θ. The definition of the viewing angle θ is the same as that in FIG. In FIG. 4B, the user can move the viewing area 403 freely by changing the viewing angle θ by dragging left and right using the mouse pointer 300.

  FIG. 4C shows an example in which panoramic video data is transmitted from the video distribution server 10. In the case of panoramic video data, the viewing position is represented by a viewing horizontal position x and a viewing vertical position y from the reference point 402b to the upper left of the viewing area 403. In FIG. 4C, the user can move the viewing area 403 freely by changing the viewing position x and the viewing position y by dragging the mouse pointer 300 up, down, left and right. .

  As shown in FIGS. 4A to 4C, the user can use the video player 200 to view an arbitrary area of the video data distributed from the video distribution server. In FIG. 4A, the moving image data has been described as a hemisphere, but can be expressed as spherical data.

  In the following description, it is assumed that hemispherical moving image data as shown in FIG. 4A is distributed from the moving image distribution server 10.

  Next, a configuration example of data transmitted and received between the moving image distribution server 10 and the PC 100 will be described with reference to FIG.

  FIG. 5A is an example of user operation information transmitted from the PC 100 to the moving image distribution server 10 when the above-described viewing position switching function is OFF. In the example shown in FIG. 5A, the user operation information includes a moving image ID for identifying the moving image being played back by the moving image player 200, a viewing angle θ and a viewing elevation angle φ representing the viewing position, and the user playing the moving image. Stored is an operation time indicating at which time the operation was performed. Note that the data in FIG. 5A is an example, and only the viewing angle θ may be transmitted as information indicating the viewing position as in the example in FIG. 4B. Further, as in the example of FIG. 4C, the viewing horizontal position x and the viewing vertical position y may be transmitted. In addition to this, parameters handled at the time of moving image reproduction by the moving image player 200 may be transmitted simultaneously.

  The moving image distribution server 10 totals operation information from the PC 100 shown in FIG. 5A and totals at which viewing position the user is viewing the moving image every predetermined time of the moving image. For example, as shown in FIG. 6, the viewing angle θ and the like are summed up every 10 seconds of the moving image playback time, and information on which viewing angle θ is frequently viewed at which time is summed up. In FIG. 6, every 10 seconds has been described. However, the time interval for performing the aggregation process may be performed in any detail.

  FIG. 5B is an example of viewing assistance information data transmitted from the moving image distribution server 10 to the PC 100 together with moving image data. The moving image distribution server 10 transmits moving image data to the PC 100 and simultaneously transmits viewing assistance information related to the moving image to be transmitted.

  The viewing assistance information transmitted from the moving image distribution server 10 may be configured to be transmitted before moving image data. In this way, the moving image player 200 of the PC 100 can switch in advance to a viewing direction that is frequently viewed by other users based on viewing direction information.

  The viewing assistance information stores the ID of the moving image being reproduced by the user, the number of views of other users, the viewing angle θ and viewing elevation angle φ representing the viewing position, the time of the moving image, and the like. As shown in FIG. 5B, the moving image distribution server 10 sets, for example, the top three viewing angles θ and viewing elevation angles φ, which are viewed by other users, for each counting time, as shown in FIG. Are transmitted to the PC 100 as viewing assistance information.

  In the present embodiment, it has been described that up to the top three with the largest number of views are transmitted, but it may be configured to transmit all the results of viewing angle distribution in FIG.

Further, in this embodiment, it has been described that the transmission data from the PC 100 to the action distribution server 10 is user operation information. You may make it transmit with the attribute information of and.
By transmitting the comment data as shown in FIG. 5C from the PC 100 to the moving image distribution server 10, not only the viewing information but also the number of comments added by other users in the moving image at the time of the above-described totaling process. It can be tabulated. Thereby, it is possible to transmit the viewing position and time where more comments are made in accordance with the viewing assistance information in FIG. 5B and the moving image distribution server 10 to the PC 100.

  By transmitting the user attribute information of FIG. 5 (d) to the moving image distribution server 10, it is possible to classify the aggregation results for each user attribute information in the above-described aggregation process. For example, if the counting results are classified for each viewing area and the viewing auxiliary information shown in FIG. 5B is switched, the possibility of obtaining a viewing position automatic switching operation that is preferable for the viewing user increases.

  Next, viewing position automatic switching processing in the video player 200, which is an application on the PC 100, will be described using the flowchart of FIG. Each process in this flowchart is realized by the CPU 101 developing and executing a program stored in the HD 102 in the memory 103.

  When the user presses the viewing position automatic switching button 205 of the moving image player 200 during the reproduction of the moving image and the viewing position automatic switching function is turned on, the viewing position automatic switching process is started.

  Note that the moving image data and viewing assistance information shown in FIG. 5B are transmitted in advance from the moving image distribution server 10 to the PC 100 via the communication I / F 106 at the start of moving image reproduction. Is already written. By transmitting the viewing assistance information to the PC 100 in advance, it is possible to switch in advance to a viewing direction that is frequently viewed by other users based on the viewing direction information.

  When the processing is started, first, the CPU 101 acquires viewing position information including the current viewing angle θ and viewing elevation angle φ of the moving image player 200 from the memory 103 (S501). Next, the CPU 101 acquires the current moving image playback time from the memory 103 (S502). The movie playback time is the time that has elapsed since the movie playback started. In S502, the CPU 101 searches the viewing assistance information obtained for whether the playback time is within a predetermined time range and whether viewing assistance information at a future time is written in the memory 103 (S503). Here, the predetermined time used for the search condition is predetermined as 10 seconds or the like.

  Next, the CPU 101 determines whether viewing assistance information has been found (S504). If found, the process proceeds to S505, and if not found, the process proceeds to S511. When the viewing assistance information is found, the CPU 101 selects data that has more than a predetermined number of viewings from the viewing assistance information, and informs the user by displaying the direction of viewing candidates as icons based on the selected data ( S505).

  The icon display in S505 will be described with reference to FIGS. In FIG. 8A, the area displayed by the video player 200 is shown as 403, viewing candidate 1 is 405a, and viewing candidate 2 is 405b.

  As shown in FIG. 8B, the CPU 101 displays icons 206a and 206b indicating directions to the viewing candidates 405a and 405b in the moving image display area 201 of the moving image player 200, respectively. When performing icon display, as shown in FIG. 8B, the icon size, color, and the like are changed according to the priority of the viewing candidate. Further, when a comment or the like can be displayed superimposed on the moving image, the comment 207a and the comment 207b related to the viewing candidate 1 are displayed in the direction of the viewing candidate 1 and related to the viewing candidate 2 as shown in FIG. Control is performed so that the comment 207c is displayed from the viewing candidate 2 direction. The sizes and shapes of the comments 207a to 207c are also displayed in different formats depending on the priority.

  The priority of a candidate is determined using the frequency of viewing by other users, the number of comments, and the like. By displaying icons and comments in this way, the user can intuitively know in which direction the highlight of the moving image exists. Further, by changing the color of the icon or comment according to the priority, the user can intuitively grasp the direction with higher priority.

  Next, the CPU 101 calculates a movement amount for changing the viewing position from the viewing position information of the data with the largest number of viewing views among the viewing candidates and the viewing position information currently displayed by the video player. As shown in FIG. 8A, when the destination is candidate 1 and the current position is 403, the moving amount of the viewing angle θ is calculated as Δθ. The viewing elevation angle φ is calculated in the same manner, but the description is omitted here.

  After calculating the movement amount, the CPU 101 determines whether the movement amount is equal to or smaller than a predetermined value (S507). If the movement amount is equal to or smaller than the predetermined value, the viewing position is moved as it is (S509), and the predetermined value is exceeded. After setting the movement amount to a predetermined value (S508), the viewing position is moved according to the movement amount (S509). After the viewing position is moved, the CPU 101 determines whether or not the viewing position after the movement has reached the position of the aforementioned viewing candidate (S510). If not, the process returns to S506, and steps S506 to S510 are performed until the movement is completed. Repeat the process.

  By executing the viewing position moving process as described above, it is possible to avoid a situation in which the viewing position suddenly changes and it becomes impossible to grasp which position the user is viewing.

  After the movement is completed, the CPU 101 determines whether or not the automatic viewing position switching function is turned off. If it is turned off, the CPU 101 ends the processing. If not turned off, the CPU 101 returns to S501 and goes from S501 to S511. Repeat the process until the video ends.

  As described above, by automatically switching the viewing direction of the moving image based on the viewing assistance information distributed from the moving image distribution server 10, the user automatically views the direction in which other users are viewing a lot. It becomes possible. Further, since the viewing direction is switched based on the result of the operation history of other users obtained from the moving image distribution server 10, it is possible to reduce the user from overlooking the highlight of the moving image.

  In this embodiment, it has been described that the viewing position automatic switching button 205 is pressed to turn off the viewing position automatic switching. However, the function is automatically turned off by detecting the user's drag operation. Also good.

  Further, in the present embodiment, it has been described that the video distribution server 10 aggregates the number of viewing of the user and uses the viewing position information having the highest number of viewing as the viewing auxiliary information. However, the user can designate the priority order at any position. You may comprise as follows.

  A method for specifying the user priority will be described with reference to FIG. When the user designates an arbitrary place on the screen with the mouse pointer 300 and right-clicks while playing the moving image on the moving image player 200, a priority designation menu 209 is displayed as shown in FIG. Is done. Note that reference numeral 208 in FIG. 9A represents a position designated by the user. The user selects an arbitrary priority from the menu after selecting the position. For example, when priority 2 is selected as shown in FIG. 9B, the angle θ of 208, the elevation angle φ, and the reproduction of the movie are transmitted from the PC 100 to the movie distribution server 10 as shown in FIG. 9C. Time is sent.

  The transmitted priority information is used in the aggregation process in the moving image distribution server 10 described above. With this configuration, it is possible to distribute the viewing assistance information described above using not only the viewing direction of the user but also the priority specified by the user. In the drawing, for convenience, the position 208 designated by the user is illustrated using a star, but the actual GUI may be configured so that nothing is displayed.

  Further, in the present embodiment, the contents mainly related to display such as icon display, comment display, and viewing direction switching in the video player 200 have been described, but the present invention can be similarly applied to sound information of a video. For example, in addition to displaying icons and comments in the state of FIG. 8B, processing such as processing and reproducing sound so that sound can be heard from the viewing candidate direction may be performed.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program code. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

10 video distribution server, 100 PC, 200 video player

Claims (18)

Display control means for controlling to display a part of all directions that can be displayed in the video as a display range;
Switching means for switching between the first display mode and the second display mode;
In the case of the first display mode, when there is a drag operation on the display area displaying the moving image, which is a display range change operation from the user without automatically changing the display range. , Change the display range of the video displayed by the display control means,
Wherein when the second display mode, even without the change operation, have a control means for controlling so as to automatically change the display range of the video being displayed by said display control means,
In the second display mode, the switching unit switches to the first display mode in response to the drag operation on the display area . Wherein, wherein when the second display mode, characterized by controlling to automatically change the display range of the video based on the supplementary information about the video to be displayed, and information indicating the view direction The display control apparatus according to claim 1. The display control apparatus according to claim 2, wherein the auxiliary information is information related to a browsing situation of a plurality of users regarding a displayed moving image . The control means controls to display a viewpoint direction determined based on the number of viewings for each viewpoint direction among a plurality of viewpoint directions based on the auxiliary information when the display range is automatically changed. the display control device according to claim 2 or 3, characterized in that. The control means displays a viewpoint direction determined based on the number of comments for each viewpoint direction out of information indicating a plurality of viewpoint directions based on the auxiliary information when the display range is automatically changed. the display control device according to claim 2 or 3, wherein the controller controls so. Before SL auxiliary information, the display control device according to any one of claims 2 to 5, characterized in that the information about the viewing conditions of a plurality of users associated with each of a plurality of sections in the duration of the video . The control means, when automatically changing the display range, displays the display item indicating the viewpoint direction based on the auxiliary information associated with the predetermined range after the current playback time together with the video being played back. The display control apparatus according to claim 6 , wherein the display control apparatus controls the display. The control means automatically changes the display range based on information indicating a viewpoint direction included in the auxiliary information associated with the predetermined range after the current reproduction time. automatically display control apparatus according to claim 6 or 7, wherein the controller controls to change. Wherein, when automatically changing the display range, any one of claims 6 to 8, characterized in that to limit the change amount of the display range of between video frames below a certain value The display control apparatus according to 1. The control means further includes
Control to acquire a moving image and auxiliary information corresponding to the moving image via a network, and information indicating a viewpoint direction corresponding to a display range of the moving image displayed by the change operation the display control device according to any one of claims 2 to 9, wherein the controller controls to transmit to. When there is additional information added to another range different from the current display range in the moving image, the display control unit is configured to add the additional information at a position corresponding to the other range in the current display range. the display control device according to any one of claims 1 to 10, wherein the controller controls to display by superimposing the display item based on. The display control apparatus according to claim 11 , wherein the additional information is comment information of a user who has viewed a moving image . The control means specifies a moving image display range based on first angle information that is an angle from a reference line in a reference plane and second angle information that is an angle from the reference plane. The display control apparatus according to any one of claims 1 to 12 . Wherein the display control unit, the display control device according to any one of claims 1 to 13, wherein the controller controls to display a display range a part of the image of the above spherical hemisphere. The video display control device according to any one of claims 1 to 14, characterized in that a celestial sphere image. A display control step for controlling the display range to display a part of the video in all directions that can be displayed;
A switching step for switching between the first display mode and the second display mode;
In the case of the first display mode, when there is a drag operation on the display area displaying the moving image, which is a display range change operation from the user without automatically changing the display range. , Change the display range of the video displayed by the display control step,
Wherein when the second display mode, even without the change operation, have a control step of controlling so as to automatically change the display range in a moving image displayed by said display control step,
The switching step switches to the first display mode in response to the drag operation on the display area in the second display mode. . The program for functioning a computer as each means of the display control apparatus as described in any one of Claims 1 thru | or 15 . Computer, according to claim 1 to 15 computer-readable storage medium which stores a program to function as each unit of the display control device according to any one of.