JP7488497B2 - Shared information processing device and control program - Google Patents

Description

The present invention relates to a mechanism for sharing elements among multiple users who each view a common viewing subject.

There is known a communication system in which comments entered in response to the content being played are shared among multiple users. For example, in the communication system described in Patent Document 1, a user terminal acquires comments to be displayed together with the content from a comment server before playback of the content starts, and excludes old comments from being displayed so that the number of comments displayed in each of the multiple time segments into which the timeline is divided is equal to or less than a predetermined number.

JP 2011-119847 A

In the communication system described in Patent Document 1, a process for leveling out the number of comments displayed over time along with the content is executed before the playback of the content starts. Therefore, it is not possible to handle sudden increases in load caused by comments added while the content is being played.

The problem that this invention aims to solve is to suppress sudden increases in the load on the terminal that displays an element in a system in which elements are shared among multiple users who each view a common viewing target.

To solve the above problem, the present invention identifies a predetermined number of shared elements as an element group from among shared elements whose timing of displaying the viewing target on the terminal of a user viewing the viewing target is associated with the viewing target based on the timing, and replaces the element group with a number of alternative elements that is less than the predetermined number.

A shared information processing device according to one embodiment of the present invention includes an identification means for identifying a predetermined number of shared elements from among shared elements associated with a viewing target based on the timing at which the viewing target is to be displayed on a terminal of a user viewing the viewing target, and a replacement means for replacing the element group identified by the identification means with alternative elements whose number is less than the predetermined number.

According to the present invention, it is possible to prevent sudden increases in load on a terminal that displays shared elements.

FIG. 1 is a diagram illustrating a configuration of a video distribution system (embodiment). 1 is a block diagram showing the electrical configuration of a server according to an embodiment of the present invention; 1 is a block diagram showing an electrical configuration of a mobile terminal according to an embodiment of the present invention. 1 is a diagram showing a display state of a screen of a mobile terminal (embodiment); FIG. 13 is a diagram showing a screen displaying an onomatopoeia image (Example). FIG. 13 is a diagram showing a screen displaying a second image (Example); 10 is a timing chart showing a case where a combined image is displayed (embodiment example). 10 is a timing chart showing a case where the second image is replaced (embodiment example). FIG. 1 is a functional block diagram of a server for video distribution (embodiment); FIG. 1 is a functional block diagram relating to onomatopoeia display of a mobile terminal (embodiment); 1 is a flowchart showing the flow of an onomatopoeia display process executed on a mobile terminal (embodiment). 10 is a flowchart showing the flow of an onomatopoeia display process executed by the server. 10 is a flowchart showing the flow of an onomatopoeia display process executed by the server. 13 is a timing chart showing a case where a combined image is displayed (Modification 2).

1. Embodiment
[1-1. Main features]
This embodiment relates to a mechanism for sharing elements among multiple users who each view a common viewing target. In this embodiment, in order to suppress a sudden increase in load on a terminal that displays the shared elements, a predetermined number of the shared elements are identified as an element group based on a timing of displaying the shared elements on a terminal of a user who is viewing the viewing target and associated with the viewing target, and the element group is replaced with a number of alternative elements that is less than the predetermined number.

"Audiovisual object" refers to an audiovisual object that the user can grasp by at least one of sight and hearing. Audiovisual objects are a concept that encompasses video, audio, and combinations of these. Audiovisual objects include objects that are not converted into signals, and objects that are reproduced based on a signal. The signal of the reproduced object can be recorded on a recording medium or carried by a carrier wave, for example. The reproduced object is reproduced as at least one of video and audio, for example, by a device with a playback function.

The "progression" of a viewing object refers to the transition of the viewing object over time. Transition is a concept that encompasses both constancy and change. Constancy means that visual and auditory information continues to remain constant. Change means that visual and auditory information is not constant. Changes can be, for example, irregular change, regular change, periodic repetition, temporary disappearance, or a combination of these. If the viewing object is an object that is not signalized, the timing (point in time) of the progression can be specified by time information. If the viewing object is an object that is played based on a signal, the point in time of the progression can be specified by time information or the elapsed time from the start of the progression (i.e., the position on the timeline).

"Viewing" an object to be viewed means grasping the progress of the object to be viewed by at least one of sight and sound. When the object to be viewed is an object that is not signalized, the object to be viewed is viewed almost simultaneously with the progress. Viewing an object that is not signalized includes, for example, viewing a sports game at a stadium, viewing a natural phenomenon (for example, a solar eclipse, a lunar eclipse, an aurora, a sunrise, a sunset, a sunset, a rare weather phenomenon, etc.) at an appropriate place, and viewing an event (for example, a festival, a street speech, etc.) at an appropriate place. When the object to be viewed is an object that is reproduced based on a signal, the object to be viewed is viewed when it is reproduced. Viewing a reproduced object includes, for example, viewing a television broadcast, viewing a radio broadcast, viewing a video distributed online, and viewing music distributed online.

A user (viewer) viewing a viewing target can associate a specific element (shared element) with the viewing target by performing a specific operation (additional operation). The specific operation is, for example, a pointing operation on an input element (operation element). The input element is, for example, a button displayed on the screen. The pointing operation is, for example, a touch operation on a touch panel, and a combination of a positioning operation and a click operation of a pointer displayed on the screen. When a specific element is associated with a viewing target, the timing for displaying the shared element during the progress of the viewing target is associated with the viewing target. The element associated with the viewing target can be displayed on the screen at the associated display timing during the progress of the viewing target.

The predetermined elements are visual elements that express the user's reaction to the viewing object. The user's reaction is, for example, the user's emotion. The visual elements are, for example, still images, animated images, moving images, text, or a combination of these. An image that expresses the user's emotion toward the viewing object is sometimes called an emotion image. An emotion image is, for example, an image that illustrates an onomatopoeia (also called an onomatopoeic word, mimetic word, or mimetic word) that shows the user's emotion. An image that illustrates an onomatopoeia is sometimes called an onomatopoeia image. The predetermined elements can also be shared among multiple users viewing a common viewing object. The elements that can be shared are transmitted and received online and displayed on the screen of a terminal. The terminal having a screen on which the elements that can be shared are displayed may be the same device as the device that plays the object that is the viewing object, or it may be a different device.

When identified "based on timing," a predetermined number of shared elements whose display timings are in a proximal relationship can be identified as an element group. The display timings of two shared elements being in a proximal relationship means that the time difference between the timing at which one is displayed on the screen and the timing at which the other is displayed on the screen is equal to or less than a predetermined threshold. The display timings of three or more shared elements being in a proximal relationship means that the display timings of two shared elements arbitrarily selected from the three or more shared elements are in a proximal relationship. When determining whether or not there is a proximal relationship in terms of display timing, a point between the start and end of the display of each shared element is used as the display timing. The point in time used as the display timing is, for example, the start point, the end point, or a predetermined reference point.

An "alternative element" refers to visual information associated with a viewing target instead of a group of elements (replacement target) that includes two or more shared elements. The alternative element may be visual information common to the replacement target, or visual information whose display mode is different from that of the replacement target. The display mode may be, for example, a display time, a display size, a display color, the presence or absence of additional visual information, or the presence or absence of motion. The number of overlapping elements may be indicated to a user by changing the display mode. Specifically, the display mode may be changed so that the greater the number of overlaps, the more flashy (rich) the display becomes. Flashy means, for example, a long display time, a large display size, a dark display color, a large amount of additional visual information, large motion, etc. Also, the number of overlapping elements may be indicated to a user by changing the display mode. Specifically, visual information indicating the number of overlaps may be added to the alternative element. The visual information indicating the number of overlaps may be, for example, an icon that displays a number corresponding to the number of overlaps, or an icon showing the number of overlaps.

[1-2. Supplementary notes]
The present invention includes at least the following embodiments [A] to [C].

[A] A shared information processing device comprising: an identification means for identifying a predetermined number of shared elements as an element group from among shared elements whose timing of displaying the viewing target on a terminal of a user viewing the viewing target is associated with the viewing target based on the timing; and a replacement means for replacing the element group identified by the identification means with alternative elements whose number is less than the predetermined number.

[B] A control program that causes a computer of an information processing device to realize a specification function that specifies a predetermined number of shared elements as an element group from among shared elements whose timing of displaying the viewing target on a terminal of a user viewing the viewing target is associated with the viewing target based on the timing, and a replacement function that replaces the element group specified by the specification function with a number of alternative elements that is less than the predetermined number. The control program may be non-temporarily recorded on a recording medium.

[C] A method for controlling an information processing device, comprising: an identification step of identifying a predetermined number of shared elements as an element group from among shared elements whose timing of displaying the viewing target on a terminal of a user viewing the viewing target is associated with the viewing target based on the timing; and a replacement step of replacing the element group identified in the identification step with alternative elements whose number is less than the predetermined number.

The following technical limitations may be added to the "information processing device" in [A] above. Similar technical limitations may also be added to the "control program" in [B] above and the "control method for the information processing device" in [C] above.

(1) The identification means identifies the group of elements based on additional attributes added to the shared elements, and the replacement means adds the alternative element that inherits the additional attributes added to at least one of the multiple shared elements included in the group of elements to the viewing target as the shared element. When identified "based on additional attributes," a predetermined number of shared elements having common additional attributes can be identified as the group of elements. Commonality means being the same or highly related. This allows the replaced alternative element to be further replaced with an alternative element.

(2) The identification means identifies the element group based on the additional attribute that can identify at least one of a user group to which a user who associated the shared element with the viewing target belongs and an element group to which the shared element associated with the viewing target belongs. In this way, multiple shared elements that tend to overlap in time are replaced with alternative elements.

A user who can associate a shared element with a viewing target can belong to at least one user group. Each user can voluntarily join one of the user groups. Also, each user may automatically join one of the user groups depending on attributes or circumstances. A shared element associated with a viewing target can belong to at least one element group. Each shared element is pre-assigned to at least one element group.

(3) The identification means identifies the element group based on the additional attribute that can identify both the user group and the element group. This allows multiple shared elements that tend to overlap in time to be replaced with alternative elements.

(4) When the type of the predetermined number of the shared elements included in the element group identified by the identification means is the same, the replacement means replaces the element group with the alternative element according to the number of overlaps of the type. This provides a visual effect to prevent an excessive increase in the load.

(5) The identification means identifies the group only when a predetermined activation condition is satisfied. This ensures that replacement with an alternative element is activated only in situations where a sudden increase in load is likely to occur.

An "activation condition" refers to a condition for determining whether or not to activate a replacement process (a process of replacing a predetermined number of shared elements with a number of alternative elements that is less than a predetermined number). The activation condition is preferably set as a condition regarding the number of elements in a group to which a shared element in a proximate relationship belongs. The condition regarding the number of elements is, for example, whether the number of elements reaches or exceeds a predetermined upper limit. The predetermined upper limit may be a fixed value or a variable value. The variable value is preferably set according to the processing capacity of the terminal. The processing capacity of the terminal may be a processing capacity regarding the display of information or a processing capacity regarding the reception of information. The processing capacity regarding the display of information is, for example, the clock frequency of the control processing device of the terminal. The processing capacity regarding the reception of information is, for example, the speed of the communication line of the terminal.

(6) The identification means identifies the predetermined number of shared elements included in a proximal element group as the element group, except when the number of elements in the proximal element group including the multiple shared elements whose timing is close does not satisfy the activation condition. This prevents replacement with an alternative element from being activated in situations where a sudden increase in load is unlikely to occur.

(7) The identification means identifies the predetermined number of shared elements included in the neighboring element group as the element group, except when the activation condition, which varies depending on the processing capacity of the terminal, is not satisfied. This makes it possible to flexibly determine whether the situation is such that a sudden increase in load is unlikely to occur.

2. Examples
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. This embodiment relates to a mechanism for displaying an image (image showing a reaction) showing the emotion of a user (viewer) on a display screen of a video (viewing target). In this embodiment, the shared information processing device is a server, and the information processing device (first device, second device) is a mobile terminal, a so-called smartphone. Note that the mobile terminal is not limited to a smartphone, but may be a tablet terminal, a notebook computer, or the like. The information processing device may be a desktop personal computer. Each program according to the embodiment is stored in the shared information processing device or the information processing device, but is not limited thereto, and may be stored in a storage medium such as an optical disk such as a CD-ROM (Compact Disk Read Only Memory) or a portable semiconductor memory such as a USB (Universal Serial Bus) flash memory.

[2-1. Features]
In this embodiment, operation information of an addition operation (first operation) performed by a first user (owner) who is watching a video (viewing target) displayed on a screen (first screen) of a first mobile terminal (first device) to add a first image (first element) to the video, and operation information of an addition operation (second operation) performed by a second user (another person) who is watching the video displayed on a screen (second screen) of a second mobile terminal (second device) to add a first image (second element) to the video are provided from each mobile terminal to a server (first instruction means, shared information processing device). In order to improve entertainment value, the server displays a combination image (third element) based on a combination of the first operation and the second operation according to the provided operation information on the first screen and the second screen, respectively.

In addition, in this embodiment, in order to suppress a sudden increase in load on a mobile terminal, two or more (a predetermined number) second images (shared elements) whose timing to be displayed on a mobile terminal of a user watching a video during playback of the video are associated with the video are identified as an image group (element group) based on the proximity relationship (timing) of the display timing and the commonality (additional attribute) of the subject and object, and a replacement process is performed in which one (a number less than a predetermined number) new second image (alternative element) that inherits the additional attribute while invalidating the second images included in the image group is added to the video. The subject refers to a user who added a second image to the video, and the subject having commonality refers to the subject belonging to the same user group. Each user belongs to at least one of the user groups. The object refers to the type of second image (element) added to the video, and the object having commonality refers to the object belonging to the same element group. Each element belongs to at least one of the element groups.

[2-2. Video distribution system]
A server according to the embodiment is included in a video distribution system and distributes videos to a mobile terminal.A mobile terminal according to the embodiment is included in the video distribution system and receives videos from the server.

[2-2-1. Overall configuration of video distribution system]
1 is a schematic diagram of a video distribution system 1 according to an embodiment of the present invention. The video distribution system 1 includes a communication line 2, a plurality of mobile terminals 3, and a server 4.

Communication line 2 forms a computer network and is, for example, a wide area communication line provided by an electric power company.

The mobile terminal 3 is, for example, an information processing terminal such as a smartphone, a tablet terminal, or a notebook computer. The mobile terminal 3 is equipped with a touch panel display 3a for displaying images, a speaker 3b for outputting sound, a microphone 3c for inputting sound, and a camera 3d for capturing an image of a subject. In order to display videos distributed from the server 4, as an example, dedicated application software (hereinafter referred to as a "video viewing application") is installed on the mobile terminal 3.

The server 4 is a shared information processing device that distributes content to multiple mobile terminals 3 via the communication line 2. One example of the content in the embodiment is a video. The video may be, for example, an animation or drama that has a story and the story progresses along the story, or a music video or a live music video, or a variety of sports videos, and is not particularly limited. Note that one example of the video in the embodiment is an animation. The content distributed from the server is also not limited to a video, and may be other services such as a collection of consecutive still images, as long as the user can view it.

Moreover, viewing here means that the user perceives the content with at least one of the senses of sight and hearing, and the content may be, in addition to the above, for example, a video without sound, or sound without images, etc.

As an example, a plurality of mobile terminals 3 to which videos are distributed are registered in the server 4 as a group 5 (hereinafter referred to as "group registration"). The server 4 distributes videos to the group-registered mobile terminals 3 simultaneously. That is, the group-registered mobile terminals 3 display the videos distributed from the server 4 at the same time, in other words, in synchronization, on their touch panel displays 3a. This allows each user of the group-registered mobile terminals 3 to watch the videos at the same time.

The video may be distributed at a predetermined date and time, or at any timing for each group 5. When the video is distributed at any timing, for example, the host mobile terminal 3 requests the server 4 to distribute the video, and the server 4 that receives the distribution request distributes the video to the mobile terminals 3 that make up the group 5.

In FIG. 1, group 5 is made up of four mobile terminals 3, but this is just an example, and the number of mobile terminals 3 is not limited as long as group 5 is made up of multiple mobile terminals 3. In addition, the types of mobile terminals 3 that make up group 5 are not limited, and a mixture of smartphones, tablet terminals, laptops, etc. may be used.

[2-2-2. Server configuration]
FIG. 2 is a block diagram showing an electrical configuration of the server 4 according to the embodiment.

The server 4 according to the embodiment includes a CPU (Central Processing Unit) 20, which is the main control unit that controls the operation of the entire server 4, a ROM (Read Only Memory) 22 in which various programs and various data are pre-stored, a RAM (Random Access Memory) 24 used as a work area when the CPU 20 executes various programs, and a HDD (Hard Disk Drive) 26 as a storage means for storing various programs and various data.

The HDD 26 stores information about the registered group 5, videos (video data) to be distributed to the mobile terminal 3, and various other information. Note that the storage means is not limited to the HDD 26, and may be other storage media such as semiconductor memory, such as flash memory.

The server 4 further includes an operation input unit 28, which is composed of a keyboard, a mouse, etc. and accepts input of various operations, a monitor 30, such as a liquid crystal display device, which displays various images, and an external interface 32, which is connected to other information processing devices, such as a mobile terminal 3, via the communication line 2 and transmits and receives various data to and from the other information processing devices.

The CPU 20, ROM 22, RAM 24, HDD 26, operation input unit 28, monitor 30, and external interface 32 are electrically connected to each other via a system bus 34. Therefore, the CPU 20 can access the ROM 22, RAM 24, and HDD 26, grasp the operation state of the operation input unit 28, display images on the monitor 30, and send and receive various data to and from other information processing devices, etc. via the external interface 32.

[2-2-3. Electrical configuration of the mobile terminal]
Fig. 3 is a functional block diagram showing the electrical configuration of the mobile terminal 3. The mobile terminal 3 includes a main control unit 40, a main memory unit 42, an auxiliary memory unit 44, a communication unit 46, and an operation button 48 in addition to the components shown in Fig. 1.

The main control unit 40 is, for example, a CPU, a microprocessor, a DSP (Digital Signal Processor), etc., and controls the overall operation of the mobile terminal 3.

The main memory unit 42 is composed of, for example, RAM or DRAM (Dynamic Random Access Memory), and is used as a work area when the main control unit 40 executes processes based on various programs.

The auxiliary storage unit 44 is, for example, a non-volatile memory such as a flash memory, and stores various data such as images, and programs used in the processing of the main control unit 40. The programs stored in the auxiliary storage unit 44 are, for example, an operating system (OS) for implementing the basic functions of the mobile terminal 3, drivers for controlling various hardware, and programs for implementing e-mail, web browsing, and various other functions. In addition, the auxiliary storage unit 44 pre-stores application software such as a video viewing app for playing videos distributed from the server 4 and displaying them on the touch panel display 3a.

The communication unit 46 is, for example, a network interface controller (NIC), and has a function of connecting to the communication line 2. Note that the communication unit 46 may have a function of connecting to a wireless local area network (LAN), a function of connecting to a wireless wide area network (WAN), and a function of enabling short-range wireless communication such as Bluetooth (registered trademark), infrared communication, etc., instead of or together with the NIC.

The operation buttons 48 are provided on the side of the mobile terminal 3 and include a power button for starting and stopping the mobile terminal 3 and a button for adjusting the volume of the sound output by the speaker 3b.

The main control unit 40, main memory unit 42, auxiliary memory unit 44, communication unit 46, touch panel display 3a, speaker 3b, microphone 3c, camera 3d, and operation buttons 48 are electrically connected to each other via a system bus 49. Therefore, the main control unit 40 can access the main memory unit 42 and auxiliary memory unit 44, display images on the touch panel display 3a, grasp the operation state of the touch panel display 3a and the operation buttons 48 by the user, input sound to the microphone 3c, output sound from the speaker 3b, control the camera 3d, and access various communication networks and other information processing devices via the communication unit 46.

[2-3. Onomatopoeia display]
The mobile terminal 3 according to the embodiment is capable of displaying images (onomatopoeia images, the details of which will be described later) in response to user operations while displaying videos distributed from the server 4 on the touch panel display 3a. In the following description, the function of displaying onomatopoeia images on the touch panel display 3a is referred to as an onomatopoeia display function, and the touch panel display 3a is also referred to as a screen 3a.

The onomatopoeia display function is, for example, one of the functions of the video viewing application according to the embodiment. Note that, for example, onomatopoeia display is not possible for all videos displayed on the screen 3a using the video viewing application, and videos for which onomatopoeia display is possible are determined in advance.

[2-3-1. screen structure]
4 to 6 show display modes of a screen 3a of a mobile terminal 3 on which a video viewing application according to an embodiment is running. The screen 3a includes a distribution video display area 50A, a first image display area 50B, and A second image display area 50C is defined.

The distributed video display area 50A is an area for displaying videos distributed from the server 4. The first image display area 50B is an area for displaying a first image 51 (see FIG. 5) in response to a user's operation. The second image display area 50C is an area for displaying a second image 52 (see FIG. 6) based on the first image 51 displayed on another mobile terminal 3 by another user's operation. As will be described in detail later, the second image 52 is a reduced image of the first image 51 displayed in the first image display area 50B of the other mobile terminal 3. In addition, the first image display area 50B and the second image display area 50C according to the embodiment are, as an example, provided in opposing positions sandwiching the distributed video display area 50A therebetween.

Here, the first image 51 is an image including an emotion image showing the emotion of the user watching the video. The emotion image according to the embodiment is, for example, an image in which an onomatopoeic word (also called mimetic sound word or mimetic word) showing the emotion of the user is illustrated, in other words, an onomatopoeia image. Examples of onomatopoeia images include "Awesome," "Awesome," and "Bravo." The illustration is, for example, a decorative drawing of letters showing the onomatopoeic word.

In addition to onomatopoeia images, the first image 51 may also include various other images, such as character images showing characters appearing in the video, dialogue images showing the lines of the characters, text entered by the user, etc.

The first image 51 may also differ depending on the scene of the video. For example, an onomatopoeia image or a character image that is likely to be selected by the user in response to a happy scene, a sad scene, a scene of confrontation between characters, etc. may be displayed.

The first images 51 that can be displayed may differ depending on the user. For example, the types of first images 51 that can be displayed may be increased by acquiring a specific item through a service such as a game related to the distributed video. Also, the types of first images 51 that can be displayed may be increased, for example, on the condition that a predetermined fee (for example, monetary value such as currency, virtual currency, or points) is paid.

In the first image display area 50B according to the embodiment, a plurality of display buttons 53A-53D are displayed as operation units that are operated by the user and each of which corresponds to a different first image 51. This allows the user to display the first image 51 in the first image display area 50B by operating the display buttons 53A-53D while watching a video. Note that, as an example, operating the display buttons 53A-53D means touching (pressing) the display buttons 53A-53D with an indicator. As an example, the indicator according to the embodiment is the user's finger. In addition, in the following description, when the display buttons 53A-53D are not to be distinguished from one another, the alphabetical notation at the end is omitted.

In the examples of Figures 4 to 6, four display buttons 53A to 53D are displayed in a row horizontally at the bottom of the first image display area 50B. Display button 53A corresponds to a stylized onomatopoeia image of "suge," display button 53B corresponds to a stylized onomatopoeia image of "yabai," and display button 53C corresponds to a stylized onomatopoeia image of "bravo." Furthermore, display button 53D corresponds to a character image. Note that the display buttons 53 are designed to enable the user to intuitively recognize the corresponding onomatopoeia image, character image, etc.

In this way, by displaying multiple display buttons 53 in the first image display area 50B, the user can easily display on the screen 3a the first image 51 that best matches the user's reaction (emotion) to the video. In addition, by including an onomatopoeia image that indicates the user's emotion in the first image 51, the user can display on the screen 3a the user's own emotion after watching the video.

In the examples of Figures 4 to 6, four types of display buttons 53 are displayed in the first image display area 50B, but this is just one example, and the number and types of display buttons 53 are not limited to this. In addition, the display buttons 53 may be appropriately switched so that different first images 51 can be displayed depending on the scene of the video, etc.

The arrangement of the display buttons 53 is also an example, and may be displayed together on the right or left side of the first image display area 50B, or may be displayed in two horizontal rows at the bottom of the first image display area 50B. Furthermore, for example, when the user swipes left or right on the bottom of the screen 3a, new display buttons 53 for displaying different types of first images 51 may be displayed.

[2-3-2. Display of the first image]
The display mode of the first image 51 will be described in detail with reference to Fig. 5. In the example of Fig. 5, the user operates the display button 53C, and an onomatopoeia image in which "Bravo" is stylized is displayed on the screen 3a.

The onomatopoeia display function according to the embodiment accepts a user operation for displaying an onomatopoeia image (first image 51) on screen 3a while a video is being displayed on screen 3a. That is, the user operates display button 53 while watching a video to display on screen 3a an onomatopoeia image that corresponds to the emotion felt while watching the video.

The onomatopoeia image according to the embodiment is displayed on the screen 3a displaying the video, changing according to the manner in which the user operates the display button 53. As an example, the onomatopoeia image according to the embodiment is displayed so as to extend from the first image display area 50B to the distribution video display area 50A according to the manner in which the user operates the display button 53.

In this way, the user can change the onomatopoeia image displayed on the screen 3a by operating the mobile terminal 3 while watching the video. Therefore, the user can easily display the emotions felt while watching the video on the screen while watching the video.

In the example of FIG. 5, the onomatopoeia image is displayed beyond the first image display area 50B and superimposed on the distributed video display area 50A. In this way, the onomatopoeia display function according to the embodiment superimposes the onomatopoeia image beyond the first image display area 50B onto the distributed video display area 50A in accordance with the user's operation mode, making it easier for the user to empathize with the video they are watching, and making the video more entertaining to watch.

In addition, in FIG. 5, the onomatopoeia image extends beyond the distribution video display area 50A and is also superimposed on the second image display area 50C. However, this is just one example, and the onomatopoeia image may not be superimposed on the second image display area 50C.

Next, we will explain in detail the onomatopoeia images that change depending on how the display button 53 is operated.

As an example, the display state of the first image 51 according to the embodiment changes depending on the duration of the user's operation of the display button 53. The duration of the operation of the display button 53 is, for example, the time during which the display button 53 is continuously pressed (hereinafter referred to as the "long press time").

When a user presses the display button 53, the onomatopoeia display function according to the embodiment starts displaying the onomatopoeia image starting from the vicinity of the position of the display button 53. Displaying the onomatopoeia image starting from the vicinity of the position of the display button 53 means, for example, that the onomatopoeia image is displayed so that the bottom edge of the onomatopoeia image is approximately aligned with the position of the display button 53.

The onomatopoeia image then extends (hereinafter referred to as "extended display") from the first image display area 50B toward the streaming video display area 50A depending on the length of time the display button 53 is pressed.

In the extended display, for example, the onomatopoeia image is displayed elongated or enlarged. Note that the extended display may be such that the onomatopoeia image is extended while changing the form, such as the number of characters or the type of characters. For example, if the onomatopoeia image is "bravo", it may be extended so that an "a" is added between "bra" and "bo" depending on the long press time, or it may be extended so that an "o" is added between "bravo" and "-" depending on the long press time.

In other words, if the user feels a stronger emotion toward the video being viewed, the user continues to press the display button 53 to display the onomatopoeia image that corresponds to this emotion. As a result, the onomatopoeia image is displayed extended from the first image display area 50B toward the distribution video display area 50A, and the onomatopoeia image is displayed on the screen 3a so that it stands out more.

In this way, the onomatopoeia image included in the first image 51 changes according to the manner in which the user operates, allowing the user to display their own reaction (emotions) to the content on the screen 3a in a more entertaining manner.

In addition, the first image 51 is displayed so as to extend from the first image display area 50B toward the distribution video display area 50A according to the manner of operation by the user, so that the user can display his or her own reaction (emotion) to the content on the screen 3a in a more entertaining manner.

When the user finishes operating the display button 53 (pressing and holding the display button 53), the display of the onomatopoeia image on the screen 3a is stopped. The display of the onomatopoeia image may be stopped, for example, by the displayed onomatopoeia image disappearing from the screen 3a instantly, or by the expanded onomatopoeia image shrinking toward the display button 53 and disappearing from the screen 3a.

In addition, in the example of FIG. 5, the periphery of the display button 53C that is being operated by the user to display the onomatopoeia image is highlighted. This allows the user to easily recognize the display button 53C that he or she is operating, and makes the onomatopoeia display more entertaining.

In addition, the character image or the like that is the first image 51 other than the onomatopoeia image in the embodiment is displayed on the screen 3a without changing, but this is not limited thereto, and the first image 51 other than the onomatopoeia image may also be displayed on the screen 3a displaying the video while changing according to the manner in which the user operates the display button 53.

[2-3-3. Display of second image]
The display mode of the second image 52 will be described in detail with reference to Fig. 6. As described above, the second image 52 is a reduced image of the first image 51 displayed on the other mobile terminal 3 by an operation of the other user.

The onomatopoeia display function according to the embodiment displays the second image 52 in the second image display area 50C with almost no time delay from the timing when the other user displays the first image 51 on the other mobile terminal 3. This allows the user to know the reactions of other users who are watching the same video while watching the video.

Furthermore, if the second image 52 is displayed superimposed on the distributed video display area 50A like the first image 51, the user may find the second image 52 to be a nuisance as it interferes with the viewing of the video. Therefore, the second image 52 is not displayed superimposed on the distributed video display area 50A beyond the second image display area 50C regardless of the operation mode of the other user. This allows the user to check the reactions of other users to the video they are watching without feeling annoyed.

As mentioned above, the first image 51 can also be superimposed on the streaming video display area 50A, but this is done at the user's own discretion, so the user does not feel bothered by the first image 51 being superimposed on the streaming video display area 50A.

In addition, since the screen 3a is configured so that the first image display area 50B and the second image display area 50C are different areas, the first image 51 displayed by the user and the second image 52 displayed by another user are not displayed together. This allows the user to easily distinguish between the first image 51 displayed by the user and the second image 52 displayed by another user.

In addition, other users according to the embodiment are, as an example, other users who are registered in the same group as the user. In other words, if a user is watching a video without group registration, the second image 52 will not be displayed in the second image display area 50C. However, this is not limited to this, and second images 52 generated by the operations of all other users watching the video may be displayed in the second image display area 50C, regardless of whether they are registered in a group.

In addition, the second image display area 50C according to the embodiment displays multiple second images 52 operated by multiple other users. In the second image display area 50C according to the embodiment, as an example, the position at which the second image 52 is displayed for each other user is not determined. That is, the second image display area 50C displays multiple second images 52 operated by multiple other users in a mixed manner at random positions. Furthermore, the second images 52 may be displayed while moving up, down, left, and right in the second image display area 50C. However, this is not limited to this, and the position at which the second image 52 is displayed for each other user may be determined in the second image display area 50C.

In addition, the second image 52 is displayed adjacent to an identification image (hereinafter referred to as a "user icon") 54 indicating the other user who caused the first image 51 designated as the second image 52 to be displayed on the user's own mobile terminal 3. This allows the user to recognize which user created the second image 52 displayed in the second image display area 50C. Note that in the example of FIG. 6, the user icon 54 is a circle with a single alphabet character displayed in the center, but this is only one example, and the user icon 54 may be an image set for each user or the user's initials, and the design of the user icon 54 is not limited.

In addition, the second image 52 shown in FIG. 6 displays the onomatopoeia image horizontally, but this is set according to the size (aspect ratio) of the second image display area 50C, and the second image 52 may display the onomatopoeia image vertically.

Although the second image 52 is not superimposed on the distribution video display area 50A, it may be displayed while changing (e.g., expanding) in the second image display area 50C in response to the operation mode by another user. Note that the second image 52 may be superimposed from the second image display area 50C onto the first image display area 50B as long as it is not superimposed on the distribution video display area 50A.

[2-3-4. Combined image display]
The mobile terminal 3 in this embodiment has a combination image display function that displays on the screen 3a a combination image corresponding to a combination of an operation by a user for displaying a first image 51 and an operation by another user for displaying a first image 51.

This allows the mobile terminal 3 to display the user's and other users' reactions to the video on the screen in a more entertaining way. Also, because images that are not displayed by an operation by one user are displayed in combination with operations by other users, the mobile terminal 3 can encourage more users to view the content.

The combination image is also displayed superimposed on the video being displayed on screen 3a. That is, the combination image is displayed superimposed not only on first image display area 50B, but also on distribution video display area 50A and second image display area 50C. This allows the user to easily recognize that the combination image is being displayed, and makes watching the video even more entertaining.

When a combination image is displayed on screen 3a, it is displayed so that the user can recognize that it is a combination image. For example, the user ID of another user may be displayed on screen 3a together with the combination image so that the user can recognize that another user performed a display operation at the same time.

Figure 7 is a timing chart showing when a combination image is displayed on screen 3a.

The combination image display function according to the embodiment displays a combination image on the screen 3a when the timing of the user's operation and the operation of another user is within a predetermined time (hereinafter referred to as the "combination period"). In other words, when multiple users perform a display operation of the first image 51 within the combination period, it is understood that this is the user's reaction to viewing the same scene. For this reason, the combination image display function displays a combination image based on the reactions of multiple users according to the scenes in the video, which further enhances the entertainment value for the user from viewing the video.

The start of the combination period is the timing when a user starts the display operation of the first image 51. That is, as shown in FIG. 7(A), if user B performs a display operation within the combination period that starts when user A starts the display operation, the combined image is displayed on the screen 3a of the mobile terminal 3 of users A and B.

The display period of the combined image is, as an example, until the user finishes the display operation. In the example of FIG. 7(A), the combined image is displayed on the screen 3a of the mobile terminal 3 of users A and B from the start of user B's display operation until it ends.

The example of FIG. 7(B) shows a case where a display operation by user C is performed within the combination period, but after the display operation by user A, which is the start timing of the combination period, is finished. In the example of FIG. 7(B), a combination image is displayed on the screens 3a of the mobile terminals 3 of users A, B, and C from the start of the display operation by user B until the end of the display operation by user C.

The display mode of the combination image shown in FIG. 7 is an example, and is not limited to this. For example, there may be no concept of a combination period, and the combination image may be displayed on the screen 3a at a timing when a display operation of a user overlaps with a display operation of another user. Also, a display time may be determined in advance for each combination image, and the display of the combination image may end when the display time has elapsed.

The combination image according to the embodiment is displayed on the screen 3a instead of the first image 51. In the example of FIG. 7(A), the first image 51 according to the display operation of user A is displayed on the screen 3a of the mobile terminal 3 of user A. However, when the display operation of user B is started, the display of the first image 51 ends and the combination image is displayed on the screen 3a of the mobile terminal 3 of user A. On the other hand, the first image 51 that user B himself intended to display is not displayed on the screen 3a of the mobile terminal 3 of user B, but the combination image is displayed. In this way, the combination image is displayed on the screen 3a instead of the first image 51, and an image different from the first image 51 that the user himself intended to display is displayed on the screen 3a, so that the entertainment value of the user watching the video is increased.

The combination image according to the embodiment is displayed on the screen 3a when the first image 51 selected by the user and the first image 51 selected by the other user are preset images. This makes it possible for a combination image that the user himself does not expect to be displayed in response to the image selection made by the user himself, thereby making watching the video more entertaining.

For example, the combination image is displayed on the screen 3a when the first image 51 selected by the user and the first image 51 selected by the other user are the same. In other words, when users watching a video select the same first image 51, it is highly likely that the users are experiencing similar emotions. For this reason, by displaying the combination image, the user can recognize that they have empathized with the other user by watching the video. More specifically, for example, when the first image 51 selected by the user and the first image 51 selected by the other user are the same first image 51, an enlarged image of the first image 51 is displayed on the screen 3a as the combination image. Note that this is not limited to this, and a special image that is not displayed by operating the display button 53 may be displayed as the combination image.

In addition, even if the first image 51 selected by the user and the first image 51 selected by another user are different first images 51, if the combination is a preset combination, a combination image may be displayed. In this case, for example, a special image that is not displayed by operating the display button 53 is displayed as the combination image.

In addition, in the combination image display function according to the embodiment, the second image based on the combination image displayed on the screen 3a of another user is not displayed in the second image display area 50C of the mobile terminal 3 of the user on whom the combination image is not displayed. However, this is not limited to the above, and the second image based on the combination image displayed on the screen 3a of another user may be displayed in the second image display area 50C of the mobile terminal 3 of the user on whom the combination image is not displayed.

[2-4. Replacement process]
The server 4 according to the embodiment has a replacement process function of identifying two second images 52 that are close in display timing from among the second images as an image group, and repeating a unit process of replacing the image group with one new second image 52 until no image group is identified. This reduces the number of second images 52 displayed during playback of a video on the mobile terminal 3, and can suppress a sudden increase in load on the mobile terminal 3.

Figure 8 is a timing chart showing when the second image is replaced.

As a premise, the timeline of the video is divided into multiple sections. As an example, the length of each section is the same as the length of the combined period. As an example, the replacement processing function according to the embodiment identifies, as an image group, two second images 52 that have commonality in subject and object from among a plurality of second images 52 each of which is included in the same section whose display period starts, and repeats a unit process of invalidating each of the two second images 52 included in the image group while associating one new second image 52 with the timeline of the video, until no image groups are identified.

Attributes (second image display information) that can identify the subject (user ID of the user who added the second image to the video and the user group to which the user belongs), object (type of the second image and the element group to which the type belongs), display period, number of times of synthesis, status, etc. are added to the second image 52. The type is determined by the object and the number of times of synthesis. The pattern of the second image 52 is determined by the type. It is preferable that the number of times of synthesis is reflected in the appearance of the pattern, such as by displaying a number indicating the number of times of synthesis. As shown in FIG. 8(A), the start of the display period of the second image A and the start of the display period of the second image B may both be included in the nth section (n is an appropriate natural number) on the timeline. It is assumed that the type of the second image A and the object of the second image B are the same. At this time, the replacement processing function identifies the second image A and the second image B as an image group, invalidates the status of the second image A and the second image B, and associates one new second image C with the timeline of the video.

The second image 52 after replacement (second image C) inherits the additional attributes of the second image 52 before replacement (second image A and second image B). The second image 52 after replacement, for example, inherits the subject and object as is. The number of times of synthesis of the second image 52 after replacement, for example, is the total number of times of synthesis of each second image 52 before replacement. The display period of the second image 52 after replacement, for example, is the period from the earlier start time to the later end time. As shown in FIG. 8(A), the display period of the second image C corresponds to the period from the start time of the second image A to the end time of the second image B.

The second image 52 before replacement is invalidated and disassociated with the video. On the other hand, the second image 52 after replacement can form an image group together with other second images 52 and be replaced with a new second image 52. As shown in FIG. 8B, the start of the display period of the second image C and the start of the display period of the second image D are both included in the nth section (n is an appropriate natural number) on the timeline. The user who added the second image C and the user who added the second image D belong to the same user group, and the type of the second image C and the type of the second image D are the same. At this time, the replacement processing function identifies the second image C and the second image D as an image group, invalidates the status of the second image C and the second image D, and associates one new second image E with the timeline of the video. The display period of the second image E is the period from the start of the second image C to the end of the second image D.

[2-5. Functional blocks]
Next, the combination image display and replacement process according to the embodiment will be described with reference to a functional block diagram. Note that the function related to the combination image display in the mobile terminal 3 is included in the onomatopoeia display function, and the function related to the combination image display and replacement process in the server 4 is included in the video distribution function.

[2-5-1. Functional block related to video distribution function of server]
FIG. 9 is a functional block diagram of the video distribution function of the server 4 according to the embodiment.

The CPU 20 of the server 4 includes a group registration unit 60, a video distribution unit 61, a first image display information reception control unit 62, a second image display information generation unit 63, a second image display information transmission control unit 64, a combination image information generation unit 65, a combination image information transmission control unit 66, and a replacement processing unit 67. Each function of the CPU 20 is realized by a program stored in the HDD 26.

The group registration unit 60 accepts registration of a group 5 consisting of multiple mobile terminals 3. As an example, the group 5 is registered by registering a user ID.

The video distribution unit 61 distributes videos to the mobile terminal 3 in response to a request from the mobile terminal 3.

The first image display information reception control unit 62 accepts the first image display information received from the mobile terminal 3 via the external interface 32, associates the first image display information with the corresponding video, and stores it in the HDD 26.

The first image display information is information about the first image 51 displayed on the screen 3a of the mobile terminal 3, and is transmitted from the mobile terminal 3 to the server 4 at the timing when the first image 51 is displayed on the mobile terminal 3. The first image display information is, for example, the type of the displayed first image 51, a user ID indicating the user who caused the first image 51 to be displayed, the operation mode performed by the user to display the first image 51, the elapsed time from the start of the video when the first image 51 was displayed, etc.

The second image display information generating unit 63 generates second image display information based on the received first image display information, associates the second image display information with the corresponding video, and stores it in the HDD 26.

The second image display information is information about the second image 52 to be displayed in the second image display area 50C, and includes the type of the second image 52, the user ID of the user who displayed the first image 51 that is the source of the second image 52 on his/her mobile terminal 3, the display period, the number of times of synthesis, the status, etc.

The second image display information transmission control unit 64 controls the external interface 32 to transmit the second image display information to the mobile terminal 3. The destination of the second image display information is another mobile terminal 3 different from the mobile terminal 3 that transmitted the first image display information.

The combination image information generating unit 65 performs a combination determination based on the received first image display information. The combination image information generating unit 65 then generates combination image information indicating a combination image according to the result of the combination determination, and stores the combination image information in the HDD 26.

The combination image information is information about the combination image to be displayed on the screen 3a, and includes the type of combination image, the user IDs of multiple users who performed display operations at the same time, etc.

The combination determination is also performed for each piece of first image display information, and it is determined whether the display operation of the first image 51 shown by the first image display information is within a combination period based on the start timing of the display operation of the first image 51 by another user.

The combination image information transmission control unit 66 controls the external interface 32 to transmit the combination image information to the mobile terminal 3. The mobile terminals 3 to which the combination image information is transmitted are the multiple mobile terminals 3 corresponding to the user IDs included in the combination image information.

The replacement processing unit 67 executes the replacement process.

[2-5-2. Functional blocks related to onomatopoeia display on mobile devices]
FIG. 10 is a functional block diagram relating to onomatopoeia display of the mobile terminal 3 according to the embodiment.

The main control unit 40 of the mobile terminal 3 includes an image display control unit 70, a display area setting unit 71, a first image display operation receiving unit 72, an operation state determination unit 73, a first image display information generation unit 74, a first image display information transmission control unit 75, a second image display information reception control unit 76, and a combination image information reception control unit 77. Each function of the main control unit 40 is realized by a program stored in the auxiliary storage unit 44.

The image display control unit 70 controls the touch panel display 3a to display an image based on image data. The image display control unit 70 according to the embodiment displays a video distributed from the server 4 in the distributed video display area 50A, displays a first image 51 in the first image display area 50B in response to a user's operation, and displays a second image 52 based on the first image 51 displayed on another mobile terminal 3 by another user in the second image display area 50C. Furthermore, the image display control unit 70 according to the embodiment displays the first image 51 beyond the first image display area 50B and superimposed on the distributed video display area 50A in response to the user's operation mode.

The display area setting unit 71 sets the positions and sizes (length and width) of the distributed video display area 50A, the first image display area 50B, and the second image display area 50C as the display mode of the screen 3a when a video is distributed from the server 4. Note that these positions and sizes are predetermined, but may be set arbitrarily by the user.

The first image display operation receiving unit 72 receives the operation of the display button 53 by the user while the screen 3a is displaying a video. That is, the first image display operation receiving unit 72 recognizes the display button 53 operated by the user among the multiple display buttons 53, and displays the first image 51 corresponding to the display button 53 on the screen 3a via the image display control unit 70. The first image display operation receiving unit 72 displays the first image 51 on the screen 3a such that the display mode of the first image 51 changes depending on the determination result by the operation state determination unit 73.

The operation state determination unit 73 determines the operation state of the display button 53 by the user. The operation state here refers to an operation mode including, for example, the time the display button 53 is pressed and held.

The first image display information generating unit 74 generates first image display information according to the first image 51 displayed on the screen 3a by the user's operation.

The first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

The second image display information reception control unit 76 outputs the second image display information received from the server 4 via the communication unit 46 to the image display control unit 70.

The combination image information reception control unit 77 outputs the combination image information received from the server 4 via the communication unit 46 to the image display control unit 70. As a result, the image display control unit 70 displays the combination image on the screen 3a displaying the video. Note that the image display control unit 70 according to the embodiment displays the combination image on the screen 3a instead of the first image 51 in response to the user's operation of the display button 53.

[2-6. Processing flow]
Next, the flow of an onomatopoeia display process in the mobile terminal 3 and the server 4 according to the embodiment will be described with reference to a flowchart.

[2-6-1. Flow for displaying onomatopoeia on a mobile device]
11 is a flowchart showing the flow of the onomatopoeia display process executed by the main control unit 40 of the mobile terminal 3. A program for executing the onomatopoeia display process is pre-stored in a predetermined area of the auxiliary storage unit 44. As an example, the onomatopoeia display process is started when the mobile terminal 3 starts playing a video distributed from the server 4, and ends when the playback of the video ends.

In step S100, occurrence of a trigger is detected. In this embodiment, the trigger is either a first trigger (acceptance of operation of the display button 53 by the user) or a second trigger (reception of second image display information). The mobile terminal 3 in this embodiment updates the display content of the screen every time the occurrence of a trigger is detected. The first trigger and the second trigger may occur in parallel. Therefore, while processing is being executed in response to the occurrence of one trigger, processing is sometimes executed in response to the occurrence of the other trigger. Also, multiple second triggers may occur in parallel. Therefore, while processing is being executed in response to the occurrence of one second trigger, processing is sometimes executed in response to the occurrence of another second trigger.

In step S102, if the detected trigger is the first trigger, the process proceeds to step S106, and if the detected trigger is the second trigger, the process proceeds to step S104.

In step S104, the image display control unit 70 displays the second image 52 indicated by the second image display information in the second image display area 50C, and the process ends.

In step S106, the image display control unit 70 displays the first image 51 corresponding to the display button 53 operated by the user in the first image display area 50B. The first image 51 may be stored in the mobile terminal 3 in advance, or may be downloaded from the server 4 each time the display button 53 is operated by the user.

In step S108, the first image display information generating unit 74 generates first image display information corresponding to the first image 51 displayed in the first image display area 50B.

In step S110, the first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

In step S112, the operation state determination unit 73 determines whether or not the user has finished operating the display button 53. If the determination is affirmative, the process proceeds to step S120, and if the determination is negative, the process proceeds to step S114.

In step S114, the combination image information reception control unit 77 determines whether or not combination image information has been received via the communication unit 46, and if the determination is positive, the process proceeds to step S118, and if the determination is negative, the process proceeds to step S116.

In step S116, the image display control unit 70 displays the first image 51 (onomatopoeia image) in an elongated manner until the user finishes operating the display button 53, and then the process returns to step S112.

In step S118, the image display control unit 70 displays the combination image indicated by the combination image information on the screen 3a, and the process returns to step S112.

In step S120, the image display control unit 70 ends the display of the first image 51 and ends the process.

In the flowchart shown in FIG. 11, the first image display information is generated together with the display of the first image 51, and the first image display information is transmitted to the server 4. However, this is not limiting, and the first image display information may be transmitted to the server 4 after the display of the first image 51 ends. Furthermore, the first image display information transmission control unit 75 may transmit to the server 4 first image display information indicating the start of the display of the first image 51 at the timing when the display of the first image 51 starts, and transmit to the server 4 first image display information indicating the end of the display of the second image 52 at the timing when the display of the first image 51 ends. This allows the server 4 to recognize that the first image 51 (onomatopoeia image) is displayed in an elongated manner during the reception interval between the first image display information indicating the start of the display of the first image 51 and the first image display information indicating the end of the display.

[2-6-2. Flow for displaying onomatopoeia on the server]
12 is a flowchart showing the flow of the onomatopoeia display process executed by the CPU 20 of the server 4. A program for executing the onomatopoeia display process is pre-stored in a predetermined area of the HDD 26. As an example, the onomatopoeia display process is started when the server 4 is distributing a video to the mobile terminal 3, and ends when the video distribution ends.

In step S200, the first image display information reception control unit 62 determines whether or not the first image display information has been received from the mobile terminal 3. If the determination is positive, the process proceeds to step S202, and if the determination is negative, the process enters a waiting state.

In step S202, the first image display information reception control unit 62 stores the received first image display information in the HDD 26.

In step S204, the combination image information generating unit 65 performs a combination judgment, and if the judgment is positive, the process proceeds to step S208, and if the judgment is negative, the process proceeds to step S206. That is, if the display operation of the first image 51 to be combined is performed by multiple users within the combination period, the judgment is positive and the process proceeds to step S208. On the other hand, if the display operation of the first image 51 to be combined is not performed by multiple users within the combination period, the judgment is negative and the process proceeds to step S206.

In step S206, the second image display information generating unit 63 generates second image display information based on the received first image display information and stores it in the HDD 26.

In step S208, the combination image information generating unit 65 generates combination image information that indicates a combination image according to the display operation of the first images 51 to be combined.

In step S210, the combination image information transmission control unit 66 controls the external interface 32 to transmit the combination image information, and the process returns to step S200. The destination of the combination image information is the multiple mobile terminals 3 that have transmitted the first image display information to be combined, that is, the mobile terminals 3 that correspond to the multiple user IDs included in the combination image information.

Figure 13 is a flowchart showing the flow of the onomatopoeia display process executed by the CPU 20 of the server 4. The flow shown in Figure 13 targets the second image display information whose display period starts within the section in question each time the end of the section in the video timeline is reached.

In step S300, the replacement processing unit 67 makes a judgment, and if the judgment is positive, the process proceeds to step S302, and if the judgment is negative, the process proceeds to step S304. That is, if there is second image display information in which the subject and object have commonality, the judgment is positive and the process proceeds to step S302. On the other hand, if there is no second image display information in which the subject and object have commonality, the judgment is negative and the process proceeds to step S304.

In step S302, the replacement processing unit 67 performs replacement processing. In a unit process of the replacement processing, the replacement processing unit 67 identifies two pieces of second image display information that have commonality between the subject and the object as an image group, invalidates the status of each of the two pieces of second image display information included in the image group, and generates one new piece of second image display information. In step S302, this unit process is repeated until there are no more pieces of second image display information that have commonality between the subject and the object.

In step S304, the second image display information transmission control unit 64 controls the external interface 32 to transmit all second image display information included in the target section to a mobile terminal 3 different from the mobile terminal 3 that transmitted the first image display information. Therefore, the display period on the mobile terminal 3 of the second image 52 based on the second image display information will be delayed from the display period associated with the video.

As described above, in the mobile terminal 3 according to the embodiment, operation information of an additional operation performed on the first mobile terminal 3 by a first user watching a video displayed on the screen 3a of the first mobile terminal 3 to add a first image 51 to the video, and operation information of an additional operation performed on the second mobile terminal 3 by a second user watching the video displayed on the screen 3a of the second mobile terminal 3 to add a first image 51 to the video are provided from each mobile terminal 3 to the server 4. In order to improve entertainment value, the server 4 displays a combination image based on a combination of an additional operation by the first user and an additional operation by the second user on the screen 3a of the first mobile terminal 3 and the screen 3a of the second mobile terminal 3, respectively. As a result, an image that is not displayed by an operation by one user is displayed by a combination with an operation by another user, so that the mobile terminal 3 can encourage more users to watch the video.

The server 4 according to the embodiment replaces, from among the second images 52 associated with a video whose display timing on the mobile terminal 3 of a user watching the video during playback of the video, two or more second images 52 that have close display timing and commonality in subject and object with one new second image 52. This reduces the number of second images 52 displayed on the mobile terminal 3, allowing the server 4 to suppress sudden increases in load on the mobile terminal 3.

3. Modifications
The technical scope of the present invention is not limited to the scope described in the above embodiments. Various modifications or improvements can be made to the above embodiments without departing from the gist of the invention, and such modifications or improvements are also included in the technical scope of the present invention.

The various processing flows described in the above embodiments are also examples, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged without departing from the spirit of the present invention.

[3-1. Modification 1 (Sharing images without going through a server)]
In the above embodiment, a form has been described in which the server 4 receives first image display information from the mobile terminal 3, transmits second image display information generated based on the first image display information to another mobile terminal 3, and the other mobile terminal 3 displays the second image 52 indicated by the second image display information in the second image display area 50C, but the present invention is not limited to this.

For example, a plurality of mobile terminals 3 may transmit and receive first image display information to each other without going through the server 4, so that the mobile terminals 3 display the second image 52 in the second image display area 50C. That is, when the mobile terminal 3 receives first image display information from another mobile terminal 3, it reads information related to the first image 51 displayed on the other mobile terminal 3 from the first image display information, and displays the second image 52 based on that information in the second image display area 50C.

[3-2. Modification 2 (Watching a video alone)]
The above embodiment is described as an example of a case where an image expressing an emotion toward a video is shared among multiple users who are simultaneously watching the video. In contrast, an image expressing an emotion toward a video of a user who has watched the video first can be shared with a user who watches the video later. This makes it possible to display a combination image according to a combination of an additional operation by a first user who has watched the video first and an additional operation by a second user who watches the video later.

In the following, we consider a case where a first user using a first mobile terminal 3 (first device) watches a video (viewing target) after a second user using a second mobile terminal 3 (second device) watches the same video (Figure 13 (A)). In this case, the additional operation by the first user and the additional operation by the second user may not be in close proximity to each other in real time. However, the two are in close proximity to each other on the progression axis (timeline) of the video (Figure 13 (B)). The first user and the second user may be the same person and another person, or may be the same person in the past and the same person in the present.

In the second modification, operation information of an addition operation (second operation) performed on the second mobile terminal 3a by a second user who is watching a video displayed on the screen 3a (second screen) of the second mobile terminal 3 to add a first image (second element) to the video is transmitted from the second mobile terminal 3a to the server 4.
The server 4 transmits in advance to the first mobile terminal 3 used by the first user who is to watch the same video, second image display information (second instruction) for displaying the second image (second element) based on an additional operation by the second user at a timing corresponding to the additional operation.

In order to improve entertainment value, when the first mobile terminal 3 receives an additional operation (first operation) performed on the first mobile terminal 3a by the first user watching a video displayed on the screen 3a (first screen) of the first mobile terminal 3 to add a first image (first element) to the video, the first mobile terminal 3 displays a combination image (third element) based on a combination of the received additional operation (first operation) and an additional operation (second operation) related to the second image display information on the screen 3a of the first mobile terminal 3. Note that in the second modification example, since the timing of the second operation performed on the second mobile terminal 3 is known in advance, it is also possible to display a combination image (third element) based on a combination of the received additional operation (first operation) and an additional operation (second operation) related to the second image display information on the screen 3a of the first mobile terminal 3 without delay after the first operation is received on the first mobile terminal 3.

[3-3. Modification 3 (First image not displayed when combined image is displayed)]
In the above embodiment, the first image is also displayed when the combined image is displayed. However, the first image may not be displayed when the combined image is displayed.

Specifically, the image display control unit 70 (second control means) of the mobile terminal 3 waits to receive combination image information until a predetermined period (combination period) has elapsed since the first image display operation receiving unit 72 (first receiving means) received the additional operation. If the image display control unit 70 receives combination image information within the predetermined time, it displays a combination image (third element) based on the received combination image information, and if the image display control unit 70 does not receive combination image information within the predetermined period, it displays the first image (first element) based on the received additional operation.

[3-4. Modification 4 (displaying the second image even when a combined image is displayed)]
In the above embodiment, when the combination image is displayed, the second image is not displayed. In contrast, the second image may be displayed even when the combination image is displayed. Specifically, when the server 4 receives the first image information, it is preferable to generate second image display information and transmit it to the mobile terminal 3 regardless of whether or not a display operation to combine has been performed.

[3-5. Modification 5 (Limiting the Activation of Replacement Processing)]
In the above embodiment, the unit process is repeated for each predetermined period (combination period), and multiple pieces of second-image display information to be replaced are replaced with one piece of second-image display information. In contrast, in a situation where a sudden increase in load is unlikely to occur on the mobile terminal 3, the replacement process may be omitted.

Specifically, when the number of elements of a proximate element group including multiple second image display information (shared elements) whose display timings are close to each other does not satisfy the activation condition, the replacement processing unit 67 (identification means) of the server 4 omits a unit process (replacement process) for identifying two (predetermined number) pieces of second image display information included in the proximate element group as an image group (element group). The activation condition is, for example, that the number of elements is equal to or greater than a predetermined threshold. The activation condition may be flexibly changed according to the processing capacity of the mobile terminal 3. For example, it is preferable to set a predetermined threshold according to the processing capacity of the terminal with the lower capacity among the multiple mobile terminals 3 belonging to group 5. The processing capacity is, for example, the display processing capacity (DSP clock frequency) or reception processing capacity (communication line speed) of the mobile terminal 3.

1 Video distribution system 2 Communication line 3 Mobile terminal (examples of "first device", "second device", and "information processing device")
3a Screen (example of "first screen" and "second screen")
4. Server (an example of the “first instruction means”, the “second instruction means”, and the “shared information processing device”)
5. Group (an example of an "information processing system")
50A: Distribution video display area 50B: First image display area 50C: Second image display area 51: First image 52: Second image (an example of a "shared element" or "alternative element")
53 Display button 60 Group registration unit 61 Video distribution unit 62 First image reception information reception control unit 63 Second image display information generation unit 64 Second image display information transmission control unit 65 Combined image information generation unit 66 Combined image information transmission control unit 67 Replacement processing unit (an example of the “identification means” and the “replacement means”)
70 Image display control unit (an example of the “first control means”, “second control means”, “third control means”, or “display control means”)
71 Display area setting unit 72 First image display operation receiving unit (an example of the “first receiving means” and the “second receiving means”)
73 Operation state determination unit 74 First image display information generation unit 75 First image display information transmission control unit (an example of the “first providing means” and the “second providing means”)
76 Second image display information reception control unit (an example of the “third reception means”)
77 Combined image information reception control unit

Claims (3)

a processor, the processor comprising:
Accepts registrations from multiple users to the same user group,
Displaying a common viewing target on terminals of a plurality of users belonging to the same user group;
accepting additional operations from a plurality of users using terminals displaying the common viewing target;
associate, based on an operation timing at which each of the additional operations is received, a display timing at which each of the elements determined based on each of the additional operations is displayed with the viewing target;
when the number of the elements of the same type included in a proximate element group including the plurality of elements whose display timings are close to each other is equal to or greater than a predetermined threshold value set according to the terminals of the users belonging to the user group, instead of the elements of the same type included in the proximate element group, a number of alternative elements less than the number of the elements of the same type included in the proximate element group are displayed on the terminals of the plurality of users currently displaying the viewing target;
Information processing device. The processor accepts registrations from multiple users to the same user group,
A processor displays a common viewing target on terminals of a plurality of users belonging to the same user group;
A processor receives additional operations by a plurality of users using terminals displaying the common viewing target,
a processor associates, with the viewing target, a display timing for displaying each of the elements determined based on each of the additional operations based on an operation timing at which each of the additional operations is received;
when the number of the elements of the same type included in a proximate element group including the plurality of elements whose display timings are close to each other is equal to or greater than a predetermined threshold value set according to the terminals of the users belonging to the user group, a processor displays, instead of the elements of the same type included in the proximate element group, a number of alternative elements that is less than the number of the elements of the same type included in the proximate element group, on the terminals of the plurality of users currently displaying the viewing target;
Information processing methods. The processor:
A function to accept registrations from multiple users to the same user group,
A function of displaying a common viewing target on terminals of multiple users belonging to the same user group;
A function of accepting additional operations by a plurality of users using a terminal displaying the common viewing target;
a function of associating, with the viewing target, a display timing for displaying each of the elements determined based on each of the additional operations, based on an operation timing at which each of the additional operations is received;
a function of displaying, in place of the elements of the same type included in the proximity element group, a number of alternative elements less than the number of the elements of the same type included in the proximity element group on the terminals of the users who belong to the user group when the number of the elements of the same type included in the proximity element group, the number of the alternative elements being less than the number of the elements of the same type included in the proximity element group, on the terminals of the users who are displaying the viewing target, when the number of the elements of the same type included in the proximity element group, the number of the alternative elements being less than the number of the elements of the same type included in the proximity element group, the number of the alternative elements being less than the number of the elements of the same type included in the proximity element group, in the case where the number of the elements of the same type included in the proximity element group, the number of the alternative elements being less than ... on the terminals of the users who belong to the user group, the viewing target being displayed;
A program that executes the following.

Images