JP7430353B2 - Video processing method, server device and computer program - Google Patents

Description

The technology disclosed in the present application relates to a video processing method for a video viewed by a viewing user, a server device and a computer program for controlling the video.

2. Description of the Related Art Video distribution services that distribute videos to terminal devices via networks have been known. In this type of video distribution service, the avatar (avatar object) of the distribution user who distributes the video is displayed, and the viewing user who watches the video and the distribution user can communicate through messages, chat, etc. A conducive environment is provided.

Furthermore, as a communication tool between viewing users and distribution users, Patent Document 1 electronically realizes the act (gifting) of providing an item acquired by a viewing user as a present to an avatar object (distribution user) in a video. It is disclosed that the

In addition, in video distribution services such as VirtualCast (registered trademark), the avatar object grasps the gift given by the viewing user to the avatar object (distribution user) on the video based on the distribution user's movements. (Non-Patent Document 1).

Note that the above-mentioned Patent Document 1 and Non-Patent Document 1 are incorporated in their entirety into this specification by reference.

Japanese Patent Application Publication No. 2012-120098

"VirtualCast", [online], [searched on September 10, 2019], Internet (URL: https://virtualcast.jp/)

The gifting disclosed in Patent Document 1 and Non-Patent Document 1 fosters a sense of intimacy and involvement of the viewing user with respect to the avatar object through direct communication between the viewing user and the distribution user (avatar). be. However, the technologies disclosed in Patent Document 1 and Non-Patent Document 1 only provide an environment in which a gift corresponding to gifting is displayed on a video, or an environment in which an avatar object can grasp the gift on a video. Therefore, gifting opportunities for viewing users are still limited.

Furthermore, gifting refers to a mode in which a viewing user purchases a gift by paying a fee to a service provider that provides a video distribution service, and then gifts the gift to a distribution user (avatar object). Therefore, from the operational perspective of video distribution services, it can be said that increasing gifting opportunities for viewing users is a challenge for service providers.

Therefore, some embodiments disclosed in the present application provide a video processing method, a server device, and a computer program that increase gifting opportunities by viewing users.

A video processing method according to one aspect is a video processing method executed by one or more processors that execute computer-readable instructions, the video processing method including an animation of an avatar object of the distribution user based on the distribution user's actions. a generation step of generating a first gift object, and a request signal generated based on the operation of the viewing user who views the video and requesting that the first gift object be given to the avatar object on the terminal device of the viewing user. a first display step of displaying the first gift object given to the avatar object on the video based on the request signal; and a first display step of displaying the first gift object given to the avatar object based on the request signal on the video; If it is determined that the avatar object has performed a predetermined action on one or more of the first gift objects displayed on the video based on the action of the avatar object that has performed the predetermined action, The method includes a second display step of displaying a second gift object different from the first gift object on the moving image.

A server device according to one aspect includes one or more processors, the processor generates a video including an animation of an avatar object of the distribution user based on the distribution user's actions, and the operation of a viewing user who views the video. A request signal generated based on the request signal requesting to give a first gift object to the avatar object is received from the terminal device of the viewing user via a communication line, and based on the request signal, The first gift object given to the avatar object is displayed on the video, and based on the distribution user's action, one or more of the first gift objects displayed on the video are On the other hand, if the terminal device of the distribution server determines that the avatar object has performed the predetermined action, a second gift object different from the first gift object that has performed the predetermined action is displayed on the video. This is what is displayed.

A server device according to one aspect includes one or more processors, and the processor generates the video by using information for generating a video including an animation of an avatar object of the distribution user based on the behavior of the distribution user. is transmitted to the distribution user's terminal device or the viewing user's terminal device, is generated based on the operation of the viewing user who views the video, and requests the avatar object to be given a first gift object. A request signal is received from the viewing user's terminal device via a communication line, and information regarding the first gift object given to the avatar object based on the request signal is transmitted to the first gift object. to the distribution user's terminal device or the viewing user's terminal device to display on the video, and one or more of the first gift objects displayed on the video based on the distribution user's operation. When the terminal device of the distribution server determines that the avatar object has executed a predetermined action, information regarding a second gift object different from the first gift object that has performed the predetermined action. , the second gift object is transmitted to the distribution user's terminal device or the viewing user's terminal device that displays the second gift object on the video.

A computer program according to one aspect is configured to generate a moving image including an animation of an avatar object of the distribution user based on the operation of the distribution user by being executed by one or more processors, and to respond to operations of a viewing user who views the video. a request signal generated based on the request signal requesting to give the first gift object to the avatar object from the server device or the terminal device of the viewing user via a communication line, and based on the request signal. The first gift object given to the avatar object is displayed on the video, and one or more of the first gifts displayed on the video are displayed based on the action of the distribution user. When it is determined that the avatar object has performed a predetermined action on the object, a second gift object different from the first gift object for which the predetermined action has been performed is displayed on the video. The processor functions as follows.

A computer program according to one aspect is executed by one or more processors to transmit information for generating a video including an animation of an avatar object of the distribution user based on the distribution user's actions to a server that generates the video. The server transmits a request signal, which is generated based on the operation of the viewing user who views the video, to the device or the terminal device of the viewing user, requesting that the first gift object be given to the avatar object. information regarding the first gift object given to the avatar object based on the request signal received from the device or the terminal device of the viewing user via a communication line; The avatar is transmitted to the server device to be displayed on the video or the terminal device of the viewing user, and based on the operation of the distribution user, the avatar is sent to the one or more first gift objects displayed on the video. When it is determined that the object has executed the predetermined action, information regarding a second gift object different from the first gift object that has performed the predetermined action is displayed on the video. The processor functions to transmit the information to the server device for display or the viewing user's terminal device.

A computer program according to one aspect is executed by one or more processors to generate a video including an animation of an avatar object of the distribution user based on the distribution user's actions, and transmits information regarding the video to a server device or A request signal received from the terminal device of the distribution user, generated based on the operation of the viewing user who views the video, and requesting to give the first gift object to the avatar object, is transmitted through the communication line. In order to display the first gift object given to the avatar object based on the request signal on the video, information regarding the first gift object is sent to the server device or the The distribution information is received from a terminal device of a distribution user, and based on the distribution user's operation, the avatar object performs a predetermined action on one or more of the first gift objects displayed on the video. information regarding the second gift object in order to display on the video a second gift object different from the first gift object for which the predetermined action has been performed when determined by the terminal device of the server; The processor functions to receive the information from the server device or the distribution user's terminal device.

A computer program according to one aspect is executed by one or more processors to receive a video including an animation of an avatar object of the distribution user based on an action of the distribution user from a server device or a terminal device of the distribution user. , a request signal generated based on an operation of a viewing user viewing the video and requesting that a first gift object be given to the avatar object is transmitted via a communication line, and the request signal is receive the video displaying the first gift object given to the avatar object from the server device or the distribution user's terminal device based on the distribution user's action; When the terminal device of the distribution server determines that the avatar object has performed a predetermined action on one or more of the first gift objects displayed above, the first gift object that has performed the predetermined action The processor is operated to receive the moving image displaying a second gift object different from the first gift object from the server device or the distribution user's terminal device.

FIG. 1 is a block diagram showing an example of the configuration of a video distribution system according to an embodiment. FIG. 2 is a block diagram schematically showing an example of the hardware configuration of the terminal device (server device) shown in FIG. FIG. 3 is a block diagram schematically showing an example of the functions of the terminal device shown in FIG. 1. FIG. 4 is a diagram illustrating an example in which a first gift object is displayed on a video including an animation of a distribution user's avatar object. FIG. 5 is a diagram illustrating a case where a first gift object is displayed on a video including an animation of a distribution user's avatar object, and two first collision areas overlap. FIG. 6 is a diagram illustrating an example in which a second gift object is displayed on a video including an animation of a distribution user's avatar object. FIG. 7 is a diagram illustrating an example in which a second gift object is displayed on a video including an animation of a distribution user's avatar object. FIG. 8 is a diagram illustrating an example of an effect object displayed on a video including an animation of a distribution user's avatar object. FIG. 9 is a block diagram schematically showing an example of the functions of the server device shown in FIG. 1. FIG. 10 is a flow diagram showing an example of operations performed in the video distribution system shown in FIG. 1.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. Note that common components in the drawings are given the same reference numerals. Also, it should be noted that components depicted in one drawing may be omitted from another drawing for convenience of explanation. Furthermore, it should be noted that the attached drawings are not necessarily drawn to scale. Furthermore, the term application may also be referred to as software or a program, as long as it is a set of instructions to a computer that can be combined to achieve a certain result.

1. Configuration of Video Distribution System The video distribution system 1 disclosed in the present application, to put it simply, allows a distribution user who performs distribution to use the terminal device to transmit data and/or data regarding the operation of the distribution user and/or data issued by the distribution user. It is possible to deliver a video containing an animation of an avatar object generated based on voice data related to utterances and singing to the terminal device of the viewing user who views the video via a communication line. .

FIG. 1 is a block diagram showing an example of the configuration of a video distribution system 1 according to an embodiment. As shown in FIG. 1, the video distribution system 1 includes a plurality of terminal devices 10 connected to a communication line (not shown) and one or more server devices 20 connected to the communication line. I can do it. Each terminal device 10 is connected to one or more server devices 20 via a communication line.

Note that the communication line may include, but is not limited to, a mobile phone network, a wireless LAN, a fixed telephone network, the Internet, an intranet, and/or Ethernet (registered trademark).

1-1. Terminal device 10
As shown in FIG. 1, the plurality of terminal devices 10 include one or more distribution user's terminal device 10 (here, terminal device 10A), one or more viewing user's terminal device 10 (here, terminal device 10A), and one or more viewing user's terminal device 10 (here, terminal device 10B to 10D). By having a common configuration, each terminal device 10 can be used as either a terminal device for distributing videos (terminal device 10 of a distribution user) or a terminal device for viewing videos (terminal device 10 of a viewing user). It is also possible.

When each terminal device 10 operates as a distribution user's terminal device 10, by executing the installed video distribution application, each terminal device 10 can collect data related to the distribution user's actions and/or utterances uttered by the distribution user. Acquire audio data related to singing, generate a video including an animation of a virtual character (avatar object) changed according to the acquired data, and send the video to the server device 20 (distribution server device 20C) via a communication line. ) can be sent to. The video is distributed via a communication line by the server device 20 (distribution server device 20C) to the terminal device 10 (viewing user's terminal device 10) for receiving the video, which executes the installed video viewing application. be done.

Furthermore, when each terminal device 10 operates as a distribution user's terminal device 10, the server device 20 sends a gift object ( one or more request signals requesting to give (gifting) a first gift object (if a plurality of request signals are transmitted from one viewing user terminal device 10, each terminal device 10 receives the request signal from the server device 20 (information processing server device 20B). The first gift object can be received and displayed on the video including the animation of the avatar object. A specific method for displaying the first gift object on the video will be described later.

Furthermore, when each terminal device 10 operates as a distribution user's terminal device 10, an avatar object is assigned to one or more first gift objects displayed on the video based on the distribution user's operation. When the terminal device 10 determines that the action has been performed, the terminal device 10 can display on the video a second gift object that is different from the first gift object for which the predetermined action was performed. Note that details of the predetermined action, details of the method of determining the predetermined action, and a specific method of displaying the second gift object on the video will be described later.

On the other hand, when each terminal device 10 operates as a terminal device for viewing videos (a viewing user's terminal device), each terminal device 10 operates as a terminal device of a distribution user by executing an installed video viewing application. The distributed video can be received via the server device 20 (distribution server device 20C).

Furthermore, when each terminal device 10 operates as a viewing user's terminal device, by executing a video viewing application, the first gift is sent to the distribution user (avatar object) based on the viewing user's operation. A request signal requesting to give an object (gifting) can be transmitted to the server device 20 (gift API server device 20A).

Furthermore, when each terminal device 10 operates as a viewing user's terminal device, by executing a video viewing application, information regarding comments from the viewing user to the distribution user (avatar object) is transmitted to the server device 20 (comment API). server device 20D).

Note that the video distribution application and the video viewing application described above can be installed and executed on each terminal device 10 as one integrated application or as separate applications.

Each terminal device 10 is any terminal device capable of performing such operations, including, but not limited to, a smartphone, a tablet, a mobile phone (feature phone), a personal computer, etc. It is something that can be done.

1-2. Server device 20
As shown in FIG. 1, the server device 20 can also be called a video distribution system in which a plurality of server devices, which will be described later, are combined.

The server device 20 can distribute a video including the animation of the avatar object transmitted from the terminal device 10 of the distribution user to the terminal device 10 of each viewing user.

Furthermore, the server device 20 sends one or more request signals (a plurality of request signals are sent from one viewing user terminal device 10) requesting that the first gift object be given to the avatar object (gifting). (including both cases in which request signals are sent from a plurality of viewing user terminal devices 10) can be received from the terminal device 10 of each viewing user. Further, the server device 20 transmits information regarding the request signal to the distribution user's terminal device 10, and then the distribution user's terminal device 10 displays a first message corresponding to the request signal on the video including the animation of the avatar object. Generate a video displaying the gift object. The server device 20 can also receive the video (video in which the first gift object is displayed) from the terminal device 10 of the distribution user.

Furthermore, based on the distribution user's actions, the terminal device 10 (the distribution user's terminal device 10) determines that the avatar object has performed a predetermined action on one or more first gift objects displayed on the video. In this case, when the terminal device 10 displays on the video a second gift object different from the first gift object for which the predetermined action has been performed, the server device 20 displays the second gift object on the video. It is also possible to receive a moving image on which is displayed from the terminal device 10.

Further, the server device 20 can receive information regarding comments from the viewing user to the distribution user (avatar object) from the viewing user's terminal device 10, and can transmit the comment to the viewing user who views the same video. Can be delivered.

In order to perform such operations, the server device 20, in one embodiment, includes a gift API server device 20A, which are connected to each other via a communication line (including a wireless line and/or a wired line not shown). It can include an information processing server device 20B, a distribution server device 20C, and a comment API server device 20D.

The gift API server device 20A sends one or more request signals requesting that a first gift object be given to the avatar object (gifting) based on the viewing user's operation, to the terminal device 10 of each viewing user. can be received from. At this time, the gift API server device 20A collects identification information of the viewing user who performed the gifting act from the request signal (serial number of the terminal device 10 of each viewing user, ID information given to each viewing user, etc.), Information regarding the first gift object that was gifted (information regarding the type, name, color, size, etc.), and identification information of the distribution user who is the avatar object to be gifted (such as the serial number of the terminal device 10 of the distribution user) ID information given to the distribution user) and transmit the read information to the information processing server device 20B automatically or when receiving an inquiry from the information processing server device 20B.

The information processing server device 20B receives the information read by the gift API server device 20A from the gift API server device 20A, automatically or by making an inquiry, and transmits the information to the distribution user's terminal device 10. be able to. Further, the information processing server device 20B can have a table (database) for each distribution user, and may store information regarding the first gift object read by the gift API server device 20A in the table for each distribution user. good.

Further, the information processing server device 20B refers to the stored identification information of the viewing user and the information regarding the first gift object, and determines which viewing user is the first gift object for the avatar object (distribution user). All or part of comment information regarding whether a gift object has been given can be transmitted to the comment API server device 20D.

Furthermore, the information processing server device 20B transmits information to the distribution user's terminal that it has been determined that the avatar object has performed a predetermined action on the first gift object and that the second gift object has been displayed on the video. It may include a function of receiving comment information from the device 10 and transmitting comment information regarding the display of the second gift object on the video to the comment API server device 20D.

The distribution server device 20C transmits the video (including the video in which the first gift object is displayed and the video in which the second gift object is displayed) transmitted from the terminal device 10 of the distribution user to the terminal of each viewing user. It can be distributed to the device 10.

The comment API server device 20D can receive the viewing user's comment from the viewing user's terminal device 10, and can distribute the received comment to each viewing user who is viewing the same video. .

The comment API server device 20D also responds to all or part of the information received from the information processing server device 20B regarding which viewing user has given what first gift object to the avatar object (distribution user). comments can be distributed to each viewing user who is viewing the same video.

Furthermore, based on the comment information regarding the second gift object being displayed on the video, which is received from the information processing server device 20B, the comment API server device 20D adds a comment corresponding to the comment information to the same video. It is also possible to distribute the content to each viewing user who is viewing the video.

In the example shown in FIG. 1, for the purpose of distributing the load, the server device 20 is configured to include a gift API server device 20A, an information processing server device 20B, a distribution server device 20C, and a comment API server device 20D. A configuration with four server devices is shown. However, at least one of these four server devices may be integrated with any of the remaining server devices. For example, the information processing server device 20B and the distribution server device 20C may be integrated.

2. Hardware Configuration of Each Device Next, an example of the hardware configuration of each of the terminal device 10 and the server device 20 will be described.

2-1. Hardware Configuration of Terminal Device 10 An example of the hardware configuration of each terminal device 10 will be described with reference to FIG. 2. FIG. 2 is a block diagram schematically showing an example of the hardware configuration of the terminal device 10 (server device 20) shown in FIG. (This is written in relation to the server device 20.)

As shown in FIG. 2, each terminal device 10 mainly includes a central processing unit 11, a main storage device 12, an input/output interface 13, an input device 14, an auxiliary storage device 15, and an output device 16. , can be included. These devices are connected to each other by a data bus and/or a control bus.

The central processing unit 11 is referred to as a "CPU" and performs calculations on instructions and data stored in the main storage device 12, and stores the results of the calculations in the main storage device 12. be. Furthermore, the central processing unit 11 can control the input device 14, the auxiliary storage device 15, the output device 16, etc. via the input/output interface 13. The terminal device 10 may include one or more such central processing units 11.

The main storage device 12 is referred to as a "memory" and stores instructions and data received via the input/output interface 13 from the input device 14, auxiliary storage device 15, and communication network 30 (server device 20, etc.). , as well as the calculation results of the central processing unit 11. The main storage device 12 may include, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), and/or flash memory.

The auxiliary storage device 15 is a storage device that has a larger capacity than the main storage device 12. Instructions and data (computer programs) constituting the aforementioned specific applications (video distribution applications, video viewing applications, etc.) and web browsers are stored and controlled by the central processing unit 11. Instructions and data (computer programs) can be transmitted to main memory 12 via input/output interface 13 . The auxiliary storage device 15 can include, but is not limited to, a magnetic disk device and/or an optical disk device.

The input device 14 is a device that takes in data from the outside, and includes, but is not limited to, a touch panel, a button, a keyboard, a mouse, and/or a sensor. Sensors may include, but are not limited to, sensors including, but not limited to, one or more cameras, etc., and/or one or more microphones, etc., as described below.

The output device 16 can include, but is not limited to, a display device, a touch panel, a printer device, and the like.

In such a hardware configuration, the central processing unit 11 sequentially loads instructions and data (computer programs) constituting a specific application stored in the auxiliary storage device 15 into the main storage device 12, and By operating instructions and data, the output device 16 can be controlled via the input/output interface 13, or other devices (for example, the server device 20 and others) can be controlled via the input/output interface 13 and the communication network 30. It is possible to send and receive various information to and from the terminal device 10 (such as the terminal device 10).

Thereby, by executing the installed specific application, the terminal device 10 performs operations necessary to generate a video, operations necessary to transmit (distribute) and/or receive a video, and gifting actions. , determination as to whether a predetermined action described below has been performed, and operation necessary for issuing a comment (including various other operations described in detail later).

Note that the terminal device 10 may include one or more microprocessors and/or graphics processing units (GPUs) instead of or together with the central processing unit 11.

2-2. Hardware Configuration of Server Apparatus 20 An example of the hardware configuration of each server apparatus 20 will be described with reference to FIG. 2 as well. The hardware configuration of each server device 20 (gift API server device 20A, information processing server device 20B, distribution server device 20C, and comment API server device 20D) is, for example, the same as the hardware configuration of each terminal device 10 described above. It is possible to use the following. Therefore, reference numerals for components included in each server device 20 are shown in parentheses in FIG. 2.

As shown in FIG. 2, each server device 20 mainly includes a central processing unit 21, a main storage device 22, an input/output interface 23, an input device 24, an auxiliary storage device 25, and an output device 26. , can be included. These devices are connected to each other by a data bus and/or a control bus.

The central processing unit 21, the main storage device 22, the input/output interface 23, the input device 24, the auxiliary storage device 25, and the output device 26 are the central processing unit 11 and the main storage device included in each terminal device 10 described above, respectively. 12, input/output interface 13, input device 14, auxiliary storage device 15, and output device 16 can be substantially the same.

In such a hardware configuration, the central processing unit 21 sequentially loads instructions and data (computer programs) constituting a specific application stored in the auxiliary storage device 25 into the main storage device 22, and By operating commands and data, the output device 26 is controlled via the input/output interface 23, or other devices (for example, each terminal device 10, etc.) are controlled via the input/output interface 23 and the communication line 30. ) various information can be sent and received between them.

Thereby, the server device 20 performs the operations necessary to distribute the video generated and transmitted by the terminal device 10 of each distribution user to the terminal device 10 of each viewing user, and performs the operations necessary to distribute the video generated and transmitted from the terminal device 10 of each distribution user to the terminal device 10 of each viewing user. Operations necessary to receive a gifting action, operations necessary to generate request information necessary to cause the terminal device 10 to generate a video, operations necessary to transmit (distribute) and/or receive a video. Operations, etc. (including various operations detailed below) may be performed.

Note that the server device 20 may include one or more microprocessors and/or graphics processing units (GPUs) instead of or together with the central processing unit 21.

3. Functions of Each Device Next, a detailed example of the functions of each of the terminal device 10 and the server device 20 will be described.

3-1. Functions of the terminal device 10 An example of the functions of the terminal device 10 will be described with reference to FIGS. 3 to 8. FIG. 3 is a block diagram schematically showing an example of the functions of the terminal device 10 shown in FIG. 1. FIG. 4 is a diagram illustrating an example in which a first gift object is displayed on a video including an animation of a distribution user's avatar object. FIG. 5 is a diagram illustrating a case where a first gift object is displayed on a video including an animation of a distribution user's avatar object, and two first collision areas overlap. 6 and 7 are diagrams showing an example in which a second gift object is displayed on a video including an animation of a distribution user's avatar object. FIG. 8 is a diagram illustrating an example of an effect object displayed on a video including an animation of a distribution user's avatar object.

As shown in FIG. 3, the terminal device 10 mainly includes a video generation section 100, a display section 110, a determination section 120, a storage section 130, a user interface section 140, a gift storage section 150, and a communication section 160.

(1) Video generation section 100
The video generation unit 100 can generate a video including an animation of the distribution user's avatar object based on data regarding the distribution user's movements and/or audio data regarding speech and singing uttered by the distribution user. To achieve this, the video generation section 100 can include, for example, an acquisition section (not shown) and a processing section (not shown).

The acquisition unit includes one or more first sensors (not shown) that acquire data regarding the face, limbs, etc. (entire body) of the distribution user, and audio data regarding speech and/or singing uttered by the distribution user. and one or more second sensors (not shown) to obtain the .

In a preferred embodiment, the first sensor can include at least an RGB camera that captures images of visible light and a near-infrared camera that captures images of near-infrared rays. Further, the first sensor can include a motion sensor, a tracking sensor, etc., which will be described later. As the above-mentioned RGB camera and near-infrared camera, it is possible to use, for example, those included in the True Depth camera of iPhone X (registered trademark). The second sensor can include a microphone that records audio.

First, regarding the first sensor, the acquisition unit uses the first sensor (camera included in the first sensor) placed close to the distribution user's face, limbs, etc. Take an image. Thereby, the acquisition unit can generate data (for example, an MPEG file) in which an image acquired by the RGB camera is recorded over a unit time in association with a time code (a code indicating the time of acquisition). Further, the acquisition unit stores data (for example, a TSV file) recorded over a unit time by associating a predetermined number (for example, 51) of numerical values (for example, floating point numerical values) indicating the depth acquired by the near-infrared camera with the above-mentioned time code. [File format that records multiple pieces of data by separating data with tabs]) can be generated.

Regarding the near-infrared camera, specifically, a dot projector emits an infrared laser in the form of a dot (dot) pattern onto the face, limbs, etc. of the broadcast user who is the performer, and the near-infrared camera emits an infrared laser that is projected onto the face of the broadcast user and is reflected. It captures infrared dots and generates images of the captured infrared dots. The acquisition unit compares the image of the dot pattern emitted by the dot projector registered in advance and the image captured by the near-infrared camera, and calculates each point (each feature point) (for example, 51 points) in both images. The depth of each point (each feature point) (the distance between each point/feature point and the near-infrared camera) can be calculated using the positional shift at each point (each feature point). The acquisition unit can generate data recorded over a unit time by associating the numerical value indicating the depth calculated in this manner with the time code as described above.

Next, regarding the second sensor, the acquisition unit acquires audio related to speech and/or singing uttered by the distribution user using the second sensor placed close to the distribution user. Thereby, the acquisition unit can generate data (for example, an MPEG file) recorded over a unit time in association with the time code. In one embodiment, the acquisition unit uses the first sensor to acquire data regarding the face and limbs of the distribution user, and at the same time uses the second sensor to acquire data regarding the speech and/or singing uttered by the distribution user. Audio data can be obtained. In this case, the acquisition unit associates the image acquired by the RGB camera and the audio data related to speech and/or singing uttered by the distribution user using the second sensor with the same time code. Data recorded over a unit time (eg, an MPEG file) can be generated.

The acquisition unit generates data related to the distribution user's face, limbs, etc. (MPEG files, TSV files, etc.) and/or audio data related to speech and/or singing uttered by the distribution user (MPEG files, etc.). ) can be output to the processing unit.

Note that here, a case has been described in which the first sensor includes an RGB camera and an infrared camera, but the first sensor may, for example, detect infrared rays emitted by (A) an infrared laser and reflected on the face, limbs, etc. of the distribution user. (B) a plurality of RGB cameras that capture visible light; and (C) a single camera that captures visible light. In the case of (A) above, the acquisition unit can calculate the depth of each feature point on the distribution user's face, limbs, etc. using a method similar to that described above. In the case of (B) above, the acquisition unit can calculate the depth of each feature point on the distribution user's face, limbs, etc., using multiple images captured by multiple RGB cameras. . In the case of (C) above, the acquisition unit uses deep learning etc. to calculate the depth of each feature point on the distribution user's face, limbs, etc. from the image captured by a single camera. can do. In the case of (C) above, the acquisition unit calculates the depth of each feature point on the distribution user's face, limbs, etc. by analysis processing (image processing, etc.) on the image captured by a single camera. may be calculated.

Further, the acquisition unit acquires detection information regarding the position and orientation of each part of the distribution user's body using a motion sensor and a tracking sensor included in the first sensor, and outputs the detection information to the processing unit. I can do it. Note that the details of the detection information regarding the position and orientation of each part of the distribution user's body acquired using the motion sensor and the tracking sensor will be described later.

The processing unit can generate a video including an animation of a virtual character (avatar object) based on data regarding the distribution user's face, limbs, etc. (and various information regarding detection information, etc.) from the acquisition unit. . Regarding the animation itself of the avatar object, the processing unit uses various information (for example, geometry information, bone information, texture information, shader information, blend shape information, etc.) stored in a character data storage unit (not shown) to It is also possible to generate a moving image of an avatar object by having a rendering unit that does not perform rendering execute rendering.

Note that any other well-known method may be used to generate a video including the animation of the avatar object of the distribution user based on data regarding the distribution user's movements and/or audio data regarding speech and singing uttered by the distribution user. It is also possible to use technology.

By the way, in response to a gifting from a viewing user, the communication unit 160, which will be described later, sends request information to the server device 20 (information processing server 20B) to display a first gift object on a video including an animation of an avatar object. ), the video generation unit 100 receives the request information via the communication unit 160, and generates identification information (name・Display the first gift object on a video including the animation of the avatar object (generate a video in which the first gift object is displayed) with reference to the shape, color, etc.).

Specifically, in order to display a first gift object on a video in response to a gifting action from a viewing user, each first gift object is provided with a first collision area corresponding to its shape in advance. The information about each first gift object and the corresponding first collision area is stored together with information about each first gift object (size, shape, color, etc.) in the gift storage unit 150, which will be described later. ing. Furthermore, in order to specify at which position on the video the first gift object is to be displayed, position coordinates in a three-dimensional orthogonal coordinate system are set for each of the first gift objects at the time of gifting. When setting the positional coordinates, positional coordinates that do not overlap on the video with various objects such as the avatar object and the first gift object that are already displayed on the video are set. The position coordinates that do not overlap on the video means that the plurality of first collision areas set for each of the plurality of first gift objects displayed on the video do not overlap with each other, and the position coordinates that do not overlap with each other on the video This means that the first collision area does not overlap with the second collision area set for each part of the object) or the third collision area set for another object (for example, the second gift object). In order to satisfy the conditions, the video generation unit 100 sets positional coordinates regarding three-dimensional orthogonal coordinates (X-axis in the horizontal direction, Y-axis in the depth direction, and Z-axis in the vertical direction) for each of the first gift objects. . Note that the position coordinates refer to position coordinates regarding the X, Y, and Z axes corresponding to the center position of the first gift object (for example, corresponding to the center position _Z0 of the first gift object 410A in FIG. 5). (e.g., corresponding to the first gift object 410A in FIG. 5) (position coordinates corresponding to positions Z ₁ to Z ₁₂ forming the outer edge of the first collision area). Based on the position coordinates (and the first collision area) set in this way, each first gift object is displayed on the video including the animation of the avatar object. Further, based on the position coordinates set in this way, it can be determined whether the plurality of first collision areas overlap. Although it has been explained that the position coordinates, first collision area, and second collision area set for each of the first gift objects are set in a three-dimensional orthogonal coordinate system, the video is expressed in two dimensions. In some cases, it may be set in a two-dimensional orthogonal coordinate system.

Further, when the determination unit 120 (described later) determines that the avatar object has performed a predetermined action on the first gift object displayed on the video based on the distribution user's action, the video generation unit 100 A signal related to the determination is received from the determination unit 120, and a second gift object different from the first gift object is displayed on the video including the animation of the avatar object (a video in which the second gift object is displayed) ).

Specifically, when the determination unit 120 determines that the avatar object has performed a predetermined action on the first gift object displayed on the video based on the distribution user's action, the second gift object is In order to display on the video, a third collision area corresponding to the shape of each second gift object is set in advance, and information regarding each second gift object and the corresponding third collision area is set in advance. is stored together with information regarding the second gift object (size, shape, color, etc.) in the gift storage unit 150, which will be described later. Furthermore, in order to specify where on the video the second gift object is to be displayed, position coordinates in a three-dimensional orthogonal coordinate system are set for each of the second gift objects. When setting the positional coordinates, positional coordinates that do not overlap on the video with various objects such as the avatar object and the first gift object that are already displayed on the video are set. As mentioned above, positional coordinates that do not overlap on a video are a plurality of first collision areas set for each of a plurality of first gift objects displayed on a video or a second collision area set for an avatar object. This means that the third collision area does not overlap, and in order to satisfy this condition, the video generation unit 100 sets each of the second gift objects in three-dimensional orthogonal coordinates ((X-axis in the horizontal direction, Y-axis in the Set the position coordinates in the depth direction (the Z axis is the vertical direction).The position coordinates are the X and Y axes corresponding to the center position of the second gift object, similar to the first gift object described above. , and positional coordinates regarding the Z-axis, and/or positional coordinates regarding the X-axis, Y-axis, and Z-axis corresponding to a plurality of positions forming the outer edge of the third collision area.Positional coordinates set in this way (and the third collision area), each second gift object is displayed on the video including the animation of the avatar object. Furthermore, the position coordinates and the third collision area set for each of the second gift objects are displayed on the video including the animation of the avatar object. Although it has been explained that the area is set using three-dimensional orthogonal coordinates, if the moving image is expressed in two dimensions, it may be set using a two-dimensional orthogonal coordinate system.

By the way, as described above, when displaying the second gift object on the video triggered by the determination unit 120 determining that the predetermined action has been performed, the first gift object for which the predetermined action has been performed is Various methods can be employed. For example, a configuration is adopted in which when a second gift object is displayed on a video, a first gift object for which a predetermined action has been performed is deleted from the video. Furthermore, the first gift object for which the predetermined action has been performed is not deleted from the video, and the form (e.g., shape, size, color, etc.) of the first gift object gradually changes over time (e.g. , when only the size of the first gift object changes, when the color of a part of the first gift object changes, etc.), a second gift object is generated (when the second gift object changes). It may also be a configuration (displayed on the video). Further, both of the above-mentioned configurations may be adopted (for example, if there are two first gift objects on which a predetermined action has been performed, one of the two first gift objects may be deleted from the video, and the form of the other first gift object may gradually change to generate a second gift object).

Note that when a predetermined condition is satisfied, the video generation unit 100 recognizes that the first gift object is held by the avatar object, and as shown in FIG. It is possible to generate a video of the object being grasped. Here, the predetermined condition means that the first collision area set on the first gift object and the second collision area set on the hand (or finger) of the avatar object are based on the distribution user's actions. This refers to a case where it is determined that at least a portion of the video overlaps (overlaps), and the hands (or fingers) of the avatar object are determined to be bent. Each of these determinations may be executed by the video generation unit 100, or a configuration may be adopted in which each determination result is transmitted to at least the video generation unit 100 after being executed by another component. Note that the method for determining that the hands (or fingers) of the avatar object are bent can be performed based on signals from the controller, which will be described later. Note that the second gift object is held in the same way by the avatar object. In other words, it is determined that the third collision area set for the second gift object and the above-mentioned second collision area overlap (overlap) at least in part in the video, and the avatar object's hand (or If it is determined that the second gift object (finger) is bent, the second gift object is grasped by the avatar object.

Note that judgments regarding whether or not the plurality of first collision areas overlap and whether or not the first collision area and the second collision area overlap are made based on the X-axis, Y-axis, and Y-axis corresponding to the outer edges of each collision area. and a plurality of position coordinates regarding the Z axis.

As described above, when displaying the second gift object on the video in response to the determination by the determination unit 120 that a predetermined action has been performed, the following describes what kind of second gift object to display on the video. , the video generation unit 100 makes the determination with reference to setting conditions stored in the gift storage unit 150, which will be described later. The setting conditions include, for example, firstly, the type of the first gift object on which the predetermined action was performed, the number of grants for each type of the first gift object on which the predetermined action was performed, and the specific position (described later). In FIG. 4, the conditions are associating what kind of second gift object corresponds to various combinations formed from the specific position Y1 or Y2). Second, in the case where at least one gift object is brought into contact with another first gift object, the setting condition is based on the type and number of first gift objects (number of grants) on which the predetermined action has been performed. This is information that associates what kind of second gift object corresponds to various combinations.

A specific example of the first case of the setting condition described above will be explained with reference to the case shown in FIG. (reference numeral 410A), when the number of grants for each type of first gift object ("boy's stuffed toy" 410A) for which the predetermined action was performed is "1", and the specific position is "Y1", the second gift As shown in FIG. 6, the objects are set in advance as a plurality of "small stuffed bears" (reference numeral 420 in FIG. 6). Similarly, in the case of FIG. 4, the type of the first gift object for which the predetermined action was performed is "boy's stuffed toy" (reference numeral 410B in FIG. 4); If the number of gifts for each item (“boy stuffed toy” 410B) is “1” and the specific position is “Y2”, the second gift object is one “large elephant stuffed toy” (not shown). It is set in advance. Various conditions related to these setting conditions may be changed as appropriate, and are stored in the gift storage unit 150 in the form of a table, for example.

On the other hand, as a specific example of the second case of the above-mentioned setting condition, it will be explained with reference to the case shown in FIG. ” 410A is “one” and “boy stuffed toy” 410B is “one”, the second gift object is one “little boy stuffed toy” (in FIG. 7), as shown in FIG. Reference number 430) is set in advance. As described above, various conditions related to these setting conditions may be changed as appropriate. For example, the contact time between the first gift objects or the collision speed when the first gift objects contact each other may be set as conditions. (Specifically, when the collision speed is equal to or higher than αkm/h, a “cheetah stuffed toy” is displayed as the second gift object, and when it is less than αkm/h, a “turtle stuffed toy” is displayed as the second gift object.) 410A) and the relative positions of the plurality of first gift objects upon contact for which a predetermined action has been performed (for example, in the case shown in FIG. 410B, the type of the second gift object to be displayed may be different depending on whether it is above or below 410B). Further, these setting conditions are stored in the gift storage unit 150 in the form of a table, for example.

Note that the above-mentioned first and second setting conditions include the second gift objects displayed on the video (for example, a plurality of "little stuffed bears" 420 in FIG. 6, and "little boy" in FIG. 430) may be displayed on the video. Specifically, in the case of FIG. 6, each of the plurality of "small stuffed bears" 420 as the second gift object has an Information regarding individual position coordinates regarding the axis, Y axis, and Z axis is preset (that is, the position where the second gift object is displayed is the position where the first gift object on which the predetermined action was performed is located). (It may also be displayed in a different position than the original position.) On the other hand, in the case of FIG. 7, the "little boy's stuffed toy" 430 as the second gift object has the same positional coordinates as the first gift object (for example, "boy's stuffed toy" 410A). Information regarding the position coordinates of is set in advance.

Furthermore, when displaying the second gift object on the video triggered by the determination unit 120 determining that the predetermined action has been performed, the video generation unit 100 displays the second gift object as shown in FIG. An effect object 500 indicating a state in the middle of being displayed on the video (a state immediately before the second gift object is displayed) may be displayed on the video. In this case, the video generation unit 100 performs a predetermined action (in FIG. 8, the first gift object "boy's stuffed toy" 410A is brought into contact with another first gift object "boy's stuffed toy" 410B). The effect object 500 is displayed on the video at the time when the determination unit 120 determines that the second gift object is actually displayed on the video, and the effect object 500 is displayed at the time when the second gift object is actually displayed on the video. Delete it from the video. The video generation unit 100 also refers to the positional coordinates where the two first collision areas (for example, the first collision areas 411A and 411B in FIG. 5) overlap, and creates an effect object at the overlapping positional coordinates. 500 can be displayed.

(2) Display section 110
The display unit 110 can display various information required for distributing and/or viewing a video (including a video displaying the first gift object and a video displaying the second gift object). can. For example, the display unit 110 can display distributed videos, received videos, generated videos, and the like. Further, the display unit 110 can also display various comments distributed from the server device 20 to each terminal device 10 together with the received video. Furthermore, the gift information (information regarding the first gift object) stored in the gift storage section 150 can be displayed as appropriate in response to the viewing user's operation.

(3) Determination unit 120
The determination unit 120 determines that the video generation unit 100 generates a state in which one or more first gift objects are displayed on a video including an animation of an avatar object (a video in which one or more first gift objects is displayed is In the generated state), it is determined whether the avatar object has performed a predetermined action with respect to the first gift object displayed on the video, based on the distribution user's action. Here, the predetermined action is, for example, an action in which the avatar object moves a first gift object to a specific position in the video, or an action in which the avatar object moves at least one first gift object to another first gift object. may include touching, sliding, or colliding actions. Note that the above-mentioned actions are all based on the actions of the distribution user.

As described above, each of the first gift objects has a unique first collision area (corresponding to the form of the first gift object) set in advance, and the hand (or finger) of the avatar object has a second collision area. When the area is preset and the predetermined conditions described above are met, the first gift object is grasped by the avatar object. Note that, although the avatar object can hold two first gift objects with both hands in principle, the invention is not necessarily limited to this. For example, the avatar object may hold three or more first gift objects in both hands, or if multiple avatar objects are displayed in the video, the avatar object may hold four or more first gift objects. A gift object can be grasped by an avatar object.

As shown in FIG. 4, the determination unit 120 starts from a state in which the avatar object 400 grips the first gift object 410A (or 410B) at the first position X1 (or first position X1'). Based on the distribution user's actions, the first gift object 410A (or 410B) is moved to a second position that is different from the first position X1 (or first position X1') while maintaining the gripped state. When moving to position Y1 (or second position Y2), it is determined that an action to move to a specific position in the video has been executed. In this case, the second position Y1 (or second position Y2) becomes the specific position. Note that the first position X1 (or first position X1') is changed as appropriate at any position on the video. Further, the second position Y1 (or the second position Y2) may be set in advance at any position on the video in the XYZ coordinate system.

Further, as shown in FIG. 4, starting from a state in which the avatar object 400 holds the two first gift objects 410A and 410B in both hands, as shown in FIG. When the first gift objects 410A and 410B are brought into contact, the determination unit 120 determines that a predetermined action has been performed. Here, when the two first gift objects 410A and 410B come into contact, it means that the first collision areas 411A and 411B set for each of the two first gift objects 410A and 410B overlap at least in part ( overlap). Furthermore, the two first gift objects 410A and 410B contacting each other also includes a manner in which they slide or collide.

By the way, as explained above, the first collision areas 411A and 411B set for each of the two first gift objects 410A and 410B overlap (overlap) at least in part. The determination is made based on the position coordinates regarding the X-axis, Y-axis, and Z-axis corresponding to 411A and 411B.

(4) Storage unit 130
The storage unit 130 can store various information required for distributing and/or viewing videos (including videos displaying the first gift object and videos displaying the second gift object). can.

(5) User interface section 140
The user interface unit 140 distributes various information required for distributing and/or viewing a video (including a video displaying a first gift object and a video displaying a second gift object). The information can be input through operation by the user or viewing user. For example, the user interface unit 140 allows the viewing user to perform an operation related to gifting to the avatar object (such as selecting one of a plurality of first gift objects prepared in advance and selecting a command to be given to the avatar object). operation data indicating the content of the operation (including the operation of the avatar object), and comment information from the viewing user to the distribution user (avatar object) (for example, the viewing user says to the distribution user, ``Let's hit the two boy stuffed animals you are holding in your hands!'' It is possible for the viewing user to input data indicating the contents of comment information (such as comment information that provides a hint for a predetermined action such as ""), and output these various data to the communication section 160. Note that the above-mentioned operation data can include, for example, information regarding what kind of first gift object (type of first gift object) the viewing user gifted to the distribution user.

(6) Gift storage section 150
When the video distribution application and the video viewing application are installed on the terminal device 10, the gift storage unit 150 can store gift information that is preset in these applications. Note that the gift information includes identification information such as the name, shape, and color of the first gift object and the second gift object, and information provided to the viewing user when the viewing user gifts the first gift object. Information regarding the amount per first gift object to be claimed may be included.

Note that the above-mentioned identification information also includes information regarding the above-mentioned first collision area and third collision area corresponding to each of the first gift object and the second gift object.

Furthermore, the gift storage unit 150 stores the above-mentioned setting conditions that the video generation unit 100 refers to when displaying the second gift object on the video triggered by the determination unit 120 determining that a predetermined action has been performed. Remember.

(7) Communication department 160
The communication unit 160 transmits various information required for distribution and/or viewing of videos (including videos displaying the first gift object and videos displaying the second gift object) to the server device 20. It is possible to communicate with For example, when the terminal device 10 operates as the terminal device 10 of the distribution user, the communication unit 160 sends request information to the server device 20 ( The generated video can be received from the information processing server device 20B) and transmitted to the server device 20 (distribution server device 20C).

Furthermore, when the video generation unit 100 displays the second gift object on the video because the determination unit 120 determines that the predetermined action has been performed, the communication unit 160 displays the second gift object. The video can be transmitted to the server device 20 (distribution server device 20C). In this way, the communication unit 160 can receive various videos generated by the video generating unit 100 from the video generating unit 100, and can transmit these various videos to the server device 20 (distribution server device 20C). .

On the other hand, when the terminal device 10 operates as the viewing user's terminal device 10, the communication unit 160 can receive the video generated by the distribution user's terminal device 10 from the server device 20. Similarly, when the terminal device 10 operates as the viewing user's terminal device 10, information regarding gifting from the viewing user to the distribution user (avatar object) and information regarding comments from the viewing user to the distribution user (avatar object) are transmitted. It can be transmitted to the server device 20 (comment API server device 20D).

(8) Studio unit A studio unit (not shown) including the aforementioned video generation section 100, display section 110, determination section 120, storage section 130, user interface section 140, gift storage section 150, and communication section 160 is used as a terminal. It can also be used as the device 10.

The studio unit can be implemented by an information processing device such as a personal computer, and, like the terminal device 10 and server device 20 described above, mainly includes a central processing unit, a main storage device, and an input/output interface. , an input device, an auxiliary storage device, and an output device. These devices are connected to each other by a data bus and/or a control bus. In addition, the studio unit functions as an information processing device that executes a specific installed application, thereby providing information on the behavior of the distribution user in the studio room (not shown) in which the studio unit is installed, and/or the distribution user. It is possible to generate a video including animation of the avatar object based on data related to speech and singing uttered by the user.

Most of the components of the video generation section 100 included in the studio unit are arranged in the studio room. Furthermore, in the studio room, distribution users perform various performances, and the studio unit detects the distribution users' movements, facial expressions, speech (including singing), and the like.

The distribution user's movements, facial expressions, speech (including singing), etc. are to be captured by various sensor groups included in the studio unit (video generation section 100). In this case, the distribution user does not necessarily have to be a human, and may be a robot capable of independent movement, for example. Further, the number of distribution users in the studio room may be one, or two or more.

The video generation section 100 in the studio unit includes various motion sensors (not shown) attached to multiple parts of the distribution user (for example, wrists, insteps, hips, top of the head, etc.) and motion sensors held in the distribution user's hands. and a camera (not shown) attached to the head of the distribution user via a mounting device (not shown). Further, the studio unit is provided with a microphone in the above-mentioned mounting device or somewhere in the studio room (for example, on the wall, floor, ceiling, etc.) in order to acquire the audio data of the distribution user. Additionally, the studio unit may have multiple base stations (not shown), tracking sensors (not shown), and displays (not shown) in addition to the configurations described above.

The aforementioned motion sensor can cooperate with the aforementioned base station to detect the location and orientation of the broadcast user. In one embodiment, the plurality of base stations are multi-axis laser emitters, and after emitting a synchronizing flashing light, one base station scans the laser light, e.g. around a vertical axis, and the other base stations is configured, for example, to scan a laser beam around a horizontal axis. The motion sensor is equipped with multiple optical sensors that detect the incidence of flashing light and laser light from the base station. It is possible to detect the incident angle of the laser beam detected by the sensor. The motion sensor may be, for example, Vive Tracker provided by HTC CORPORATION, or Vive Tracker provided by ZERO C SEVEN Inc. It may also be Xsens MVN Analyze provided by Microsoft.

The video generation section 100 in the studio unit acquires detection information indicating the position and orientation of each motion sensor calculated by the motion sensor. By being attached to the distribution user's wrist, instep, waist, top of the head, etc., the motion sensor can detect the position and orientation of the motion sensor and detect the movement of each part of the distribution user's body. can. Note that the detection information indicating the position and orientation of the motion sensor is calculated as position coordinate values in the XYZ coordinate system for each body part of the distribution user within the video (in the virtual space included in the video). The X axis corresponds to the horizontal direction within the video, the Y axis corresponds to the depth direction within the video, and the Z axis corresponds to the vertical direction within the video. Therefore, the movements of each part of the distribution user's body are all detected as position coordinate values in the XYZ coordinate system.

In one embodiment, a plurality of motion sensors are equipped with a large number of infrared LEDs, and light from the infrared LEDs is detected by an infrared camera installed on the floor or wall of a studio room. Position and orientation may also be detected. Furthermore, the position and orientation of the motion sensor may be detected by using a visible light LED instead of the infrared LED and detecting the light from the visible light LED with a visible light camera.

In one embodiment, a plurality of reflective markers may be used instead of a motion sensor. The reflective marker is attached to the distribution user using adhesive tape or the like. By photographing the distribution user with the reflective marker affixed in this way to generate photographic data, and performing image processing on this photographic data, the position and orientation of the reflective marker (as described above, the position coordinate value in the XYZ coordinate system) can be calculated. ) may be configured to detect.

The controller outputs a control signal in response to an operation such as bending a finger by the distribution user, and the video generation unit 100 acquires this control signal.

The tracking sensor generates tracking information for determining virtual camera setting information for constructing a virtual space included in the video. The tracking information is calculated as a position in a three-dimensional orthogonal coordinate system and an angle around each axis, and the video generation unit 100 acquires the tracking information.

A camera attached to the distribution user's head is arranged so as to be able to capture an image of the distribution user's face. The camera acquires image data of the distribution user's face by continuously capturing images of the distribution user's face, and the video generation unit 100 acquires this image data from the camera. Note that the camera may be a 3D camera that can detect the depth of the distribution user's face.

The display functions as the display unit 110 and is placed in a position where it can be viewed by the distribution user, and the distribution user can acquire various information and view the video generated by the video generation unit 100 through the display. It is possible.

The video generation unit 100 can generate a video including the animation of the avatar object by acquiring information from the aforementioned motion sensor, tracking sensor, various cameras, controller, and the like.

In addition to the components and functions described above, the studio unit also includes the video generation section 100, display section 110, determination section 120, storage section 130, user interface section 140, and gift storage section described above. 150, and a communication unit 160 (or another component having these functions). Thereby, it is possible to make a determination regarding the above-mentioned predetermined action, display the first gift object and/or the second gift object on the video including the animation of the avatar object, and the like. The components of the studio unit described above are merely examples, and may include various other applicable components.

3-2. Functions of the server device 20 An example of the functions of the server device 20 will be described with reference to FIG. 9. FIG. 9 is a block diagram schematically showing an example of the functions of the server device 20 shown in FIG. 1.

As shown in FIG. 9, the server device 20 can mainly include a storage section 200, a communication section 210, a gift processing section 220, and a comment processing section 230.

(1) Storage unit 200
The storage unit 200 can store various information required for distributing and/or viewing videos (including videos displaying the first gift object and videos displaying the second gift object). can.

(2) Communication department 210
The communication unit 210 transmits various information necessary for distributing and/or viewing a video (including a video displaying the first gift object and a video displaying the second gift object) to the distribution user. Communication is possible between the terminal device 10 and/or the terminal device 10 of the viewing user. For example, the communication unit 210 receives a video (including a video displaying a first gift object and a video displaying a second gift object) from the distribution user's terminal device 10 and receives the video from the terminal device of each viewing user. It can be distributed to the device 10.

Further, when receiving information regarding gifting from the viewing user to the distribution user (avatar object) and information regarding comments from the viewing user to the distribution user (avatar object) from the viewing user's terminal device 10, the communication unit 210 receives information regarding gifting from the viewing user to the distribution user (avatar object). Information is output to the gift processing section 220, and information regarding comments is output to the comment processing section 230.

Furthermore, the communication unit 210 obtains information regarding the first gift object read by the gift processing unit 220 from the gift processing unit 220, and causes the first gift object to be displayed on the video including the animation of the avatar object. Request information to that effect can be transmitted to the distribution user's terminal device 10. Note that the information regarding the first gift object that the communication unit 210 obtains from the gift processing unit 220 means that the video generation unit 100 in the distribution user's terminal device 10 collects a large number of first gift objects stored in the gift storage unit 150. Any information is sufficient as long as it is possible to specify which first gift object's information should be referred to from among the information related to the gift object. For example, when a "boy's stuffed toy" is gifted as a first gift object from the viewing user's terminal device 10, the communication unit 210 receives the first gift as the "boy's stuffed toy" from the gift processing unit 220. It may be only the name of the object (or a character string similar to the name, etc.).

Furthermore, the communication unit 210 receives information from the distribution user's terminal device 10 that it has been determined that the avatar object has performed a predetermined action on the first gift object and that the second gift object has been displayed on the video. It is also possible to receive the information and output the information to the comment processing section 230.

(3) Gift processing section 220
When the gift processing unit 220 receives the information regarding gifting from the communication unit 210, the gift processing unit 220 receives the identification information of the viewing user who has performed the gifting act (the serial number of the terminal device 10 of each viewing user and the ID information given to each viewing user). etc.), information regarding the gifted first gift object (identification information such as name, size, shape, color, etc. of the first gift object), and identification information of the distribution user who is the avatar object to be gifted (distribution The serial number of the user's terminal device 10, ID information given to the distribution user, etc.) are read out.

Regarding the first gift object, the gift processing unit 220 determines which one the video generation unit 100 in the distribution user's terminal device 10 selects from among the many pieces of information regarding the first gift object stored in the gift storage unit 150. It is only necessary to read out enough information to specify whether to refer to the information of the first gift object (for example, as described above, the name of the first gift object or a character string similar to the name, etc.). Identification information such as the size, shape, color, etc. of the gift object does not necessarily have to be read out. Further, the information regarding the first collision area corresponding to each first gift object is not read out by the gift processing unit 220 but is stored in the gift storage unit 150 described above.

Note that the various information read by the gift processing section 220 is output to the storage section 200 and stored in the storage section 200.

(4) Comment processing unit 230
The comment processing unit 230 receives comment information from the viewing user to the distribution user (avatar object) (for example, a predetermined action of the viewing user to the distribution user such as "Try hitting the two "boy stuffed animals" you are holding in both hands!") (including comment information that may give hints) can be received from the viewing user's terminal device 10 via the communication unit 210, and the comment information from the viewing user can be received by each viewer who is viewing the same video. It can be distributed to viewing users.

Further, the comment processing unit 230 refers to the various information read out by the gift processing unit 220 (information stored in the storage unit 200), and determines which viewing user has given which first gift object to the avatar object. Create comment information that corresponds to all or part of the information regarding whether the gift was given (for example, comment information such as ``Viewing user X gifted a boy's stuffed animal!''), and It is also possible to distribute the video to each viewing user who is viewing the same video.

Furthermore, when the comment processing unit 230 receives information from the communication unit 210 that the second gift object is displayed on the video, the comment processing unit 230 transmits comment information corresponding to that (for example, “Many small bears It is also possible to create comment information such as ``A stuffed animal has appeared!'' and distribute the comment information to each viewing user who is viewing the same video.

Furthermore, the comment processing unit 230 receives comment information from the viewing user to the distribution user (avatar object) from the viewing user's terminal device 10 via the communication unit 210. By referring to the various information read out by It is also possible to create comment information such as "!" by yourself and distribute the comment information to each viewing user who is viewing the same video.

4. Overall Operation of Video Distribution System 1 Next, the overall operation performed in the video distribution system 1 having the above configuration will be described with reference to FIG. 10. FIG. 10 is a flow diagram showing an example of operations performed in the video distribution system 1 shown in FIG. 1.

First, in step (hereinafter referred to as "ST") 300, the terminal device 10 (herein, the terminal device 10A) starts and executes a video distribution application according to the operation of the distribution user.

Next, in ST301, the terminal device 10A generates a video including an animation of the distribution user's avatar object based on data regarding the distribution user's actions and/or audio data regarding speech and singing uttered by the distribution user. As a result, a video including an animation of the distribution user's avatar object 400 (for example, a video of the state before the first gift objects 410A and 410B are displayed in FIG. 4) is displayed on the display unit 110 of the terminal device 10A. be done.

Further, the terminal device 10A transmits the generated video to the distribution server device 20C of the server device 20. The distribution server device 20C distributes the video received from the terminal device 10A to the terminal devices 10 (here, the terminal device 10B, the terminal device 10C, and the terminal device 10D) of each viewing user who is running a video viewing application. do. As a result, the same video is also displayed on the display unit 110 of the terminal device 10 of each viewing user.

Next, in ST302, the viewing user's terminal device 10 requests the distribution user (avatar object) to give the first gift object based on the viewing user's operation on the user interface unit 140. A signal is sent to the server device 20 (gift API server device 20A).

Next, in ST303, the server device 20 (gift API server device 20A) that has received the request signal regarding the first gift object from the viewing user's terminal device 10 sends the identification information of the viewing user who performed the gifting act, the gifting information regarding the first gift object and identification information of the distribution user who is the avatar object to be gifted.

Then, in ST304, the server device 20 (information processing server device 20B) distributes request information to display the first gift object on the video including the animation of the avatar object, based on the read information. It is transmitted to the user's terminal device 10. At this time, the information processing server device 20B receives at least part of the information regarding the first gift object (for example, as described above, the name of the first gift object or a character string similar to the name, etc.) from the gift API server device 20A. and generates request information to display the first gift object on the video including the animation of the avatar object.

Next, in ST305, the distribution user's terminal device 10A displays the first gift object corresponding to the request information on the video including the animation of the avatar object based on the request information transmitted from the server device 20 in ST304. Display. Note that when the distribution user's terminal device 10A displays the first gift object corresponding to the request information on the video, the distribution user's terminal device 10A displays the first gift object stored in the gift storage unit 150 described above. Identification information such as the name, shape, color, first collision area, etc. of the gift object is referenced.

Next, in ST306, when the distribution user's terminal device 10A determines that the avatar object has performed a predetermined action on one or more first gift objects displayed on the video, the distribution user's terminal device 10A executes a second gift object. Display on video. Note that the determination regarding the predetermined action is performed by the determination unit 120 of the distribution user's terminal device 10A, as described above. Furthermore, the method of processing the first gift object for which the predetermined action has been performed, the method of displaying the second gift object on the video, etc. are as described above.

As described above, the step ends when the second gift object is displayed on the video.

5. Regarding Modifications In the embodiment described above, the generation of the video (including the video displaying the first gift object and the video displaying the second gift object) is executed by the distribution user's terminal device 10. However, in another embodiment, it may be executed by the server device 20 or the viewing user's terminal device 10, for example. In this case, a rendering unit (not shown) is provided in the server device 20 and/or the viewing user's terminal device 10, and the distribution user's terminal device 10 is connected to the server device 20 and/or the viewing user's terminal device 10. can generate a video (including a video displaying the first gift object and a video displaying the second gift object). A configuration may be adopted in which only the identification information and various types of identification information regarding the second gift object are generated.

In order to realize such a configuration, the functions of the server device 20, the viewing user's terminal device 10, and the distribution user's terminal device 10 are changed from the above-mentioned embodiment as appropriate. sell. For example, in one embodiment, the video generation unit 100 is provided in the terminal device 10, but it may be provided in the server device 20, or may be provided in the terminal device 10 or the viewing user's terminal device 10.

Further, in the above-described embodiment, the determination as to whether or not the predetermined action has been performed is performed by the distribution user's terminal device 10, but in another embodiment, the server device 20 or the terminal device 10 may also be used for viewing and listening. It may be executed on the user's terminal device 10. In this case, the server device 20 or the viewing user's terminal device 10 is equipped with the same function as the determination unit 120.

Therefore, according to various embodiments, it is possible to provide a video processing method, a server device, and a computer program that increase gifting opportunities by viewing users. Furthermore, by changing the first gift object to the second gift object, it is possible to increase the variety of gift objects displayed on the video. Furthermore, in the case of changing a plurality of first gift objects into one second gift object among various embodiments, the object displayed within a limited area of the display unit 110 of the viewing user's terminal device 10 Since the number of first gift objects can be smoothly reduced (in this case, the size of the second gift object displayed on the video is preferably smaller than the first gift object), It becomes possible to efficiently improve the viewing user's gifting motivation. Furthermore, by creating an embodiment in which the first gift object is changed into a second gift object through a predetermined action of the distribution user (avatar object), an opportunity for the distribution user to perform some action is created. be able to. Furthermore, according to various embodiments, the viewing user may provide advice related to a predetermined action to the distribution user using comment information (for example, the viewing user's message regarding the distribution user's message "Two boy stuffed animals held in both hands") "Try hitting me!" (comment information), thereby activating communication between the broadcasting user and the viewing user.

6. Regarding Various Aspects A video processing method according to a first aspect is a video processing method executed by one or more processors that execute computer-readable instructions, the video processing method being performed by one or more processors that execute computer-readable instructions, A generation step of generating a video including an animation of an avatar object, and a request signal generated based on an operation of a viewing user who views the video and requesting a first gift object to be given to the avatar object. , a receiving step of receiving from the viewing user's terminal device via a communication line, and a first step of displaying the first gift object given to the avatar object based on the request signal on the video. If it is determined that the avatar object has performed a predetermined action on one or more of the first gift objects displayed on the video based on the display step and the action of the distribution user, a second display step of displaying a second gift object different from the first gift object on which a predetermined action has been performed on the moving image.

In the video processing method according to the second aspect, in the first aspect, "the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user." be.

In the video processing method according to the third aspect, in the first aspect, the predetermined action is such that the avatar object transfers at least one of the first gift objects to another gift object based on the action of the distribution user. 1 is an action of touching the gift object of No. 1.

In the video processing method according to the fourth aspect, in the third aspect, "a first collision area is set for each of the first gift objects, and at least two of the first collision areas are set within the video. At least two first gift objects are recognized as being in contact when it is determined that they overlap at least in part.

In the video processing method according to the fifth aspect, in the fourth aspect, "the determination as to whether or not at least two of the first collision areas overlap at least in part within the video is performed in the horizontal direction within the video. This is executed based on the position coordinates of each of the first collision areas based on the X axis, the Y axis in the depth direction, and the Z axis in the vertical direction.

In the video processing method according to the sixth aspect, in the second aspect, "a first collision area is set for each of the first gift objects, and a second collision area is set for the hand of the avatar object." and the first collision area and the second collision area are determined to overlap at least in part in the video, and the finger of the avatar object is bent based on the action of the distribution user. If it is determined that the first gift object is being held by the avatar object, it is recognized that the first gift object is being held by the avatar object, and the first gift object is being held by the avatar object at a first position in the video. When the state is started and the grasped state is maintained up to a second position in the video different from the first position, it is recognized that the action of moving to the specific position has been performed. It is.

In the video processing method according to the seventh aspect, in the sixth aspect, "the determination of whether or not the first collision area and the second collision area overlap at least in part within the video is performed in accordance with the sixth aspect. This is executed based on the position coordinates of the first collision area and the position coordinates of the second collision area based on the horizontal X axis, the depth Y axis, and the vertical Z axis.

In the video processing method according to the eighth aspect, in the second aspect, "the second gift object is assigned according to the specific position, the type of the first gift object, and the type of the first gift object. ``determined based on the number and displayed on the video.''

In the video processing method according to the ninth aspect, in the third aspect or the fourth aspect, "the second gift object is arranged for each type of the first gift object and for each type of the first gift object". is determined based on the number of assignments and displayed on the video.

A video processing method according to a tenth aspect is the method according to the ninth aspect, in which "the second gift object is further determined based on the contact time between the first gift objects and displayed on the video." It is.

In the video processing method according to the eleventh aspect, in the ninth aspect or the tenth aspect, "the second gift object is further determined based on the collision speed when the first gift objects contact each other." and displayed on the video.

A video processing method according to a twelfth aspect includes, in any one of the first to eleventh aspects, "the second displaying step includes, when it is determined that the avatar object has performed the predetermined action, The first gift object for which the predetermined action has been performed is deleted from the video.

A video processing method according to a thirteenth aspect includes, in any one of the first to eleventh aspects, "the second displaying step includes, when it is determined that the avatar object has performed the predetermined action, the first gift object for which the predetermined action has been performed is changed into the second gift object, and the second gift object is displayed on the video.

A video processing method according to a fourteenth aspect includes, in any one of the first to eleventh aspects, "the second displaying step includes, when it is determined that the avatar object has performed the predetermined action, Among the plurality of first gift objects for which the predetermined action has been performed, at least one of the first gift objects is erased from the video, and at least one other of the first gift objects is erased from the second gift object. ``transform into a gift object''.

A video processing method according to a fifteenth aspect includes, in any one of the twelfth to fourteenth aspects, "the second gift object is deleted from the video or The first gift object is displayed in a different position from the position in the video before the first gift object was changed.

A video processing method according to a sixteenth aspect includes, in any one of the first to fifteenth aspects, "the second display step includes the avatar object starting the predetermined action, and the second gift object displaying, on the video, an effect object indicating a state in the middle of being displayed on the video.

The video processing method according to the seventeenth aspect is characterized in that in any one of the first to sixteenth aspects, “the generation step, the first display step, and the second display step are performed on a terminal device of the distribution user. "is carried out by."

In the video processing method according to the eighteenth aspect, in the seventeenth aspect, "the receiving step is performed by a server device connected to the terminal device of the viewing user and/or the terminal device of the distribution user via a communication line. and that the distribution user's terminal device determines that the avatar object has performed a predetermined action on the first gift object displayed on the video.

A video processing method according to a nineteenth aspect is a video processing method according to any one of the first to sixteenth aspects, in which “the generation step, the reception step, the first display step, and the second display step include the viewing and viewing steps. This is executed by a user's terminal device and/or a server device connected to the distribution user's terminal device via a communication line.

The video processing method according to the 20th aspect includes, in the 19th aspect, "It is determined that the avatar object has performed a predetermined action on the first gift object displayed on the video," "determined by the device."

A video processing method according to a twenty-first aspect is characterized in that in any one of the first to sixteenth aspects, "the generation step, the first display step, and the second display step are performed on the viewing user's terminal device." "is carried out by."

In the video processing method according to the twenty-second aspect, in the twenty-first aspect, “the receiving step is performed by a server device connected to the terminal device of the viewing user and/or the terminal device of the distribution user via a communication line. and that the distribution user's terminal device determines that the avatar object has performed a predetermined action on the first gift object displayed on the video.

A video processing method according to a twenty-third aspect is provided in any one of the first aspect to the twenty-second aspect, in which "the processor is a central processing unit (CPU), a microprocessor, or a graphics processing unit (GPU)." It is something.

A video processing method according to a twenty-fourth aspect is one in which "the communication line includes the Internet" in any of the first to twenty-third aspects.

The server device according to the twenty-fifth aspect is characterized in that the server device includes “one or more processors, the processor generates a video including an animation of an avatar object of the distribution user based on the distribution user's actions, and the viewing and viewing of the video is performed. A request signal generated based on a user's operation and requesting that a first gift object be given to the avatar object is received from the viewing user's terminal device via a communication line, and the request signal is applied to the request signal. The first gift object given to the avatar object based on the distribution user is displayed on the video, and one or more of the first gift objects displayed on the video are displayed on the video based on the distribution user's actions. When the terminal device of the distribution server determines that the avatar object has performed a predetermined action on the gift object, a second gift object different from the first gift object on which the predetermined action has been performed is sent. "displayed on the video."

The server device according to the twenty-sixth aspect includes “one or more processors, and the processor generates information for generating a moving image including an animation of an avatar object of the distribution user based on the operation of the distribution user. to the terminal device of the distribution user that generates the video or the terminal device of the viewing user, the video is generated based on the operation of the viewing user who views the video, and a first gift object is given to the avatar object. A request signal for requesting the avatar object is received from the viewing user's terminal device via a communication line, and information regarding the first gift object given to the avatar object based on the request signal is transmitted to the first gift object. to the distribution user's terminal device or the viewing user's terminal device to display the gift object on the video, and based on the distribution user's operation, one or more of the first gift objects displayed on the video a second gift object different from the first gift object on which the predetermined action was performed when the terminal device of the distribution server determines that the avatar object has performed the predetermined action on the gift object; information regarding the second gift object to the terminal device of the distribution user or the terminal device of the viewing user who causes the second gift object to be displayed on the video.

A server device according to a twenty-seventh aspect is one in which "the processor is a central processing unit (CPU), a microprocessor, or a graphics processing unit (GPU)" in the twenty-fifth aspect or the twenty-sixth aspect.

A server device according to a twenty-eighth aspect is one in which “the communication line includes the Internet” in any one of the twenty-fifth aspect to the twenty-seventh aspect.

The computer program according to the twenty-ninth aspect is ``generated by one or more processors to generate a video including an animation of an avatar object of the distribution user based on the distribution user's actions, and a viewing user who views the video. A request signal generated based on the operation of the avatar object and requesting that a first gift object be given to the avatar object is received from the server device or the viewing user's terminal device via a communication line, and the request signal is generated based on the operation of The first gift object given to the avatar object based on the signal is displayed on the video, and the one or more gift objects displayed on the video are displayed on the video based on the distribution user's action. If it is determined that the avatar object has performed a predetermined action on a first gift object, a second gift object different from the first gift object for which the predetermined action has been performed is displayed on the video. "to cause the processor to function in such a way as to cause the processor to function in a manner that

A computer program according to a thirtieth aspect provides: ``When executed by one or more processors, information for generating a moving image including an animation of an avatar object of the distribution user based on the movement of the distribution user is generated. a request signal that is transmitted to a generating server device or a viewing user's terminal device, is generated based on an operation of the viewing user who views the video, and requests that a first gift object be given to the avatar object; is received from the server device or the viewing user's terminal device via a communication line, and information regarding the first gift object given to the avatar object based on the request signal is transmitted to the first gift object. transmitting the object to the server device that displays the object on the video or the viewing user's terminal device, and transmitting the object to the one or more first gift objects displayed on the video based on the operation of the distribution user. If it is determined that the avatar object has executed a predetermined action, information regarding a second gift object different from the first gift object on which the predetermined action has been performed is transmitted to the second gift object. The processor functions to transmit the information to the server device or the viewing user's terminal device to be displayed on the video.

A computer program according to a 31st aspect is one in which "the processor is a central processing unit (CPU), a microprocessor, or a graphics processing unit (GPU)" in the 29th aspect or the 30th aspect.

A computer program according to a thirty-second aspect is one in which "the communication line includes the Internet" in any of the twenty-ninth to thirty-first aspects.

A computer program according to a thirty-third aspect provides: ``In order to generate a moving image including an animation of an avatar object of the distribution user based on an action of the distribution user by being executed by one or more processors, the computer program generates information regarding the video. A request signal received from a server device or a terminal device of the distribution user, generated based on an operation of a viewing user who views the video, and requesting to give a first gift object to the avatar object, The server sends information regarding the first gift object via a communication line and displays the first gift object given to the avatar object based on the request signal on the video. received from the device or the distribution user's terminal device, and the avatar object performed a predetermined action on the one or more first gift objects displayed on the video based on the distribution user's action. In order to display on the video a second gift object that is different from the first gift object for which the predetermined action has been performed, "The processor is made to function so as to receive information regarding an object from the server device or the distribution user's terminal device."

The computer program according to the thirty-fourth aspect provides the following method: ``By being executed by one or more processors, a moving image including an animation of an avatar object of the distribution user based on the distribution user's action is generated by a server device or a terminal device of the distribution user. transmitting, via a communication line, a request signal generated based on an operation of a viewing user viewing the video and requesting that a first gift object be given to the avatar object; receiving the video displaying the first gift object given to the avatar object based on the request signal from the server device or the terminal device of the distribution user; , when the terminal device of the distribution server determines that the avatar object has performed a predetermined action on one or more of the first gift objects displayed on the video, the predetermined action is executed. the processor is configured to receive, from the server device or the distribution user's terminal device, the moving image displaying a second gift object different from the first gift object.

A computer program according to a thirty-fifth aspect is one in which "the processor is a central processing unit (CPU), a microprocessor, or a graphics processing unit (GPU)" in the thirty-third aspect or the thirty-fourth aspect.

A computer program according to a thirty-sixth aspect is one in which "the communication line includes the Internet" in any one of the thirty-third to thirty-fifth aspects.

A distribution user's terminal device according to a thirty-seventh aspect “generates a video including an animation of the distribution user's avatar object based on the distribution user's actions, and generates the video based on the operation of the viewing user who views the video,” A request signal requesting that a first gift object be given to the avatar object is received from the server device or the viewing user's terminal device via a communication line, and the request signal is sent to the avatar object based on the request signal. The first gift object given by the user is displayed on the video, and the avatar is assigned to one or more of the first gift objects displayed on the video based on the distribution user's actions. When it is determined that the object has performed a predetermined action, a second gift object different from the first gift object for which the predetermined action has been performed is displayed on the video.

The distribution user's terminal device according to the thirty-eighth aspect provides information for generating a video including an animation of the distribution user's avatar object based on the distribution user's actions to the server device that generates the video or the viewing user's terminal. A request signal generated based on the operation of the viewing user who views the video and requesting that the first gift object be given to the avatar object is sent to the server device or the viewing user. Displaying the first gift object on the video, the information regarding the first gift object received from the terminal device via the communication line and given to the avatar object based on the request signal. transmitted to a server device or a terminal device of the viewing user, and the avatar object executes a predetermined action on the one or more first gift objects displayed on the video based on the action of the distribution user. If it is determined that the predetermined action has been performed, the server device displays information regarding a second gift object different from the first gift object on which the predetermined action has been performed, or displays the second gift object on the video; "to be transmitted to the viewing user's terminal device."

The viewing user's terminal device according to the thirty-ninth aspect is configured to generate a video that includes an animation of an avatar object of the distribution user based on the distribution user's actions by being executed by one or more processors. a request that is generated based on an operation of a viewing user who views the video, and requests that a first gift object be given to the avatar object; information regarding the first gift object in order to transmit a signal via a communication line and display the first gift object given to the avatar object on the video based on the request signal; is received from the server device or the distribution user's terminal device, and the avatar object performs a predetermined action on the one or more first gift objects displayed on the video based on the distribution user's action. If the terminal device of the distribution server determines that the predetermined action has been performed, the second gift object, which is different from the first gift object for which the predetermined action has been performed, is displayed on the video. 2 from the server device or the distribution user's terminal device.

The viewing user's terminal device according to the fortieth aspect “receives a video including an animation of the distribution user's avatar object based on the distribution user's actions from the server device or the distribution user's terminal device, and views the video. A request signal that is generated based on an operation of a viewing user and requests the avatar object to be given a first gift object is transmitted via a communication line, and the request signal is sent to the avatar object based on the request signal. The video in which the first gift object given to the user is displayed is received from the server device or the terminal device of the distribution user, and the first gift object displayed on the video is received based on the operation of the distribution user. Alternatively, when the terminal device of the distribution server determines that the avatar object has performed a predetermined action on a plurality of the first gift objects, the first gift object for which the predetermined action has been performed is the video displaying a different second gift object is received from the server device or the distribution user's terminal device.

7. Fields to which the technology disclosed in the present application is applicable The technology disclosed in the present application can be applied, for example, to the following fields.
(1) Application services that deliver live videos featuring avatar objects (2) Application services that allow communication using text and avatar objects (chat applications, messengers, email applications, etc.)

1 Video distribution system 10 (10A to 10D) Terminal device 20 Server device 20A Gift API server device 20B Information processing server device 20C Distribution server device 20D Comment API server device 100 Video generation section 110 Display section 120 Judgment section 130 Storage section 140 User interface Face section 150 Gift storage section 160 Communication section 400 Avatar object 410A, 410B First gift object 411A, 411B First collision area 420, 430 Second gift object 500 Effect object

Claims (55)

A video processing method performed by one or more processors executing computer-readable instructions, the method comprising:
a generation step of generating a video including an animation of an avatar object of the distribution user based on the distribution user's actions;
A request signal generated based on an operation of a viewing user who views the video and requesting that a first gift object be given to the avatar object is received from the viewing user's terminal device via a communication line. a receiving process,
a first display step of displaying the first gift object given to the avatar object based on the request signal on the video;
If it is determined that the avatar object has performed a predetermined action on one or more of the first gift objects displayed on the video based on the distribution user's action, at least the predetermined action , a second display step of displaying on the video a second gift object that is associated in advance with a combination that includes at least the type of the first gift object for which the predetermined action was performed and first information ;
including;
The first information is
If the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user, the predetermined action is the specific position;
When the predetermined action is an action in which the avatar object contacts at least one of the first gift objects with another first gift object based on the action of the distribution user, the first gift object A video processing method that is any one of a contact time between the first gift objects, a collision speed when the first gift objects contact each other, and a relative position when the first gift objects contact each other. The video processing method according to claim 1, wherein the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user. The combination includes at least the specific location and the type of the first gift object,
The moving image processing method according to claim 2, wherein the first information is the specific position. The predetermined action is an action in which the avatar object causes at least one of the first gift objects to come into contact with another first gift object, based on an action of the distribution user. Video processing method. The combination includes at least the type of the first gift object and the contact time between the first gift objects,
The moving image processing method according to claim 4, wherein the first information is the contact time. The combination includes at least the type of the first gift object and the collision speed when the first gift objects contact each other,
The moving image processing method according to claim 4, wherein the first information is the collision speed. The combination includes at least the type of the first gift object and the relative position when the first gift objects come into contact with each other,
The moving image processing method according to claim 4, wherein the first information is the relative position. A first collision area is set in each of the first gift objects,
A second collision area is set on the hand of the avatar object,
If it is determined that the first collision area and the second collision area overlap at least in part in the video, and the finger of the avatar object is bent based on the movement of the distribution user. If determined, the first gift object is recognized as being held by the hand of the avatar object;
The video processing method according to any one of claims 1 to 7, wherein the first gift object is movable when held by the hand of the avatar object. The video processing method according to claim 8, wherein the predetermined action is executable when it is determined that the avatar object is holding the first gift object. The video processing method according to any one of claims 1 to 9 , further comprising a comment information display step of displaying comment information on the video to guide execution of the predetermined action. The video processing method according to any one of claims 1 to 10 , wherein the second gift object displayed on the video is associated in advance with the type of the predetermined action. The second gift object displayed on the video is associated in advance with the number of the first gift objects for which the predetermined action has been performed. Video processing method. The video processing method according to claim 10 , further comprising a comment information receiving step of receiving the comment information from the viewing user's terminal device based on the viewing user's operation. The video processing method according to claim 10 , further comprising a comment information creation step of creating the comment information based on the request signal received in the reception step or the determination in the second display step. In the second display step, when it is determined that the avatar object has performed the predetermined action, the first gift object for which the predetermined action has been performed is deleted from the video, and then the first gift object is displayed. The video processing method according to any one of claims 1 to 14 , wherein the gift object No. 2 is displayed on the video. In the second display step, when it is determined that the avatar object has performed the predetermined action, the first gift object on which the predetermined action has been performed is changed into the second gift object over time. The video processing method according to any one of claims 1 to 14 , wherein the second gift object is displayed on the video. In the second display step, when it is determined that the avatar object has performed the predetermined action, at least one of the first gift objects for which the predetermined action has been performed is displayed. 15. The video processing method according to any one of claims 1 to 14 , wherein the first gift object is deleted from the video, and at least one other of the first gift objects is changed to the second gift object over time. . The video processing method according to any one of claims 1 to 17 , wherein the processor is a central processing unit (CPU), a microprocessor, or a graphics processing unit (GPU). The video processing method according to any one of claims 1 to 18 , wherein the communication line includes the Internet. comprising one or more processors,
The processor is
generating a video including an animation of an avatar object of the distribution user based on the distribution user's actions;
A request signal generated based on an operation of a viewing user who views the video and requesting that a first gift object be given to the avatar object is received from the viewing user's terminal device via a communication line. death,
displaying the first gift object given to the avatar object based on the request signal on the video;
When the terminal device of the distribution user determines that the avatar object has performed a predetermined action with respect to the one or more first gift objects displayed on the video, based on the distribution user's actions; , displaying on the video a second gift object that is associated in advance with a combination that includes at least the predetermined action, the type of the first gift object on which the predetermined action was performed , and first information. let me ,
The first information is
If the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user, the predetermined action is the specific position;
When the predetermined action is an action in which the avatar object contacts at least one of the first gift objects with another first gift object based on the action of the distribution user, the first gift object contact time between the two, a collision speed when the first gift objects contact each other, and a relative position when the first gift objects contact each other,
server equipment. comprising one or more processors,
The processor is
transmitting information for generating a video including an animation of an avatar object of the distribution user based on the distribution user's actions to a terminal device of the distribution user that generates the video or a terminal device of a viewing user;
A request signal generated based on the operation of the viewing user who views the video and requesting that a first gift object be given to the avatar object is transmitted from the viewing user's terminal device via a communication line. receive,
The distribution user's terminal device or the viewing user's terminal displays information regarding the first gift object given to the avatar object based on the request signal on the video. send to the device,
When the terminal device of the distribution user determines that the avatar object has performed a predetermined action with respect to the one or more first gift objects displayed on the video, based on the distribution user's actions; information about a second gift object associated in advance with a combination including at least the predetermined action, the type of the first gift object on which the predetermined action was executed, and first information; 2 to the terminal device of the distribution user or the terminal device of the viewing user to display the gift object on the video ,
The first information is
If the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user, the predetermined action is the specific position;
When the predetermined action is an action in which the avatar object contacts at least one of the first gift objects with another first gift object based on the action of the distribution user, the first gift object contact time between the two, a collision speed when the first gift objects contact each other, and a relative position when the first gift objects contact each other,
server equipment. The server device according to claim 20 or 21, wherein the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user. The combination includes at least the specific location and the type of the first gift object,
The server device according to claim 22, wherein the first information is the specific location. According to claim 20 or 21, the predetermined action is an action in which the avatar object causes at least one of the first gift objects to come into contact with another first gift object based on the operation of the distribution user. The server device described. The combination includes at least the type of the first gift object and the contact time between the first gift objects,
The server device according to claim 24, wherein the first information is the contact time. The combination includes at least the type of the first gift object and the collision speed when the first gift objects contact each other,
The server device according to claim 24, wherein the first information is the collision speed. The combination includes at least the type of the first gift object and the relative position when the first gift objects come into contact with each other,
The server device according to claim 24, wherein the first information is the relative position. A first collision area is set in each of the first gift objects,
A second collision area is set on the hand of the avatar object,
In the case where it is determined by the distribution user's terminal device that the first collision area and the second collision area overlap at least in part within the video, and based on the distribution user's movement, the avatar object If the distribution user's terminal device determines that the finger of the avatar object is bent, the first gift object is recognized as being held by the hand of the avatar object;
The server device according to any one of claims 20 to 27, wherein the first gift object is movable when held by the hand of the avatar object. The server device according to claim 28, wherein the predetermined action is executable when the distribution user's terminal device determines that the avatar object is holding the first gift object. The server device according to any one of claims 20 to 29 , wherein the processor is a central processing unit (CPU), a microprocessor, or a graphics processing unit (GPU). The server device according to any one of claims 20 to 30 , wherein the communication line includes the Internet. By being executed by one or more processors,
generating a video including an animation of an avatar object of the distribution user based on the distribution user's actions;
A request signal generated based on the operation of the viewing user who views the video and requesting that the first gift object be given to the avatar object is transmitted from the server device or the terminal device of the viewing user to the communication line. received through,
displaying the first gift object given to the avatar object based on the request signal on the video;
If it is determined that the avatar object has performed a predetermined action on one or more of the first gift objects displayed on the video based on the distribution user's action, at least the predetermined action , displaying on the video a second gift object that is associated in advance with a combination that includes at least the type of the first gift object for which the predetermined action was performed and first information ;
The first information is
If the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user, the predetermined action is the specific position;
When the predetermined action is an action in which the avatar object contacts at least one of the first gift objects with another first gift object based on the action of the distribution user, the first gift object contact time between the two, a collision speed when the first gift objects contact each other, and a relative position when the first gift objects contact each other,
A computer program that causes the processor to function as described above. By being executed by one or more processors,
transmitting information for generating a video including an animation of an avatar object of the distribution user based on the distribution user's actions to a server device that generates the video or a terminal device of a viewing user;
A request signal generated based on an operation of the viewing user who views the video and requesting that a first gift object be given to the avatar object is communicated from the server device or the viewing user's terminal device. received via the line,
transmitting information regarding the first gift object given to the avatar object based on the request signal to the server device or the viewing user's terminal device that displays the first gift object on the video; death,
If it is determined that the avatar object has performed a predetermined action on one or more of the first gift objects displayed on the video based on the distribution user's action, at least the predetermined action , information regarding a second gift object previously associated with a combination including at least the type of the first gift object for which the predetermined action was performed and first information; to the server device or the viewing user's terminal device to be displayed on the screen;
The first information is
If the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user, the predetermined action is the specific position;
When the predetermined action is an action in which the avatar object contacts at least one of the first gift objects with another first gift object based on the action of the distribution user, the first gift object contact time between the two, a collision speed when the first gift objects contact each other, and a relative position when the first gift objects contact each other,
A computer program that causes the processor to function as described above. 34. The computer program according to claim 32, wherein the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the distribution user's action. The combination includes at least the specific location and the type of the first gift object,
The computer program product according to claim 34, wherein the first information is the specific location. According to claim 32 or 33, the predetermined action is an action in which the avatar object causes at least one of the first gift objects to contact another first gift object based on the operation of the distribution user. The computer program described. The combination includes at least the type of the first gift object and the contact time between the first gift objects,
37. The computer program product according to claim 36, wherein the first information is the contact time. The combination includes at least the type of the first gift object and the collision speed when the first gift objects contact each other,
37. The computer program product according to claim 36, wherein the first information is the collision speed. The combination includes at least the type of the first gift object and the relative position when the first gift objects come into contact with each other,
37. The computer program product according to claim 36, wherein the first information is the relative position. A first collision area is set in each of the first gift objects,
A second collision area is set on the hand of the avatar object,
If it is determined that the first collision area and the second collision area overlap at least in part in the video, and the finger of the avatar object is bent based on the movement of the distribution user. If determined, the first gift object is recognized as being held by the hand of the avatar object;
40. A computer program product according to any one of claims 32 to 39, wherein the first gift object is movable when held by the hand of the avatar object. The computer program product according to claim 40, wherein the predetermined action is executable when it is determined that the avatar object is holding the first gift object. 42. A computer program product according to any one of claims 32 to 41 , wherein the processor is a central processing unit (CPU), a microprocessor or a graphics processing unit (GPU). 43. A computer program product according to any one of claims 32 to 42 , wherein the communication line includes the Internet. By being executed by one or more processors,
In order to generate a video including an animation of the distribution user's avatar object based on the distribution user's actions, receiving information regarding the video from a server device or the distribution user's terminal device,
transmitting, via a communication line, a request signal that is generated based on an operation of a viewing user who views the video and requests that a first gift object be given to the avatar object;
In order to display the first gift object given to the avatar object based on the request signal on the video, information regarding the first gift object is transmitted to the server device or the distribution user's terminal device. received from
When the terminal device of the distribution user determines that the avatar object has performed a predetermined action with respect to the one or more first gift objects displayed on the video, based on the distribution user's actions; , displaying on the video a second gift object that is associated in advance with a combination that includes at least the predetermined action, the type of the first gift object on which the predetermined action was performed , and first information. receiving information regarding the second gift object from the server device or the distribution user's terminal device ,
The first information is
If the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user, the predetermined action is the specific position;
When the predetermined action is an action in which the avatar object contacts at least one of the first gift objects with another first gift object based on the action of the distribution user, the first gift object contact time between the two, a collision speed when the first gift objects contact each other, and a relative position when the first gift objects contact each other,
A computer program that causes the processor to function as described above. By being executed by one or more processors,
receiving a video including an animation of an avatar object of the distribution user based on the distribution user's actions from a server device or a terminal device of the distribution user;
transmitting, via a communication line, a request signal that is generated based on an operation of a viewing user who views the video and requests that a first gift object be given to the avatar object;
receiving the video displaying the first gift object given to the avatar object based on the request signal from the server device or the distribution user's terminal device;
When the terminal device of the distribution user determines that the avatar object has performed a predetermined action with respect to the one or more first gift objects displayed on the video, based on the distribution user's actions; , the video displaying a second gift object associated in advance with a combination including at least the predetermined action, the type of the first gift object on which the predetermined action was performed , and first information. from the server device or the distribution user's terminal device ,
The first information is
If the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the action of the distribution user, the predetermined action is the specific position;
When the predetermined action is an action in which the avatar object contacts at least one of the first gift objects with another first gift object based on the action of the distribution user, the first gift object contact time between the two, a collision speed when the first gift objects contact each other, and a relative position when the first gift objects contact each other,
A computer program that causes the processor to function as described above. 46. The computer program according to claim 44, wherein the predetermined action is an action in which the avatar object moves the first gift object to a specific position based on the distribution user's action. The combination includes at least the specific location and the type of the first gift object,
47. The computer program product according to claim 46, wherein the first information is the specific location. According to claim 44 or 45, the predetermined action is an action in which the avatar object causes at least one of the first gift objects to come into contact with another first gift object, based on the operation of the distribution user. The computer program described. The combination includes at least the type of the first gift object and the contact time between the first gift objects,
49. The computer program product according to claim 48, wherein the first information is the contact time. The combination includes at least the type of the first gift object and the collision speed when the first gift objects contact each other,
49. The computer program product according to claim 48, wherein the first information is the collision speed. The combination includes at least the type of the first gift object and the relative position when the first gift objects come into contact with each other,
49. The computer program product according to claim 48, wherein the first information is the relative position. A first collision area is set in each of the first gift objects,
A second collision area is set on the hand of the avatar object,
In the case where it is determined by the distribution user's terminal device that the first collision area and the second collision area overlap at least in part within the video, and based on the distribution user's movement, the avatar object If the distribution user's terminal device determines that the finger of the avatar object is bent, the first gift object is recognized as being held by the hand of the avatar object;
52. A computer program product according to any one of claims 44 to 51, wherein the first gift object is movable when held by the hand of the avatar object. 53. The computer program product according to claim 52, wherein the predetermined action is executable when the distribution user's terminal device determines that the avatar object is holding the first gift object. 54. A computer program product according to any one of claims 44 to 53 , wherein the processor is a central processing unit (CPU), a microprocessor or a graphics processing unit (GPU). 55. A computer program product according to any one of claims 44 to 54 , wherein the communication line includes the Internet.

Images