JP2023082903A - Information processing device, method, and program - Google Patents

Abstract

To enable billing to be performed on a billing target desired by a viewer without impairing the viewer's sense of immersion.SOLUTION: In an information processing device 10 used for viewing a video distributed online, a control unit 11 calculates a viewer's satisfaction level on the basis of the viewer's health care information during viewing of the online distribution. The control unit 11 then identifies a billing target according to the viewer's satisfaction level on the basis of at least one of a direction and position of a viewer's line of sight during viewing of the online distribution and executes control of superimposing the billing target on the video distributed online to perform display for billing processing on the identified billing target.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing apparatus, method, and program used for viewing video distributed online.

2. Description of the Related Art In recent years, streaming distribution in VR (Virtual Reality) has been carried out so that a user can experience an image captured by a camera without actually being there. As a billing method in streaming distribution, there is a method in which viewers tip money to the distributor, and an interactive viewing experience is possible in which the distributor responds to it.
In Patent Document 1, an emotion is recognized based on at least one of the viewer information and the performer information, and an effect is added based on the emotion for the purpose of further enhancing the sense of unity between the performer and the viewer. is disclosed to control the

WO2016/088566

When there are a plurality of objects to be tipped in the streamed video, the viewer needs to specify the object before tipping. However, there is a problem that the viewer's sense of immersion is spoiled if an operation for specifying the target is required. Patent Document 1 does not disclose means for solving such problems.

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and it is an object of the present invention to enable a viewer to charge for a desired charging target without impairing the viewer's sense of immersion.

An information processing apparatus according to the present invention is an information processing apparatus used for viewing videos distributed online, and comprises an obtaining means for obtaining state information relating to a user's state, and an online distribution obtained by the obtaining means. index calculation means for calculating an index representing the user's emotion based on the state information of the user during viewing; line-of-sight detection means for detecting at least one of the user's line-of-sight direction and position; and the index calculation means. In accordance with the index calculated in step 1, a billing target is specified based on at least one of the direction and position of the line of sight of the user who is viewing the online distribution detected by the line of sight detection means, and the video distributed online is identified. and a control means for superimposing and executing a display for billing processing for the specified billing target.

ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to charge for the charge object which a viewer desires, without impairing a viewer's sense of immersion.

1 is a block diagram showing the configuration of a video distribution system according to an embodiment; FIG. FIG. 4 is a diagram showing a display example of the display unit of the VR device according to the embodiment; FIG. 4 is a diagram showing a display example of the display unit of the VR device according to the embodiment; 4 is a flowchart showing processing executed by the VR device according to the embodiment; 4 is a flowchart showing processing executed by the VR device according to the embodiment; FIG. 6 is a flow chart showing the details of charge button display processing in the flow chart of FIG. 5 ; FIG.

Preferred embodiments of the present invention will now be described with reference to the accompanying drawings.
FIG. 1 is a block diagram showing the configuration of a video distribution system according to an embodiment. The video distribution system includes a VR device 10 , a distribution server 20 , a controller 30 , and a network 40 such as the Internet that connects the VR device 10 and the distribution server 20 . Although only one VR device 10 is illustrated, a plurality of VR devices 10 may be configured to communicate with the distribution server 20 .

The VR device 10 includes a control section 11 , a line-of-sight detection section 12 , an operation section 13 , a display section 14 , a ROM 15 , a RAM 16 , a communication section 17 and an internal bus 18 . The VR device 10 may be equipped with a function for realizing AR (Augmented Reality). In this embodiment, the VR device 10 functions as an information processing device to which the present invention is applied.
The control unit 11 is configured by processors such as one or more CPUs (Central Processing Units), MPUs (Micro Processor Units), GPUs (Graphics Processing Units), etc., and controls the VR device 10 . In this embodiment, the control unit 11 functions as acquisition means, index calculation means, control means, setting means, and execution means of the present invention by executing programs held in the ROM 15, which will be described later.

The line-of-sight detection unit 12 detects, under the control of the control unit 11, the line of sight and blinks of the user viewing the image displayed on the display unit 14 by, for example, a method called a corneal reflection method. The corneal reflection method is a method that detects the direction and position of the line of sight from the positional relationship between the infrared light reflected by the user's eyeball (especially the reflected light reflected by the cornea) and the pupil of the eyeball. be. Various methods for identifying the direction and position of the line of sight can be used without being limited to the corneal reflection method. Also, in order to operate the VR device 10, the line of sight of the user may be detected. In this embodiment, the line-of-sight detection unit 12 functions as line-of-sight detection means of the present invention.

The operation unit 13 receives user operations and notifies the control unit 11 of input information. The operation unit 13 is composed of, for example, buttons included in the VR device 10 . Further, the line-of-sight information of the user for the image displayed on the display unit 14 is detected by the line-of-sight detection unit 12, and the input information from the operation unit 34 of the controller 30 is acquired via the communication unit 17. may
The display unit 14 is composed of a liquid crystal panel, an LED, and the like, and notifies the user of various messages such as the operating state of the device, and displays video content distributed from the distribution server 20 .

A ROM (Read Only Memory) 15 is a rewritable non-volatile memory, and holds programs such as an OS (Operation System) and programs supplied from an external device. The programs held by the ROM 15 include a program for acquiring user healthcare information from the controller 30, and a program for identifying a billing target in an image and displaying a billing process for the billing target, as will be described later. .
A RAM (Random Access Memory) 16 is a volatile memory that temporarily stores programs, photographed image data, communication data, and the like.

The communication unit 17 is a communication unit for realizing wireless communication. The communication unit 17 includes an antenna for wireless communication and a communication controller for processing wireless signals. Realize wireless communication according to system standards such as 5G. The control unit 11 controls the communication unit 17 to realize communication with the distribution server 20 via the network 40 . The control unit 11 also controls the communication unit 17 to directly communicate with the controller 30 . Although the communication unit 17 performs wireless communication, wired communication may be used.
An internal bus 18 communicably connects the units 11 to 17 .

Distribution server 20 includes control unit 21 , ROM 22 , RAM 23 , communication unit 24 , storage unit 25 and internal bus 26 . The distribution server 20 may be realized by a single computer device, or may be realized by distributing each function to a plurality of computer devices as necessary. When a plurality of computer devices are configured, they are connected via a LAN (Local Area Network) or the like so as to be able to communicate with each other.
The control unit 21 is configured by, for example, a CPU, and controls the distribution server 20 .
The ROM 22 is a rewritable non-volatile memory and holds programs such as an OS and programs supplied from external devices.
The RAM 23 is a volatile memory and temporarily stores programs and data.

The communication unit 24 is a communication unit for realizing wired communication. The communication unit 24 is composed of hardware for wired communication and a communication controller for processing wired signals, and realizes wired communication according to IEEE 802.3 series standards, for example.
The storage unit 25 is an HDD (hard disk drive) fixedly installed in the distribution server 20, an SSD (Solid State Drive) composed of flash memory, a hybrid drive or memory card using both a hard disk and a flash memory, or the like. . The storage unit 25 holds distributor information such as distributor account information and viewer information such as viewer account information. The viewer information includes information regarding payment when tipping.
An internal bus 26 communicably connects the units 21 to 25 .

The controller 30 is a wearable controller worn by a user and includes a control unit 31 , a communication unit 32 , a healthcare information acquisition unit 33 , an operation unit 34 and an internal bus 35 .
The control unit 31 is configured by, for example, an MPU, and controls the controller 30 .
The communication unit 32 is a communication unit for realizing short-range wireless communication for direct communication with the VR device 10 such as Bluetooth. In some cases, the controller 30 is controlled by the communication unit 32 communicating only with the communication unit 17 of the VR device 10 and receiving control information of the VR device 10 .
The healthcare information acquisition unit 33 is, for example, an optical heart rate sensor, and measures the user's heart rate. Health care information is information generally used for health management, and is not limited to heart rate, and may be, for example, pulse. Health care information is an example of state information relating to a user's state as referred to in the present invention.
The operation unit 34 includes, for example, a cross key and buttons, receives user operations, and notifies the control unit 11 of input information.
An internal bus 35 communicably connects the units 31 to 34 .
In this embodiment, the configuration in which the VR device 10 and the controller 30 are separated has been described, but they may be integrated.

FIG. 2 shows a display example of the display unit 14 of the VR device 10. As shown in FIG.
As shown in FIG. 2 , a video 200 distributed from the distribution server 20 is displayed on the display unit 14 . Two performers 201 and 202 are displayed in the image 200, and the viewer can specify the performer and give money to the specified performer. In this case, in the image 200, a billing button 203 for instructing the execution of billing processing is superimposed on the side of the performer 201 specified as a billing object (a tipping object) as will be described later. The billing button 203 is added with a billing amount determined as described later.
FIG. 3 shows another display example of the display unit 14 of the VR device 10. As shown in FIG. It is the same that the charge button is superimposed on the side of the performer 201 specified as the charge target, but the charge button 301 is added with the information of the performer specified as the charge target so that the charge target can be confirmed. It is The performer's information may be, for example, the name and icon of the performer, an image cut out from the video 200, and the like.

The operation of the VR device 10 will be described below.
First, with reference to FIG. 4, a process executed by the VR device 10 before a user (hereinafter referred to as a viewer) views an online-delivered video will be described. FIG. 4 is a flow chart showing processing executed by the VR device 10 .
In step S<b>401 , the control unit 11 communicates with the distribution server 20 via the communication unit 17 to acquire distributor information and store it in the RAM 16 . The distributor information includes the distributor's account information and streaming information necessary for viewing the video. Further, the distributor information includes billing target information in the video distributed by the distributor. For example, when a video includes a plurality of performers (either the distributor himself or a different person from the distributor), information on whether or not to be billed is included. Information such as performer names and icons may also be included.

In step S<b>402 , the control unit 11 communicates with the distribution server 20 via the communication unit 17 to acquire viewer information and store it in the RAM 16 . Although it is assumed that the viewer information is obtained from the distribution server 20, the viewer information held in the VR device 10 may be obtained. Viewer information includes viewer account information. The viewer information includes information regarding payment when tipping.

In step S403, the control unit 11 sets the average heart rate of the viewer. Specifically, the control unit 11 communicates with the controller 30 via the communication unit 17 and transmits a control signal for acquiring the average heart rate of the viewer. The controller 30 receives control information from the VR device 10 via the communication unit 32 . The control unit 31 of the controller 30 acquires the viewer's heart rate from the healthcare information acquisition unit 33 based on the control information from the VR device 10 . The control unit 31 acquires a plurality of heartbeats within a predetermined period of time and calculates the average value. This average value is, for example, an average value of heart rate values obtained a predetermined number of times within a predetermined period of time. The control unit 31 transmits the average heart rate of the viewer to the VR device 10 via the communication unit 32 . The control unit 11 of the VR device 10 receives the heart rate average value of the viewer from the controller 30 via the communication unit 17 and stores it in the RAM 16 . Note that the average heart rate of the viewer may be calculated on the VR device 10 side by receiving the heart rate from the controller 30 via the communication unit 17 within a predetermined period of time. good.

In step S404, the control unit 11 sets a threshold for satisfaction. As will be described later, based on the viewer's heart rate obtained from the controller 30, an index called "satisfaction level" representing the user's emotion is calculated, and a threshold value for this is set. For example, the threshold is a value obtained by multiplying the average value of the heart rate set in step S403 by a predetermined coefficient. In this way, the threshold for satisfaction is set based on the user's heart rate before viewing the online distribution.
Although the flow chart has been described assuming that it is executed immediately before the viewer views the video distributed online, for example, the processing of steps S403 and S404 can be performed at any time during initial setting or according to the operation of the operation unit 13. It may be executed in time.

Next, with reference to FIGS. 5 and 6, processing executed by the VR device 10 when a viewer views video distributed online will be described. FIG. 5 is a flow chart showing processing executed by the VR device 10 .
In step S501, the control unit 11 acquires the viewer's healthcare information. Specifically, the control unit 11 communicates with the controller 30 via the communication unit 17 and transmits a control signal for acquiring the heart rate of the viewer. The controller 30 receives control information from the VR device 10 via the communication unit 32 . The control unit 31 of the controller 30 acquires the viewer's heart rate from the healthcare information acquisition unit 33 based on the control information from the VR device 10 and transmits it to the VR device 10 via the communication unit 32 . It should be noted that the heart rate of the viewer at each time may be obtained, or the average value of the heart rate for each fixed period may be obtained.

In step S502, the control unit 11 calculates an index called "satisfaction level" representing the viewer's emotion based on the healthcare information acquired in step S501. The degree of satisfaction is a numerical representation of the heightened emotions of the viewer. For example, a value obtained by multiplying the heart rate obtained in step S501 by a predetermined coefficient, or a weighted value in combination with other information about the viewer. is. Other information about the viewer is, for example, the number of blinks. As a weighting method, for example, a weighted moving average or the like that reduces the weight of past healthcare information is used in order to emphasize recent healthcare information. In the present embodiment, the index is called a satisfaction level, but it can be called in any way, for example, the degree of heightened emotion, the degree of excitement, the degree of immersion, the degree of concentration, and the like.

In step S503, the control unit 11 determines whether or not the degree of satisfaction calculated in step S501 is greater than or equal to the threshold set in step S404. If the control unit 11 determines that the degree of satisfaction is greater than or equal to the threshold value, it proceeds to the process of step S504; otherwise, it returns to the process of step S501.

In step S504, the control unit 11 performs charging button display processing. The billing button display process will be described later with reference to FIG.

In step S505, the control unit 11 determines whether or not the charging buttons (charging buttons 203 and 301 as shown in FIGS. 2 and 3) are displayed on the display unit 14 by the charging button display processing in step S504. If the control unit 11 determines that the billing button has been displayed, it proceeds to the process of step S506; otherwise, it returns to the process of step S501.

In step S<b>506 , the control unit 11 detects the viewer's line-of-sight information via the line-of-sight detection unit 12 . The line-of-sight information is, for example, at least one of the direction and position of the viewer's line of sight and the number of times the viewer blinks.

In step S507, the control unit 11 determines whether or not the viewer has permitted the billing process based on the line-of-sight information detected in step S506. For example, it is determined whether or not the charging process has been permitted based on whether or not the viewer gazes at the charging button displayed on the display unit 14 for a predetermined period of time. In this case, an indicator icon may be displayed so that the viewer can recognize how long the viewer must pay attention to allow the billing process. Alternatively, it may be determined whether or not the billing process is permitted based on whether or not the user blinks a predetermined number of times within a predetermined period of time while looking at the billing button displayed on the display unit 14 . If the control unit 11 determines that the viewer has permitted the billing process, the process proceeds to step S508; otherwise, the process proceeds to step S509.

In step S508, the control unit 11 executes billing processing. The control unit 11 transmits information for executing billing processing to the distribution server 20 via the communication unit 17 . The information for executing the billing process is, for example, the account information of the viewer who performs the billing and the account information of the billing destination. The control unit 21 of the distribution server 20 receives information for executing billing processing from the VR device 10 via the communication unit 24, and executes billing processing. After completing the billing process, the control unit 21 notifies the VR device 10 of the completion of the billing process via the communication unit 24 . The control unit 11 of the VR device 10 receives the completion of the billing process from the distribution server 20 via the communication unit 17, and completes the billing process.

In step S509, the control unit 11 stops displaying the charge button on the display unit 14 and deletes the charge button.

Details of the charging button display processing in step S504 of FIG. 5 will be described below with reference to FIG.
In step S601, the control unit 11 determines whether the satisfaction level calculated in step S502 of FIG. A predetermined value (for example, 0) is used as the initial value of the maximum value. If the control unit 11 determines that the degree of satisfaction is higher than the maximum value, it proceeds to the process of step S602, otherwise it proceeds to the process of step S603.
In step S<b>602 , the control unit 11 updates the maximum satisfaction level stored in the RAM 16 . The processing of steps S601 and S602 is intended to operate so that the maximum value of the degree of satisfaction stored in the RAM 16 becomes the maximum value in repeated processing of steps S601 to S608, S613 and S614.

In step S<b>603 , the control unit 11 detects the viewer's line-of-sight position via the line-of-sight detection unit 12 .
In step S604, the control unit 11 detects performers in the video displayed on the display unit 14 based on the viewer's gaze position detected in step S602. At this time, the control unit 11 may analyze the image displayed on the display unit 14 and use the result of face detection or face recognition in the image.
In step S605, the control unit 11 refers to the distributor information stored in the RAM 16 in step S401 of FIG. 4, and acquires the performer information detected in step S604.
In step S606, the control unit 11 determines whether or not the performer detected in step S604 is charged. To determine whether or not the performer is subject to billing, refer to the distributor information stored in the RAM 16 in step S401 of FIG. Alternatively, the distributor information held by the distribution server 20 may be referred to by communicating with the distribution server 20 . If the control unit 11 determines that the performer is to be billed, the process proceeds to step S607; otherwise, the billing button display process is terminated, and the process proceeds to step S505 in FIG.
In step S<b>607 , the control unit 11 specifies that the performer detected in step S<b>604 is to be billed, and saves account information of the billing object in the RAM 16 .

In step S608, the control unit 11 determines whether the degree of satisfaction calculated in step S502 of FIG. 5 or step S614 described later is below the threshold set in step S404. If the control unit 11 determines that the degree of satisfaction is less than the threshold, it proceeds to the processing of step S609; otherwise, it proceeds to the processing of step S614. The purpose of step S608 is to suppress display for billing that is not originally included in the content video while the viewer is concentrating on the video. That is, while the viewer's satisfaction level is high (YES in step S503), while that state is maintained, in other words, while the viewer is concentrating on the video, steps S601 to S608, S613, Repeat processing of S614 is performed. Then, at the timing when the viewer's satisfaction level has decreased (YES in step S608), in other words, at the timing when the viewer has calmed down a little, the process proceeds to step S609, and the process of displaying for billing is started. I have to.
Although the threshold used in step S608 is the same as the threshold used in step S503, a different threshold may be used. After the degree of satisfaction reaches or exceeds the threshold value (YES in step S503), the calculation of the degree of satisfaction is repeated until it falls within a predetermined range, and when it falls within the predetermined range, the process of displaying for billing is started. You should do it like this.

In step S609, the control unit 11 determines whether or not the descent speed, which is the rate of decrease in satisfaction, is equal to or less than a threshold representing a predetermined rate of decrease. If the control unit 11 determines that the rate of decrease in the satisfaction level is equal to or less than the threshold, the process proceeds to step S610, otherwise, the process proceeds to step S615. The purpose of step S609 is not to display a billing indication and not to encourage tipping when the satisfaction level of the viewer is remarkably lowered.

In step S610, the control unit 11 determines the billing amount. The billing amount may be determined based on the maximum satisfaction level stored in the RAM 16, for example. Also, the billing amount may be set in advance by the viewer.
In step S611, the control unit 11 determines the display position of the charge button. The display position of the charging button may be determined, for example, based on the viewer's line-of-sight position acquired in step S603. Also, the display position of the billing button may be set in advance by the viewer, such as displaying it near the performer's face.
In step S612, the control unit 11 superimposes the billing button on the video distributed from the distribution server 20 and displays the billing button to which the billing amount determined in step S601 is added at the display position determined in step S611. As a result, the display as shown in FIGS. 2 and 3 is performed. Then, the control unit 11 ends the billing button display process, and proceeds to the process of step S505 in FIG.

In step S613, the control unit 11 acquires the viewer's healthcare information. Note that the processing of step S613 is the same as that of step S501, and detailed description thereof will be omitted.
At step S614, the control unit 11 calculates the degree of satisfaction based on the healthcare information acquired at step S613. Note that the processing of step S614 is the same as that of step S501, and detailed description thereof will be omitted.

In step S615, the control unit 11 resets the maximum satisfaction level stored in the RAM 16 to the initial value. Then, the control unit 11 ends the billing button display process, and proceeds to the process of step S505 in FIG.

As described above, the degree of satisfaction is calculated based on the healthcare information of the viewer who is viewing the online distribution. Then, according to the degree of satisfaction, the billing target is specified based on the line of sight of the viewer watching the online distribution, and the display for billing processing for the identified billing target is superimposed on the video distributed online. to perform control. As a result, it is possible to charge for the charging target desired by the viewer without impairing the viewer's sense of immersion. Especially in VR distribution that sells the sense of immersion by allowing users to participate in content, it is possible to prevent the sense of immersion from being impaired.

Note that in the present embodiment, the person in the video is the object of the charge, but the object may also be the object of the charge.
Also, health care information has been described as status information relating to the user's status in the present invention, but other information may be used in addition to or instead of this. Specifically, the motion of the user's face or body may be detected, and the degree of satisfaction may be calculated based on the information. For example, the level of satisfaction may be calculated by obtaining the degree of smile or surprise based on how the corners of the user's mouth is raised, how the mouth is opened, or how the eyebrows are raised. Alternatively, if the video distributed online is a concert video, the degree of satisfaction may be calculated by detecting that the user is making large movements or singing.

As described above, the present invention has been described together with the embodiments, but the above-described embodiments merely show specific examples for carrying out the present invention, and the technical scope of the present invention is not construed in a limited manner. It should not be. That is, the present invention can be embodied in various forms without departing from its technical concept or main features.
(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

10: VR device, 11: control unit, 12: line-of-sight detection unit, 13: operation unit, 14: display unit, 15: ROM, 16: RAM, 17: communication unit, 18: internal bus, 20: distribution server, 30 : controller, 40: network

Claims (12)

An information processing device used for viewing video distributed online,
Acquisition means for acquiring state information about a user's state;
index calculation means for calculating an index representing a user's emotion based on the state information of the user viewing the online distribution acquired by the acquisition means;
a line-of-sight detection means for detecting at least one of the direction and position of the user's line of sight;
According to the index calculated by the index calculation means, a billing target is specified based on at least one of the direction and position of the line of sight of the user watching the online distribution detected by the line of sight detection means, and the online distribution is performed. and a control means for superimposing a display for charging processing on the specified charging target by superimposing the display on the image to be charged. 2. The information processing apparatus according to claim 1, wherein said state information includes at least heart rate or pulse. 3. The information processing apparatus according to claim 1, wherein said acquisition means acquires said state information from a wearable controller. The index is a numerical representation of the heightened emotion of the user,
4. The information processing apparatus according to any one of claims 1 to 3, wherein said control means executes said control when said index calculated by said index calculation means is greater than or equal to a threshold value. After the index calculated by the index calculation means becomes equal to or greater than the threshold value, the control means repeats calculation of the index by the index calculation means until the index falls within the predetermined range. 5. The information processing apparatus according to claim 4, wherein the display for the billing process is performed when the number falls within. After the index calculated by the index calculation means has reached or exceeded the threshold value, the control means controls the charging process when the decrease rate of the index calculated by the index calculation means exceeds a predetermined decrease rate. 6. The information processing apparatus according to claim 4, wherein the display for the purpose is not performed. 7. The information according to any one of claims 4 to 6, further comprising setting means for setting said threshold value based on said state information of said user before viewing online distribution obtained by said obtaining means. processing equipment. The billing specified based on at least one of the line-of-sight direction and position of the user viewing the online distribution detected by the line-of-sight detection means while the control means displays the display for the billing process. 8. The information processing apparatus according to any one of claims 1 to 7, further comprising execution means for executing billing processing for an object. 9. The information processing apparatus according to any one of claims 1 to 8, wherein said control means displays a button for instructing execution of charging processing. 10. The information processing apparatus according to any one of claims 1 to 9, wherein said control means displays information on said specified billing object. An information processing method used for viewing video distributed online, comprising:
calculating an index representing the user's emotion based on state information about the state of the user while viewing the online distribution;
According to the index, a billing target is specified based on at least one of the direction and position of the user's line of sight while viewing the online distribution, and the specified billing target is superimposed on the video distributed online. and a step of executing control for performing display for billing processing. A program used for viewing video distributed online,
a process of calculating an index representing the user's emotion based on state information about the user's state during viewing of online distribution;
According to the index, a billing target is specified based on at least one of the direction and position of the user's line of sight while viewing the online distribution, and the specified billing target is superimposed on the video distributed online. A program for causing a computer to execute processing for performing display control for billing processing.

Images