WO2018142494A1

WO2018142494A1 - Display control system and display control method

Info

Publication number: WO2018142494A1
Application number: PCT/JP2017/003496
Authority: WO
Inventors: 幸司細見; 栗山　孝司; 祐輝勝俣; 和明出口; 志茂　諭
Original assignee: 株式会社ニコン
Priority date: 2017-01-31
Filing date: 2017-01-31
Publication date: 2018-08-09
Also published as: JP6965896B2; JPWO2018142494A1; TW201832161A; CN110249631B; CN110249631A; TWI701628B

Abstract

A display control system that: delivers live a real-space video in which a player is present; acquires three-dimensional position information of the real space; detects a user motion for a user to present an item to the player; calculates, on the basis of the acquired three-dimensional position information and user motion information pertaining to the detected user motion, an item position in the real space at which the item is to be disposed; and displays the calculated item position in the real space in such a way as to be recognizable by the player.

Description

Display control system and display control method

The present invention relates to a display control system that performs live distribution and a display control method.

Patent Document 1 and the like describe a server for distributing content.

Japanese Patent No. 5530557

According to an aspect of the present invention, a display device control unit that causes a display device to display an image of a real space in which a performer exists as a target of live distribution, an acquisition unit that acquires three-dimensional position information of the real space, and a user Based on the detection unit that detects a user action for presenting an item to the performer, the three-dimensional position information acquired by the acquisition unit, and the user action information of the user action detected by the detection unit. There is provided a display control system including an item display control unit that calculates an item position where the item is to be arranged in space and displays the calculated item position on the real space so that the performer can recognize the calculated item position. .

According to another aspect of the present invention, an image of a real space in which a performer exists is distributed live, three-dimensional position information of the real space is acquired, and a user action for a user to present an item to the performer is detected. Then, based on the acquired three-dimensional position information and user action information of the detected user action, an item position where the item is to be arranged in the real space is calculated, and the calculated item position is calculated as the performer. A display control method for displaying on the real space is provided.

The figure which shows the whole structure of a live delivery system. The block diagram of a server and a user terminal. (A) is a flowchart of a live delivery process, (b) is a figure which shows the display screen of a user terminal at the time of live delivery. The flowchart of the item / actor selection process by a user. (A) is a figure which shows the display screen of the user terminal at the time of selecting an item, (b) is a figure which shows the display screen of the user terminal at the time of selecting a performer, (c) is when an item and a performer are selected The figure which shows the display screen of a user terminal, (d) is a figure which shows the display screen of a studio monitor when an item and a performer are selected, (e) is the user terminal and studio which a user performed the operation | movement which throws an item The figure which shows the display screen of a monitor. The flowchart which shows the acquisition process in which a performer acquires the said item. (A) is a figure which shows the display screen of a user terminal and a studio monitor when a performer picks up the headband item of a cat ear, (b) is a user terminal and a studio monitor when a performer wears the item of a headband. (C) is a figure which shows the display screen of a user terminal and a studio monitor when a performer turns sideways. The flowchart of the return process from a performer to a user. (A) The figure which shows the display screen of a user terminal and a studio monitor when a performer has a return sign ball in his hand, (b) is the display screen of a user terminal and a studio monitor when a performer throws a sign ball (C) is a figure which shows the display screen of a user terminal and a studio monitor when a user receives a sign ball. (A) is a figure which shows the display screen of a user terminal and a studio monitor when an effect is added with respect to a performer, (b) is the display screen of a user terminal and a studio monitor when a user performs the operation | movement which throws an item. The figure which shows, (c) is the figure which shows the display screen of the user terminal and studio monitor when the performer receives the item, (d) is the display screen of the user terminal and studio monitor when the tower is displayed in the background image FIG.

A live distribution system to which the present invention is applied will be described below with reference to the drawings with reference to FIGS.
[Overview of live distribution system]
As shown in FIG. 1, the live distribution system 1 includes a studio 10 in which a performance such as a live performance is demonstrated, a server 20 that performs live distribution of content data acquired in the studio 10, and content data distributed by the server 20.

User terminals

40, 60, and 70 are provided. The server 20 and the

user terminals

40, 60, 70 are connected via the network 2. The number of user terminals is not limited to two as shown here, and may be one, tens or hundreds.

In the real space in the studio 10, as an example, three performers A, B, and C as subjects perform music on the stage and sing. Of course, the number of performers is not limited to three, but may be one, two, or four or more. Further, the performers A, B, and C may be a group such as one band, or may be a group of performers A, B, and C that are active independently. The studio 10 includes a playback device 11, a speaker 12, a microphone 13, an RGB camera 14, a depth camera 15, a projector 16, and a studio monitor 17.

The playback device 11 plays back the music data to be played and emits music based on the music data from the speaker 12 connected to the playback device 11. The microphone 13 is owned by each performer A, B, C, and collects the voices of the performers A, B, C. As an example, the RGB camera 14 is the first camera in the live distribution system 1. The RGB camera 14 is a digital camera having a moving image shooting function. As an example, the RGB camera 14 is a video camera. As an example, the RGB camera 14 is a display data generation camera or the like. The RGB camera 14 is a video camera that captures an actual space in which performers A, B, and C are playing. For example, the RGB camera 14 includes an image sensor such as a CCD or CMOS, detects light such as visible light, and outputs display data composed of color signals of three colors (red, green, and blue). As an example, the RGB camera 14 images subjects such as performers A, B, and C, and outputs ornamental data that can display the captured subjects on the display unit such as the

user terminals

40, 60, and 70 as display data. Further, as an example, the RGB camera 14 outputs imaging data to be displayed on the studio monitor 17 as display data. Further, as an example, the RGB camera 14 outputs, as display data, video data displayed on a large display device having a large screen installed in a public place where users A, B, and C are located, a live venue, or a concert hall. To do. Note that the RGB camera 14 does not have to be a video camera, and may be, for example, a smart device terminal having a moving image capturing function. In this case, by fixing the smart device terminal to a tripod or the like, it can function in the same manner as a video camera.

As an example, the depth camera 15 is the second camera in the live distribution system 1. As an example, the depth camera 15 is an infrared camera. As an example, the depth camera 15 is a three-dimensional position information acquisition camera. The depth camera 15 acquires depth information that is the distance from the subject to the subject. For example, the depth camera 15 is an acquisition unit that acquires depth information, which is a distance from performers A, B, C, and the like that are subjects. As an example, the depth camera 15 may include a distance to a performer A that is a part of the subject (depth information), a distance to a performer B that is a part of the subject (depth information), and a performer C that is a part of the subject. Each distance (depth information) is acquired. As an example, the depth camera 15 acquires depth information that is a distance to each point of a studio that is a part of the subject. As an example, the depth camera 15 acquires three-dimensional position information of a real space including performers A, B, and C as a subject and a studio. The depth camera 15 includes a light projecting unit that projects infrared rays and an infrared detection unit that detects infrared rays. For example, the depth camera 15 acquires three-dimensional position information such as depth information in real space from the time until the infrared pulse projected from the light projecting unit is reflected and returned. The RGB camera 14 and the depth camera 15 may be an integrated device or separate devices.

As an example, the projector 16 displays an object of an item that is a present for the performers A, B, and C on a stage by a technique such as projection mapping. The studio monitor 17 is a display device that is disposed in the studio 10 that is a real space and displays video. As an example, the studio monitor 17 is a display installed in front of the stage so that the performers A, B, and C are visible. Device. The studio monitor 17 is a flat display as an example, and is an LCD display device or an organic EL display device. The studio monitor 17 displays the performance images of the performers A, B, and C captured by the RGB camera 14.

The server 20 generates live data as content data played by the performers A, B, and C. As an example, the server 20 delivers to the

user terminals

40, 60, and 70 based on various data such as music data from the playback device 11, audio data from the microphone 13, and video data from the RGB camera 14. Performance live data by the performers A, B, and C is generated, and the live data is distributed live to the

user terminals

40, 60, and 70. That is, the server 20 relays live performances by the performers A, B, and C to the

user terminals

40, 60, and 70.

In the performance by the performers A, B, and C, the accompaniment is not played back by the playback device 11, but the performers A, B, and C actually play instruments such as guitars and drums, and the sound is collected by the microphone. You may do. Live data may be generated by the data generation device or the like in the studio 10 and transmitted to the server 20.

The users A, B, and C participating in the live distribution system 1 are fans of performers A, B, and C as an example, and can view live data using the

user terminals

40, 60, and 70. As an example, the user terminal 40 includes a desktop or laptop personal computer 40a, and a wearable terminal connected to the personal computer 40a or a smart watch 50 as a smart device terminal. When the personal computer 40a is a desktop type, the user terminal 40 includes a desktop type personal computer 40a, a monitor connected to the personal computer 40a, and a smart watch 50 connected to the personal computer 40a. When the personal computer 40a is a laptop computer, the user terminal 40 includes a laptop personal computer 40a having a display unit and a smart watch 50 connected to the laptop personal computer 40a. Yes. As an example, the user A of the user terminal 40 wears the smart watch 50 on the dominant arm or the like, and the smart watch 50 is connected to the personal computer 40a by wire or wirelessly. The smart watch 50 includes a detection unit such as an acceleration sensor or a gyro sensor. For example, when the user A performs an operation of throwing an object, the smart watch 50 detects the acceleration, angle (posture), and angular velocity as user operation information.

The personal computer 40a may be connected to a head mounted display (HMD) by wire or wirelessly. Further, the HMD may have a configuration as the personal computer 40a. Examples of the HMD include an optical see-through head mounted display, a video see-through head mounted display, and a non-transmissive head mounted display. In the case of an optical see-through head mounted display, display by AR (augmented reality) can be performed. In the case of a video see-through type head mounted display or a non-transmissive type head mounted display, display by VR (virtual reality) can be performed. On the HMD, an object of an item for a gift or a return to be described later can be displayed.

The user terminal 60 is a smart device terminal such as a smartphone or a tablet as an example, and is a portable information processing terminal. As an example, the smartphone includes a touch panel on the display surface. In addition, the user terminal 60 includes a detection unit such as an acceleration sensor or a gyro sensor. For example, when the user B performs an operation of throwing an object, the user terminal 60 detects the acceleration, angle, and angular velocity as user operation information. Since the user terminal 60 is a small portable information processing terminal, the user of the user terminal 60 can view live data anywhere.

The user terminal 70 of the user C includes a smart device terminal 60a and a smart watch 50. In this case, the smart device terminal 60a functions like the laptop personal computer 40a of the user A. Thus, even when the user C is throwing an object with the dominant arm wearing the smart watch 50, the user can view the video displayed on the display unit of the smart device terminal 60a held in the other hand. it can. Further, it is possible to perform an operation of throwing an object with the smart device terminal 60a placed on a desk or the like or fixed to a tripod. Thereby, it is possible to perform a throwing operation or the like with the dominant arm on which the smart watch 50 is worn while watching the video displayed on the display unit of the smart device terminal 60a.

The

user terminals

40, 60, and 70 can virtually present items to performers A, B, and C who are actually performing at that time while viewing live data. As an example, on the display surface of the

user terminals

40, 60, and 70, an item selection in which a list of objects as first objects of items that can be presented to the performers A, B, and C together with live data is displayed. The object is displayed. Examples of the items include ornaments such as bouquets and headbands, effects for directing the performer's actions when viewed on the

user terminals

40, 60, and 70, background images of the place where the performer performs. Then, the users A, B, and C select any one item from the object list in the item selection object, and further select performers to be presented from the performers A, B, and C. FIG. 1 shows an example in which the performer A is selected as the performer and the cat-ear headband is selected as the item.

Thereafter, the user A of the user terminal 40 performs an operation of throwing an object by waving his arm while wearing the smart watch 50. The user B of the user terminal 60 performs an operation of swinging an arm and throwing an object while holding the user terminal 60. The user C of the user terminal 70 performs an operation of throwing an object by waving his arm while wearing the smart watch 50. Then, the

user terminals

40, 60, and 70 transmit operation data such as acceleration data, angle (attitude) data, angular velocity data, and the like as user operation information as detection results to the server 20. The

user terminals

60 and 70 use the finger or stylus pen to trace the display surface on which live data is displayed in the direction of the performers A, B, and C, and the coordinate data and the like are displayed. The operation data may be transmitted to the server 20.

Then, the server 20 displays the video of the item 18 indicated by the item ID transmitted from the

user terminals

40, 60, and 70 on the floor surface of the studio 10 using the projector 16. As an example, the item 18 displayed as a video is displayed in front of the performer indicated by the performer ID transmitted from the

user terminals

40, 60, and 70. FIG. 1 shows an example in which the headband item 18 is displayed in front of the performer A. As an example, the projector 16 causes the item 18 to be displayed as if the item was thrown in the direction of the performers A, B, C from the user A, B, C side, i.e., the position in front of the performers A, B, C. Display video. In the real space, the position of the floor where the item falls, that is, the item position where the item 18 is finally displayed is a specific position specified in the studio 10 and is transmitted from the

user terminals

40, 60, and 70. It is determined based on operation data such as acceleration data, angle data, angular velocity data, etc. as user operation information. The item position is specified by three-dimensional position information such as depth information. As an example, the item position can be specified by a three-dimensional coordinate system having the detection unit of the depth camera 15 as an origin.

As an example, when the user A, B, or C performs an action of throwing an item weakly, the item position where the item 18 is displayed is far away in front of the performers A, B, and C. When the users A, B, and C perform an action of strongly throwing an item, the item position where the item 18 is displayed is a relatively close position in front of the performers A, B, and C. In addition, if the item is thrown too hard by users A, B, C, the item position at which item 18 is displayed is the performer A, B after the item bounces off the wall behind performers A, B, C. , C, the front and rear positions. Thus, the performers A, B, and C can visually recognize that the items are thrown in the direction of the users A, B, and C that are not actually in the studio 10. When an item is thrown at the performers A, B, and C, the item object is displayed on the display surface of the

user terminals

40, 60, and 70 at the item object position in the image captured by the RGB camera 14 along with the image. Is done. Accordingly, the users A, B, and C can also visually recognize that the item that the user has thrown has reached the performers A, B, and C. Furthermore, as an example, when the user A, B, or C performs an action of throwing an item to the right side or the left side with respect to a specific performer A, the item position where the item 18 is displayed in the real space is the user A, B, Corresponding to the direction in which C threw the item, the position is on the right or left side with respect to performer A. As an example, when an item is thrown to the right side with respect to the performer A, the item position where the item 18 is displayed may be the front of the performer B or the performer C. Such an item position is specified by three-dimensional position information determined based on operation data such as acceleration data, angle data, and angular velocity data as detection results detected by the

user terminals

40, 60, and 70.

Note that the studio monitor 17 may display the same image as the display screen of the

user terminals

40, 60, and 70.
As an example, the depth camera 15 of the studio 10 constantly calculates three-dimensional position information such as depth information at various locations in the studio 10. For example, the depth camera 15 extracts the person areas of the performers A, B, and C and divides them into a person area and a non-person area. For example, the depth camera 15 acquires 25 skeleton positions of each of the performers A, B, and C as skeleton data, and further calculates depth information of each skeleton position. Examples of the skeleton positions include skeleton positions such as left and right hands, heads, necks, left and right shoulders, left and right elbows, left and right knees, and left and right feet. The number of skeleton positions to be acquired is not limited to 25. Further, the depth camera 15 calculates the distance from the wall or floor surface in the studio 10. Here, the depth information is, for example, the distance from the objective lens or sensor surface in front of the depth camera 15 to the measurement target position (places on the wall of the studio 10 or places on the floor). The depth information is, for example, the distance from the objective lens or sensor surface in front of the depth camera 15 to the skeleton position of the performer as the subject.

When the position based on the depth information of the right hand or left hand of the performer A overlaps the item position with respect to the item position where the item 18 is displayed, for example, when the performer A picks up the item It is. Therefore, the projector 16 hides the item 18 when the right hand or the left hand of the performer A overlaps the item position. The item here is a headband. Therefore, the headband picked up by the performer A behaves so as to be worn on the head. During this time, the server 20 displays the headband object picked up as if held by the right or left hand of the performer A on the display surface of the

user terminals

40, 60, and 70, and the headband object is displayed by the performer A. Is displayed on the head. As a result, the users A, B, and C can recognize the headband that the presenter has given to the performer A on the display surfaces of the

user terminals

40 and 60, and the performer A can visually recognize how the headband is worn. .

Thereafter, the users A and B can visually recognize the performance of the performer A wearing the headband on the display surfaces of the

user terminals

40, 60 and 70. As an example, when the performer A turns sideways, the direction of the object of the headband is displayed in accordance with the direction of the performer A. The orientation of each performer A, B, C is determined by detecting the faces of the performers A, B, C from the display data from the RGB camera 14 and calculating the skeleton positions of the performers A, B, C from the depth camera 15. To do. Further, the data for displaying the item object is also three-dimensional data. As an example, when it is detected that the performer A is turned sideways based on these data, the direction of the headband object is changed in accordance with the direction of the performer A. The state in which the performer A wears the headband is also displayed on the studio monitor 17 so that the performers A, B, and C can visually recognize the states displayed on the

user terminals

40, 60, and 70.

As an example, the server 20 is presented to the performers A, B, and C when the item is presented to the performers A, B, and C by the user (when the items are received by the performers A, B, and C). All items (received by A, B, C) and their corresponding user IDs can be identified. Using this, the server 20 can specify the user ID that presented the item when the performers A, B, and C pick up the item.

When an interlude is entered, as an example, an image of the object of the user ID that presents the headband item to the performer A is displayed on the display screen of the studio monitor 17 of the studio 10. As an example, the video of the object with the user ID is displayed on the floor surface of the studio 10 by the projector 16. Thereby, the performer A can grasp | ascertain the users A, B, and C who gave the headband.

As an example, when the performer A speaks the user ID and is collected by the microphone 13, the server 20 identifies the user ID of the user who gives the return by voice recognition processing from the voice data from the microphone 13, and prompts the user. Be able to return. This voice recognition process can also be performed by a device other than the microphone 13 (such as a server 20 or a device installed in the studio 10). Further, as an example, when performing an action of throwing an item to be returned while the performer touches an item (for example, a headband) worn by the performers A, B, and C as a return process, the performers A, B, and C are touching. The user ID that gave the item is specified so that it can be returned to the user. Furthermore, as an example, the server 20 identifies all live distribution viewers by user ID, and performers A, B, and C perform return processing for users who have purchased items of a certain amount or more. It can be so. Furthermore, as an example, the user terminal may include a line-of-sight detection unit that detects the line of sight of the user, and the time when the user gazes at a specific performer may be calculated. In such a case, when the performer A performs the return process, it is possible to return to the user with the user ID who has been watching the performer A for a certain period of time. In addition, for the user with the user ID who has not seen the performer A for a certain period of time, the user with the user ID extracted at random is returned.

As described above, when performing a return, performer A displays the video of the item object as the second object as a present of return for users A, B, and C on display on studio monitor 17 and

user terminals

40, 60, and 70. Display on the surface. Specifically, when the return item is the sign ball of the performer A, the sign is displayed as if the sign ball is gripped by the right or left hand of the performer A on the display surface of the studio monitor 17 and the

user terminals

40, 60, and 70. The video of the ball item object is displayed at the position of the right hand or left hand of the performer A. As a result, the performer A can now grasp the present situation with the sign ball, and the users A, B, and C also throw the sign ball that the performer A will return to himself / herself. Can be grasped.

When the performer A performs an action of throwing a sign ball, the depth camera 15 detects that the performer A has thrown the sign ball based on a change in depth information of a hand holding the sign ball of the performer A. Then, the item object is displayed on the display screen of the

user terminals

40, 60, and 70 as if the performer A threw a sign ball toward the users A, B, and C. When the

user terminal

40, 60, 70 is operated to catch the sign ball displayed on the display surface, the users A, B, C have received the sign ball. The gift of return is not limited to a sign ball. The users A, B, and C can also throw items to the performers A, B, and C when live delivery is being performed again on the received return items such as sign balls. And the present as a return item may be mailed to users A, B, and C at a later date. In the case of mailing, it may be an actual sign ball, goods such as colored paper and performers, albums such as CDs and DVDs, concert coupons, and the like.

[Depth camera 15]
The depth camera 15 includes, for example, a projector such as a projector that projects pulse-modulated infrared light and an infrared detection unit such as an infrared camera until the projected infrared pulse is reflected and returned. Depth information is calculated from the time (Time of Flight (TOF) method). As an example, the depth camera 15 constantly calculates three-dimensional position information such as depth information at various locations in the studio 10.

Further, the depth camera 15 extracts the person areas of the performers A, B, and C and divides them into a person area and a non-person area. As an example, the person area is calculated based on the difference value before and after the person appears in the same place (studio 10 as an example). Further, as an example, an area where the amount of detected infrared rays exceeds a threshold is determined as a person area.

Furthermore, the depth camera 15 detects the skeleton position. The depth camera 15 obtains depth information of various places in the person area, and based on the depth and shape feature quantities, a part of the person in the person area in the real space (left and right hands, head, neck, left and right shoulders). Left and right elbows, left and right knees, left and right feet, etc.) and the center position of each part is calculated as the skeleton position. The depth camera 15 uses the feature amount dictionary stored in the storage unit to collate the feature amount determined from the person region with the feature amount of each part registered in the feature amount dictionary, thereby Each part in is calculated.

The depth camera 15 outputs the detection result of the infrared detection unit to other devices (server 20,

user terminals

40, 60, 70, a calculation device installed in the studio 10, etc.), and the other devices send the depth information. Processing such as calculation, extraction of a person area, division into a person area and a non-person area, detection of a person area, detection of a skeleton position, and identification of each part in the person area may be performed.

In addition, the motion capture process as described above is performed without adding markers to the performers A, B, and C. However, the performers A, B, and C may be performed with markers.

Furthermore, when calculating the depth information, a method may be used in which the projected infrared pattern is read and the depth information is obtained from the distortion of the pattern (Light Coding method).
Furthermore, the depth information may be calculated from parallax information obtained by a twin-lens camera or a plurality of cameras. Further, the depth information can be calculated by recognizing an image obtained by the RGB camera 14 and analyzing the image using a photogrammetry technique or the like. In this case, since the RGB camera 14 functions as a detection unit, the depth camera 15 becomes unnecessary.

[Server 20]
As shown in FIG. 2, the server 20 includes an interface (hereinafter simply referred to as “IF”) with each unit of the studio 10 and is connected by wire or wirelessly. As an IF connected to each part of the studio 10, an audio IF 21, an RGB camera IF 22, a depth camera IF 23, a projector IF 24, and a display IF 25 are provided. Furthermore, a database 26, a data storage unit 27, a network IF 28, a main memory 29, and a control unit 30 are provided. The server 20 distributes live data to the

user terminals

40, 60, and 70, and further functions as a display control device that controls the display of the projector 16, the studio monitor 17, and the

user terminals

40, 60, and 70.

The audio IF 21 is connected to the playback device 11 and the microphone 13 of the studio 10. The audio IF 21 receives music data to be played from the playback device 11 and voice data of the performers A, B, and C from the microphone 13.

The RGB camera IF 22 receives the video data of the studio 10 captured by the RGB camera 14. The depth camera IF 23 receives depth information of various places in the studio 10 and the performers A, B, and C, data on the person area, depth information on the skeleton position, and the like. The projector IF 24 controls the projector 16 to display items on the floor surface of the stage of the studio 10. The display IF 25 controls the studio monitor 17 installed in the studio 10. As an example, the display IF 25 displays the item object and user ID presented to the performers A, B, and C on the display surface of the studio monitor 17. Thereby, performers A, B, and C can know from whom the item was presented.

The database 26 as a management unit manages the items of each user in association with the user ID of the user registered in this system for each live. Specifically, the database 26 manages item IDs, item destination IDs, presence / absence of item reception, presence / absence of return, and success / failure of return receipt in association with each user ID for each live.

The item ID is an ID that uniquely identifies an item purchased by the user, and is an ID that uniquely identifies an item presented by the user in each live.
The item destination ID is an ID that uniquely identifies the performer who presented the item.

Whether or not the item has been received manages whether or not the performer selected by the user succeeded in receiving the item presented by the user.
The presence / absence of return is managed to determine whether the performer who received the item presented by the user gave a return to the user.

Whether or not the return is successfully received manages whether or not the return has been successfully received.
In addition, the database 26 manages all users who can participate in live distribution in association with user IDs. A user who participates in each live is selected from all registered users. Moreover, the database 26 manages the price of each item in association with the item ID as an example. Further, as an example, the total purchase price corresponding to each user's performer is managed.

In FIG. 2, the user A presents the item ID “A (bouquet)” to the performer C, but shows that the present has not been received by the performer C. The user B presents the item ID “C (headband)” to the performer A, the item A is received by the performer A, the performer A gives a return, and the user A receives the return. ing. The user C presents the item ID “B (effect)” to the performer B, the item B is received by the performer B, and the performer B returns, but the user C fails to receive the return. It is shown that.

The data storage unit 27 is a storage device such as a hard disk. The data storage unit 27 stores control data related to the live distribution system 1, display data for displaying item objects, and the like. The display data for displaying the object of the item is, for example, three-dimensional data when the item is a tangible object, such as a decoration, and the decoration is also displayed according to the direction of the performer. Yes. As an example, the data of the ornament is displayed from the front of the ornament when the performer is facing the front, and the ornament is also displayed sideways when the performer is facing sideways. As an example, the control program is a distribution program that distributes live data to the

user terminals

40, 60, and 70. As an example, the control program is an item display control program for displaying the object of the item presented by the user on the floor surface of the studio 10 by the projector 16. Further, as an example, the control program controls the display device that displays the item object presented by the user in association with the performers A, B, and C on the display surface of the studio monitor 17 and the

user terminals

40, 60, and 70. It is a program. The control program is, for example, a display device control program that displays an object of the sender's user ID on the display surface of the studio monitor 17 or the

user terminals

40, 60, and 70. Furthermore, as an example, when the performer who gave the item gives a return to the user, the control program displays the return item object on the display screen of the studio monitor 17 or the

user terminal

40, 60, or 70. The display device control program to be executed.

The network IF 28 connects the server 20 and the

user terminals

40, 60, 70 via the network 2 such as the Internet. The main memory 29 is, for example, a RAM, and temporarily stores live data being distributed, a control program, and the like.

The control unit 30 is a CPU as an example, and controls the overall operation of the server 20. For example, the control unit 30 is a distribution unit that distributes live data to the

user terminals

40, 60, and 70 in accordance with a distribution control program. As an example, the control unit 30 is an item display control unit that displays the item presented by the user on the floor surface of the studio 10 by the projector 16 according to the item display control program. As an example, the control unit 30 is a display device control unit that controls display of the

user terminals

40, 60, and 70, and is a display device control unit that controls display of the studio monitor.

Such a control unit 30 generates display data for displaying the item object presented by the user in association with the performers A, B, and C on the display surfaces of the studio monitor 17 and the

user terminals

40, 60, and 70. And display. Further, as an example, display data for displaying the sender's user ID is generated and displayed on the display surface of the studio monitor 17 and the

user terminals

40, 60, and 70. Furthermore, the control part 30 displays the item of return on the display surface of the studio monitor 17 or the

user terminals

40, 60, 70, for example, when the performer who presented the item returns the item to the user. Generate and display data.

When the item object is displayed on the display screen of the studio monitor 17 or the

user terminals

40, 60, and 70, the item object is displayed at the item object position that should originally exist in the real space of the studio 10. That is, as for the item position, the position in the real space of the studio 10 is specified by the three-dimensional position information. Even if the orientation of the RGB camera 14 is changed, the item object displayed on the display surface of the studio monitor 17 or the

user terminal

40, 60, 70 is an appropriate item object position in the video acquired in that orientation. Is displayed. Further, when the item position is out of the imaging range of the RGB camera 14, the item object is not displayed on the display screen of the studio monitor 17 or the

user terminals

40, 60, and 70. On the display surfaces of the studio monitor 17 and the

user terminals

40, 60, and 70, the item object is operated by the performers A, B, and C even when the performers A, B, and C squat down or jump up. Will be displayed according to.

The control unit 30 does not have to perform all of the above processes, and may perform some processes in cooperation with other devices. As an example, a control device such as a personal computer may be installed in a studio 10, and the control device installed in the studio 10 and the server 20 may be linked to perform a process as described above. In this case, for example, the server 20 includes a database 26, a main memory 29, and a control unit 30. Further, the control device includes an audio IF 21, an RGB camera IF 22, a depth camera IF 23, a projector IF 24, a display IF 25, a data storage unit 27, and a network IF 28. As an example, the control device may perform processing other than the update of the database 26. For example, a process for displaying an object of an item presented by the user on the projector 16 or a process for displaying the object on the display surface of the studio monitor 17 or the

user terminals

40, 60, and 70.

Further, a part of the above processing may be performed in cooperation with the

user terminals

40, 60, and 70. For example, real space video data acquired by the RGB camera 14, three-dimensional position information acquired by the depth camera 15, and the like are transmitted to the

user terminals

40, 60, and 70. The

user terminals

40, 60, and 70 detect the movements of the users A, B, and C, and the final arrival positions based on the real space video data, the three-dimensional position information, and the detection results of the movements of the users A, B, and C The trajectory of the item up to and the display of the item object are displayed on its own display surface.

[User terminal 40]
For example, the user terminal 40 is a device managed by the user A, and includes a desktop or laptop personal computer 40 a and a smart watch 50. As an example, the laptop personal computer 40a includes an audio IF 41, a display IF 42, a network IF 43, a communication IF 44, a data storage unit 45, an operation IF 46, a main memory 47, and a control unit 48. Yes. The audio IF 41 is connected to an audio output device such as a speaker, an earphone, and headphones, and an audio input device such as a microphone. As an example, the display IF 42 is connected to a display unit 49 configured by a display device such as a liquid crystal display device.

The network IF 43 communicates with the server 20 via the network 2 as an example. The communication IF 44 communicates with the smart watch 50 as an example. The communication IF 44 and the smart watch 50 are connected by a wireless LAN or a wired LAN, and acceleration data, angle data, angular velocity data, and the like as user operation information are input from the smart watch 50. The data storage unit 45 is a non-volatile memory, and is, for example, a hard disk or a flash memory. The data storage unit 45 stores a live data reproduction program, a communication control program with the smart watch 50, and the like. The operation IF 46 is connected to an operation device such as a keyboard and a mouse. When a touch panel is provided on the display surface of the display unit 49 connected to the display IF 42, the touch panel is connected to the touch panel. The main memory 47 is a RAM, for example, and temporarily stores live data being distributed, a control program, and the like. The control unit 48 is a CPU, for example, and controls the overall operation of the user terminal 40. As an example, the control unit 48 selects one or a plurality of performers A, B, and C when playing live data, transmits the performer selection data to the server 20, and item One or a plurality of item selection data is transmitted to the server 20 from the list of objects. Furthermore, as an example, the control unit 48 transmits operation data such as acceleration data, angle data, and angular velocity data as user motion information detected by the smart watch 50 to the server 20.

Smart watch 50 is a wristwatch-type information processing terminal that is worn on the wrist of user A's dominant arm as an example. The smart watch 50 includes a sensor 51, a communication IF 52, a data storage unit 53, a main memory 54, and a control unit 55. The sensor 51 is, for example, an acceleration sensor or a gyro sensor. As an example, the communication IF 52 transmits acceleration data detected by the sensor 51, angle data of the smart watch 50, and angular velocity data to the personal computer 40a. For example, when the user A performs a motion of throwing an object, the sensor 51 transmits operation data such as acceleration data, angle data, angular velocity data, etc. as user motion information related to the swing of the arm from the communication IF 52 to the personal computer 40a. . The data storage unit 53 is a non-volatile memory, and is, for example, a hard disk or a flash memory. The data storage unit 53 stores a driver for driving the sensor 51, a communication control program with the personal computer 40a, and the like. The control unit 55 is a CPU as an example, and controls the overall operation of the smart watch 50.

Note that the terminal connected to the user terminal 40 may be a small and portable information processing terminal such as a smartphone provided with an acceleration sensor and a gyro sensor instead of the smart watch 50.

[User terminal 60]
As an example, the user terminal 60 is a device managed by the user B, and is a smart device terminal such as a smartphone or a tablet. As an example, the user terminal 60 includes an audio IF 61, a display IF 62, an operation IF 63, a sensor 64, a network IF 65, a data storage unit 66, a main memory 67, and a control unit 68. The audio IF 61 is connected to a sound output device such as a built-in speaker or earphone or a sound input device such as a built-in microphone. As an example, the audio IF 61 emits live data from an audio output device. The display IF 62 is connected to a small display unit 69 such as a built-in liquid crystal panel or organic EL panel. The display unit 69 is provided with a touch panel, and the operation IF 63 is connected to the touch panel. The sensor 64 is, for example, an acceleration sensor or a gyro sensor. The network IF 65 communicates with the server 20 via the network 2 as an example. For example, when the user performs an action of throwing an object, the network IF 65 transmits, to the server 20, acceleration data, angle data, and operation data of angular velocity data regarding arm swing as user operation information detected by the sensor 64. Also, when a swipe operation is performed in which the display surface on which live data is displayed is traced in the direction of the performers A, B, and C using a finger or a stylus pen, the operation data such as the coordinate data is obtained. Send to server 20. The data storage unit 66 is a non-volatile memory, for example, a flash memory. The data storage unit 66 stores a live data reproduction program and the like. The main memory 67 is a RAM, for example, and temporarily stores live data being distributed, a control program, and the like.

The control unit 68 is a CPU as an example, and controls the overall operation of the user terminal 60. As an example, when playing back live data, the control unit 68 selects one or a plurality of performers A, B, and C, transmits the selected data to the server 20, One or more selection data from the list of objects is transmitted to the server 20. Further, as an example, when the user performs an operation of throwing an object with the user terminal 60, the control unit 48 receives operation data such as acceleration data, angle data, angular velocity data, coordinate data, etc. of the arm swing. Send to.

Hereinafter, the operation of the live distribution system 1 will be described.
[Live distribution processing]
Prior to the live distribution, first, in the studio 10, the depth camera 15 acquires depth information of each place in the studio 10, calculates the person area, then calculates the skeleton position in the person area, Allow depth information to be calculated. Thereafter, in the depth camera 15, motion capture processing is executed. In addition, the

user terminals

40, 60, and 70 log in to the server 20 so that live distribution can be viewed.

As shown in FIG. 3A, when performers A, B, and C start playing, in step S1, the server 20 generates live data as content data. Specifically, the server 20 is input with real space video data played by the performers A, B, and C in the studio 10 captured by the RGB camera 14. In addition, music data is input from the playback device 11 to the server 20, and voice data of performers A, B, and C are input from the microphone 13. And the server 20 produces | generates the live data of the performance by the performers A, B, C for delivering with respect to the

user terminals

40, 60, 70 based on these various data. Further, in step S2, the server 20 receives depth information on the skeleton positions in various locations of the studio 10 and the performers A, B, and C. In step S 3, the server 20 delivers live data to the

user terminals

40, 60, and 70 live. That is, the server 20 delivers performances by the performers A, B, and C to the

user terminals

40, 60, and 70 in real time. As a result, as shown in FIG. 3B, the

user terminals

40, 60, and 70 display the live video 71 based on the live data on the display surface and output live audio.

[Item / Performer selection process]
Next, the item / performer selection process will be described with reference to FIG. Here, a case where the user B views and operates live data on the user terminal 60 will be described as an example. In step S11, the server 20 displays an item selection object 72 that displays a list of selectable item objects on the display surface of the user terminal 60 so as to be superimposed on the live video 71 (see FIG. 5A). . As an example, in FIG. 5A, the item selection object 72 includes, in order from the left, an object 72a indicating a bouquet item, an item object 72b to which an effect for directing the performer's action is added, and a cat ear headband. An object 72c indicating an item and an object 72d indicating an item of a background image of live distribution are listed in a line.

The items listed in the item selection object 72 are items prepared by the operator. The prepared items may be different for each live, or may be common to all live. Moreover, some items may overlap in a plurality of live performances. In the database of the server 20, the price of each item is managed in association with the item ID. As an example, the server 20 stores video data, image data, audio data, music data, and the like as item data for displaying the item in association with the item ID. The item data is three-dimensional data as an example.

Each item is charged, the amount is determined according to the item, and the price is associated with the item ID. As an example, the item of the bouquet with the item ID “A” is 200 yen. The item to which the effect of item ID “B” is added is 300 yen. The item of the cat ear headband of item ID “C” is 500 yen. The item “D” of the background image with the item ID “D” is 1000 yen. As an example, user A shown in the database of FIG. 2 purchases a bouquet of item ID “A” for 200 yen, and user B purchases a headband of item ID “C” for 500 yen. The user C has purchased the effect of the item ID “B” for 300 yen. Thus, the users A, B, and C can present items to the performers A, B, and C by purchasing these items through the

user terminals

40, 60, and 70. Thus, the performers A, B, C and the operator can obtain sales corresponding to the items presented by the users A, B, C. Regardless of whether or not the performers A, B, and C receive (for example, pick up) items, all items presented by the users A, B, and C become sales for the performers A, B, and C and their operators. There may be free items among the items. Moreover, one user may purchase one item in one live, and may purchase several items. In the database, the total amount of purchases corresponding to the performers of each user is managed. Thereby, as an example, the server 20 managed by the operator can perform a process of giving a preferential return from the performer to a user who has purchased many items.

When one item is selected by the user B from the list of item selection objects 72 on the user terminal 60, the user terminal 60 sends item selection data including the user ID and the item ID of the selected item to the server 20. Send. In step S12, the server 20 performs an item selection process based on the item selection data. The server 20 transmits selection data for displaying only the object 72 c of the selected item on the user terminal 60 to the user terminal 60, and the object is superimposed on the live video 71 on the display surface of the user terminal 60. 72c is displayed. FIG. 5B shows a state in which the server 20 selects an object 72c indicating a headband item and displays it on the lower corner of the display surface of the user terminal 60 as an example. At the same time, the server 20 displays a similar display on the studio monitor 17 in order to inform the performers A, B, and C that the item selection process is currently in progress.

Moreover, the server 20 displays the performer selection object 73 surrounding the performers A, B, and C one by one on the display surface of the user terminal 60 as an example. Here, on the display surface of the user terminal 60, as an example, the first teaching object 73a in which the user B next notifies the performer selection operation is also displayed. When one performer selection object 73 is selected on the user terminal 60, the user terminal 60 transmits performer selection data including the user ID and the performer ID of the selected performer to the server 20. In step S13, the server 20 performs a performer selection process based on the performer selection data. The server 20 displays the performer determination object 74 for the selected performer A so as to be superimposed on the live video 71 on the display surface of the user terminal 60. FIG. 5C shows a state in which the server 20 selects the performer A and displays it on the display screen of the user terminal 60 as an example. The performer selection object 73 and the performer determination object 74 may be rectangular, but are not limited thereto, and may be, for example, circular or triangular. In addition, as shown in FIG. 5D, the server 20 displays the same on the studio monitor 17 in order to inform the performers A, B, and C that they are currently in the performer selection process. The studio monitor 17 displays a second teaching object 74a indicating that the item selected by the user B is about to be sent.

The server 20 registers the selected item and the performer in the database 26 when the item and the performer are selected.
Thereafter, the user of the user terminal 60 is in a state in which items can be presented to performers A, B, and C in the studio 10 while operating the user terminal 60. Specifically, the user B of the user terminal 60 can perform a pseudo-experience of throwing the selected item against the performer he / she selects by throwing an object while holding the user terminal 60 in his / her hand. Specifically, the server 20 starts synchronization processing with the user terminal 60, and the user terminal 60 operates data such as acceleration data, angle data, and angular velocity data as user motion information detected by the sensor 64 for each unit time. Is transmitted to the server 20. In step S 14, the server 20 stores a threshold value for determining that the user has performed the throwing motion, and determines that the throwing motion has been performed at the user terminal 60 when the threshold value is exceeded. As an example, the server 20 stores threshold values such as acceleration data, angle data, and angular velocity data in order to specify a throwing motion. The server 20 determines that a throwing motion has been performed when acceleration data, angle data, angular velocity data, etc. exceed a threshold value. Further, when the distance between the start point and the end point when the touch panel swipe operation is performed exceeds a threshold value, it is determined that the throwing operation has been performed. In the case of the user terminal 40, when acceleration data, angle data, angular velocity data, or the like exceeds a threshold value, it is determined that a throwing motion has been performed.

In step S15, the server 20 analyzes the direction and speed of the user's arm swing based on operation data such as acceleration data, angle data, and angular velocity data related to arm swing transmitted from the user terminal 60. Thereby, the server 20 calculates an item position which is a trajectory or a drop position when the thrown item is thrown. As an example, the item position can be specified by a three-dimensional coordinate system having the detection unit of the depth camera 15 as the origin.

In step S 16, the server 20 generates display data of the falling object 75 indicating the falling headband item displayed on the display surface of the user terminal 60 based on the analysis result, and transmits the display data to the user terminal 60. As a result, as shown in FIG. 5 (e), the falling object 75 is displayed in real time on the display surface of the user terminal 60 so as to come toward the performer A. In addition, similar display data of the fall object 75 is transmitted to the studio monitor 17 of the studio 10, and the fall object 75 is also displayed on the studio monitor 17 in real time. When the fall object 75 is displayed on the display screen of the studio monitor 17 or the user terminal 60, the fall object 75 is displayed at the item object position that should originally exist in the real space of the studio 10. That is, the item position in the real space of the studio 10 is specified by the three-dimensional position information. Therefore, even if the orientation of the RGB camera 14 is changed, the falling object 75 is displayed at an appropriate item object position in the image acquired in the orientation of the RGB camera 14. Further, when the item position is out of the imaging range of the RGB camera 14, the falling object 75 is not displayed.

Note that the process of displaying the falling object 75 on the studio monitor 17 may be omitted because the falling object 75 is displayed on the floor surface of the studio 10 by the projector 16 in the next step S17.

In step S 17, the server 20 transmits the display data of the falling object 75 to the projector 16, and the projector 16 is the object object and the item position of the item that comes to the performer A in a flying state on the floor surface of the studio 10. The object of the item that has fallen on is displayed in real time. Thereby, performers A, B, and C can grasp the falling position of the falling object 75.

The falling object 75 only needs to be displayed at least at the item position, and it is not necessary to display a state or locus in the middle of flying to the item position.
Further, the falling object 75 may be displayed so that the item position does not fall within the performer action range where the performers A, B, and C act. Further, the object may enter the performer action range until the item position is reached, but the falling object 75 is displayed so that the object of the item presented by the user does not finally enter the performer action range. It may be. Further, even if the item position calculated by detecting the operation of presenting the item to the performer (for example, throwing) is located within the performer action range, the object may not be displayed within the performer action range. In this case, for example, in consideration of the calculated item position, the object is displayed near the outside of the performer action range (the position outside the performer action range and closest to the calculated item position). According to such a display form, it is possible to prevent the performers A, B, and C from stepping on the items that were presented by mistake.

The performer action range is, for example, a stage in the studio 10 or the like. In addition, the performer action range may be set to a range that is different between the periods of the prelude, interlude, and postlude in one song, and other periods. Also, different ranges may be set for the period during which the song is played and the period during which the song is not played (for example, the period during which the song is played is the range in which the performer acts in accordance with the song, and the song is played) If there is no period, the range of performer activity is assumed to be none). Furthermore, different ranges may be set according to the music being played, or the same range may be set throughout the live distribution period. Not only the performance but also a different range may be set for the acting period and the period before and after the acting.

Furthermore, when an item hits the performer, the item may be dropped on the spot, the direction of the item flying may be changed, or the item may be moved.
[Item acquisition processing]
Next, an operation in which the performer A acquires the headband item presented by the user B will be described with reference to FIG.

In the server 20, the skeleton positions of the performers A, B, and C and the depth information of each skeleton position are always input from the depth camera 15. The server 20 detects the faces of the performers A, B, and C from the video from the RGB camera 14. Thereby, the server 20 tracks each skeleton position of the performer A, depth information of the skeleton position, and the position of the face of the performer A. In step S21, the server 20 determines that the performer A has picked up the headband item. As an example, in order to identify the action of picking up an item that has fallen on the floor of the studio 10, the server 20 determines the distance between the skeleton position of either the left or right hand and the skeleton position of either the left or right hand, or the left or right hand. A threshold value relating to the distance between the skeleton position and the floor surface is stored. The server 20 determines that the calculated distance between the left and right hand skeleton position and the left or right foot skeleton position, the distance between the left or right hand skeleton position and the floor position at the item position, etc. exceeds a threshold. It is determined that performer A has picked up the headband item. As an example, the server 20 determines that the performer A has picked up the headband item when the position of either the left or right hand overlaps the item position on the floor. In other words, it is determined whether the performer is positioned within the range of the object of the item presented to the performer by the user. For example, the range of the object of the item is determined by the three-dimensional position information, the user action information, and the item type. As an example, in the case of a headband, it is the range of the outer shape of the headband. As an example, the determination of whether the left or right hand position and the item position on the floor overlap each other includes, as an example, three-dimensional information (one point) of the performer's hand position and three-dimensional information (one point) of the item position It is determined whether or not. Further, as an example, it is determined whether or not the three-dimensional information (multiple points) of the position of the performer's hand and the three-dimensional information (one point) of the item position overlap. Furthermore, as an example, it is determined whether the three-dimensional information (one point) of the performer's hand position and the three-dimensional information (multiple points) of the item position overlap. Furthermore, as an example, it is determined whether the three-dimensional information (plural points) of the position of the performer's hand overlaps with the three-dimensional information (multiple points) of the item position. Further, as an example, it is determined whether or not the three-dimensional information on the performer's hand position (region such as a fingertip) overlaps with the three-dimensional information on the item position (region where the falling object 75 is displayed). Whether or not the position of either the left or right hand and the item position on the floor are overlapped is determined by the fact that the 3D information of the performer's hand position and the 3D information of the item position are not a single point but a plurality of points or areas. It is easier to do this by determining whether they overlap.

In step S 22, the server 20 performs control to hide the falling object 75 displayed on the floor surface of the studio 10. This is because if the item is picked up by the performer A, the item disappears from the floor.

From the time the item is picked up by the performer A until the item is put on the head of the performer A, it is held by the performer A. As an example, FIG. 7A shows a state where the performer A has picked up a headband item. Therefore, in step S23, the server 20 analyzes the acquisition operation of the performer A. That is, the server 20 analyzes each skeleton position of the performer A, the depth information of the skeleton position, and the operation from the position of the face of the performer A to the mounting of the headband item on the head. In step S 24, the server 20 generates display data of the acquisition object 76 indicating the headband item from picking up to wearing on the head on the display surface of the studio monitor 17 and the user terminal 60 based on the analysis result. And transmitted to the studio monitor 17 and the user terminal 60. Thereby, as an example, the acquisition object 76 is displayed on the display surfaces of the studio monitor 17 and the user terminal 60 in association with the picked-up hand until the acquisition object 76 moves from the item position on the floor surface to the head.

In step S25, the server 20 analyzes the mounting operation in which the performer A mounts the headband item on the head. That is, the server 20 analyzes the operation of mounting the headband item on the head from each skeleton position of the performer A, depth information of the skeleton position, and the position of the face of the performer A. As an example, the server 20 detects the mounting operation when the position of either the left or right hand overlaps the position of the head. In step S 26, the server 20 generates display data for displaying the acquisition object 76 on the display positions of the studio monitor 17 and the user terminal 60 at the mounting positions of the heads of the performers A, B, and C. Transmit to the terminal 60. As an example, the server 20 generates display data for displaying the acquisition object 76 along the boundary between the hair color and the background. Thereby, as an example, the state where the acquisition object 76 is mounted on the head of the performer A is displayed on the display surfaces of the studio monitor 17 and the user terminal 60 (see FIG. 7B). The server 20 tracks the head of the performer A and displays the headband item as always being worn even if the performer A moves.

Actor A may turn sideways depending on choreography. Even in such a case, the server 20 displays the acquisition object 76 in accordance with the direction of the performer A (see FIG. 7C). The direction of each performer A can be determined by detecting the face of the performer A from the display data from the RGB camera 14, and calculating the skeleton position of the performer A from the depth camera 15, and data for displaying the object of the item Is also three-dimensional data, and can be displayed from any orientation. Based on these data, when it is detected that the performer A has turned sideways, the direction of the headband object is also changed in accordance with the direction of the performer A. The acquisition object 76 is displayed in accordance with the action of the performer A even when the performer A squats down or jumps up.

When the server 20 acquires the item by the selected performer, the server 26 registers the success in the database 26.
When the headband item is attached to the performer A, in step S27, the server 20 sends the ID object 76a of the user ID indicating the user B who presents the headband item to the performer A in the studio monitor 17 and the user terminal 60. Display on the display surface. Thereby, the performer A can visually recognize the user ID of the user who presented the headband item, and the user B can also display his / her own user ID so that his / her head is presented. It can be visually confirmed that the item of the band is attached to the performer A. The server 20 may be displayed on the floor surface of the studio 10 by the projector 16.

The period in which the item object is displayed in association with the performer may be the entire period in which live distribution has been performed since acquisition or may be for each song. Further, it may not be displayed during the interlude.

Also, an item once worn by the performer can be removed and stored or placed in a box (which may be a real object such as being installed in a studio or a virtual object similar to an item). Thereby, when many items are presented to the performer, the performer can wear a plurality of presents. For example, the performer can wear a plurality of headbands at once, or the headband previously worn by the performer can be removed and a new headband can be picked up and worn. In such a case, the box can be used like a desk or a storage box, and a headband replacement operation or the like can be produced without a sense of incongruity.

[Return processing]
Since performers A, B, and C do not sing during the interlude, a return operation can be performed in return for user B who presented the item. In step S 31, the server 20 determines whether the music being played live has entered an interlude. For example, the server 20 can determine that an interlude has been entered when there is no voice input from the microphone 13 for a predetermined period. For example, the server 20 can determine that an interlude has been entered when a detection signal indicating that an interlude has been entered from the playback device 11. As an example, it can be determined that an interlude has been entered by detecting an action indicating that an interlude has been entered. As an example, the server 20 starts synchronization processing with the user terminal 60 so that the display on the user terminal 60 and the return receipt processing operated on the user terminal 60 can be detected.

Note that the return process may be performed between songs instead of between interludes. Moreover, when performing a play or a musical on the stage or the like, it may be between the Nth act and the N + 1th act.

It should be noted that the end of the interlude can be determined when there is a voice input from the microphone 13 or when a detection signal indicating the end of the interlude is input from the playback device 11. Moreover, it can be determined that the interlude has ended by detecting an action indicating that the interlude has ended.

In step S27 described above, the server 20 displays the user ID of the user B who presented the headband item to the performer A on the display screen of the studio monitor 17 and the user terminal 60. Therefore, the performer A calls the user ID of the user B who gave the headband item to the performer A toward the microphone 13. Then, in step S32, the server 20 recognizes the voice data collected by the microphone 13 and identifies the user ID of the user B. It should be noted that the server 20 registers in the database 26 that a user who is a returnee is to return.

In step S33, the server 20 detects the return operation of the performer A. For example, the server 20 detects a characteristic specific action indicating that the performer shifts to a return action. As an example, the server 20 stores a threshold value of each skeleton position for determining the specific action, and determines that the performer A has performed the specific action when the data of each skeleton position exceeds the threshold value. Here, the return item from the performer A to the user B is, for example, a sign ball of the performer A. The performer A signs the user B who is not actually in the studio 10 from the studio 10 following the specific action. Move the ball. In step S34, the server 20 analyzes the return operation from each skeleton position of the performer A, depth information of the skeleton position, and the position of the face of the performer A. In step S35, as an example, the server 20 displays a sign ball return object 77 on the display surface of the studio monitor 17 and the user terminal 60 at the position of the left or right hand of the performer A at the start of the throwing motion by the performer A. Is generated and transmitted to the studio monitor 17 and the user terminal 60. Thereby, as shown in FIG. 9A, the return object 77 is displayed in real time on the display screens of the studio monitor 17 and the user terminal 60 as an example. Further, the server 20 generates display data for displaying the receiving object 78 imitating the hand of the user B on the display surfaces of the studio monitor 17 and the user terminal 60 and transmits the display data to the studio monitor 17 and the user terminal 60. This is a virtual target when the receiving object 78 throws a sign ball.

In step S36, the server 20 analyzes the throwing motion by the performer A. Specifically, the server 20 detects the swing of the performer A's arm from each skeleton position of the performer A, depth information of the skeleton position, and the position of the face of the performer A. In step S 37, the server 20 generates display data for displaying the return object 77 on the left or right hand position of the performer A in the middle of the throwing motion on the display surfaces of the studio monitor 17 and the user terminal 60. Also, display data for displaying the return object 77 in a state of flying away from either the left or right hand is generated. And it transmits to the studio monitor 17 and the user terminal 60. Accordingly, as shown in FIG. 9B, as an example, the display screens of the studio monitor 17 and the user terminal 60 are displayed in real time as if a sign ball was thrown in the direction of the receiving object 78.

It should be noted that an effect may be given to the arm of the performer A who throws the sign ball. As an example, this effect is to detect the movement of the performer and display it on the downstream edge of the movement direction of the performer A's arm as an effect according to the detected movement. It is. As an example, the effect is displayed in association with the arm that throws the sign ball of the performer A when the return operation is detected. Then, when the operation moves to the throwing motion, it is displayed on the downstream edge of the arm moving direction in accordance with the movement of the waving arm. Further, when the return operation is detected, the background image may be changed to a specific image displayed during the return process.

When the receiving operation is performed at the timing when the return object 77 arrives at the receiving object 78 at the user terminal 60, the user terminal 60 transmits the received data including the user ID to the server 20. The receiving operation is an operation of holding the ball, and is an operation of clicking any position on the screen or the receiving object 78 with the mouse. The receiving operation is an operation of touching the touch panel as an example. In step S38, when receiving the received data, the server 20 registers in the database 26 that the return has been received. At this time, the server 20 generates display data for displaying the state where the return object 77 is caught by the receiving object 78 on the display surfaces of the studio monitor 17 and the user terminal 60, and Send. Thereby, as shown in FIG.9 (c), the state which caught the sign ball with the hand is displayed on the display surface of the studio monitor 17 and the user terminal 60 as an example.

Note that if the receiving operation is not performed when the return object 77 arrives at the receiving object 78, the receiving failure data including the user ID is transmitted to the server 20, and the server 20 receives the receiving failure data. The fact that receipt of return has failed is registered in the database 26.

[Other items]
The item object 72b to which an effect for directing the performer's action shown in FIG. 5A is added to the performers A, B, and C as follows. In FIG. 10A, an effect object 81 is added to the selected performer A. The video of the effect object 81 is detected on the downstream edge of the movement direction of the performer A's arm, which detects the movement of the performer and, as an effect corresponding to the detected movement, moves a large number of blinking star-shaped figures. ing. Such an effect object 81 is not a tangible object like the above-mentioned headband. As shown in FIG. 10B, when throwing against the selected performer A, the box object 82 with a present ribbon or the like is used. Then, as shown in FIG. 10C, when the selected and acquired by the performer A, that is, when the position of either the left or right hand overlaps the item position, the box object 82 is hidden, and thereafter the effect object Control to display 81 is performed.

As an example, even if the performer A squats down or jumps up, it is displayed according to the operation of the performer A. As an example, the effect may be changed before and after the performers A, B, and C jump. As an example, effects are displayed as blinking star shapes before jumping, and different shapes are blinking displayed after jumping. Further, as an example, a plurality of specific operations are defined in advance, and when one specific operation is detected, a display for giving a specific effect associated with the operation is performed. As an example, when a specific operation is detected, the display for applying the effect is stopped. Further, as an example, the display for applying the effect is not performed until a specific operation is detected.

FIG. 10D shows a state in which an object 72d indicating an item of a live distribution background image is selected. Such a background image object 72d is not a tangible object like the above-described headband. For this reason, when throwing against the selected performer A, it is preferable to use the present box object 82.

Even when an item of the headband is selected, when throwing against the selected performer A, a present box object 82 may be displayed.
The return object 77 may be displayed only on the

user terminals

40, 60, and 70 of the user who makes the return. Thereby, the one-to-one communication between the performer and the user can be realized.

[Other item / performer selection process]
In the above example, although the case where the user B selected the performer A was demonstrated, the performer B and the performer C can be selected instead of the performer A, and one user is one

user terminal

40,60, A plurality of performers such as performer B and performer C may be selected together with performer A from 70. When a plurality of performers are selected, different items may be selected or the same item may be selected as long as the object is not the object 72d indicating the item of the background image of the live distribution.

According to the live distribution system 1, the effects listed below can be obtained.
(1) In the live distribution system 1, as a user desire, for example, a user purchases an item from a desire for the performer to receive the item until the performer receives it, and presents it to the performer. In addition, the user wants to increase the possibility that the performer will receive it, so he tries to throw an item close to the performer. In addition, as a competitive consciousness between users, for example, a competitive consciousness is born that the performer has received the item or has received a reward. Thereby, item purchase by the user can be prompted. In this way, the profits of the operator and the performer can be increased.

(2) Users A, B, and C can give gifts to the studio 10 by throwing items to the studio 10 for the performers A, B, and C performing in the studio 10. The presented item is displayed as a falling object 75 on the floor surface of the studio 10. Thereby, the performers A, B, and C can also visually recognize that the item is presented from the users A, B, and C such as fans. Also, since the

user terminals

40, 60, and 70 are displayed in front of the performers A, B, and C, the items that the users A, B, and C have thrown also arrive in front of the performers A, B, and C. Can be visually recognized. Thus, even if the users A, B, and C are not actually in the studio 10, they present items to the performers A, B, and C from the

user terminals

40, 60, and 70 as if they were in the studio 10. can do. In addition, performers A, B, and C can also behave like receiving actual presents.

(3) When presenting performers A, B, and C pretend to pick up an item object displayed on the floor of the studio 10, the presenting performers A, B, and C are displayed on the

user terminals

40, 60, and 70, respectively. It is displayed as if it was actually worn. Therefore, the presenting user can visually recognize that the item he / she received has been received.

(4) When the user presents the performers A, B, and C performing in the studio 10 by throwing items to the studio 10 and presents them, the

user terminals

40, 60, and 70 Operation data such as acceleration data, angle data, angular velocity data, etc. relating to the swing is transmitted. Therefore, the item position where the item falls can be changed according to the swing speed of the arm or the like. Therefore, the users A, B, and C can adjust the speed of swinging the arm so that the item falls as much as possible before the performers A, B, and C selected, and can enhance entertainment. .

(5) The performers A, B, and C can visually recognize the user IDs of the users A, B, and C who presented the items.
(6) The performers A, B, and C can return to the users A, B, and C who gave them presents. Thereby, two-way communication can be realized between the performers A, B, and C and the user.

(7) Return items can also be displayed on the

user terminals

40, 60, and 70 according to the actions of the performers A, B, and C. Furthermore, by making it possible to perform an operation for catching a return item with good timing on the

user terminals

40, 60, and 70, it is possible to further enhance entertainment.

In addition, the said live delivery system can also be suitably changed and implemented as follows.
When performers A, B, and C give a return to users A, B, and C who gave presents, performers A, B, and C do not perform a return operation using return object 77 Also good. In this case, a tangible gift for return may be mailed to the users A, B, and C who gave the gift. Thereby, the process of the live delivery system 1 can be simplified.

・ For return gifts, tangible gifts may be mailed to users A, B, and C at a later date. In the case of mailing, it may be an actual sign ball, goods such as colored paper and performers, albums such as CDs and DVDs, concert coupons, and the like. When the return is mailed, it is performed for the user who has registered in the database 26 that the return has been received. The sender in this case may be performers A, B, and C, or may be the operator of this system. In the case of failure in receiving, the user may not be able to receive a tangible present (not mailed).

・ Performers A, B, and C do not have to give back to users A, B, and C who gave them presents. That is, the server 20 may omit the return process, and even if an item is received, the return item need not be mailed.

As an example, when the user A, B, or C who gave the gift is not returned, the user ID may not be managed in association with the gift item. Thereby, the process of the live delivery system 1 can be simplified.

When the touch panel is provided like the user terminal 60, the display surface on which the live data is displayed is traced in the direction of the performers A, B, and C by using a finger or a stylus pen. Then, an operation of presenting items to the selected performers A, B, and C may be performed. In this case, an acceleration sensor and a gyro sensor are not required for the user terminal. When the user is using a user terminal that does not include a touch panel, the selected performer A is performed by moving the pointer in the direction of the performers A, B, and C displayed using the mouse. , B, C may be operated to present items.

· At least the fall object of the item thrown to the performers A, B, and C may be displayed at least at the item position, and the trajectory until the fall object reaches the item position may be omitted.

・ The actual space where the performer will perform may be other than the studio 10, or a live venue or a concert venue. In this case, the projector 16 will display the item object on the stage, and the user will be present to the performer using his / her small and portable information processing terminal such as the user terminal 60 while in the audience seat. To throw items.

The means for displaying the item object on the studio 10 is not limited to the projector 16. As an example, the floor of the studio 10 is configured by arranging a plurality of flat display panels such as liquid crystal display panels so that the display surface faces the floor surface, and laying a transparent synthetic resin plate on the display surface. The object of the item may be displayed. Further, the item position may be simply indicated by the laser pointer. The item may be displayed using an aerial display technology, an aerial imaging technology, or an aerial imaging technology. The item may be displayed as a two-dimensional image (computer graphics (CG)) or a three-dimensional image (computer graphics (CG)). Furthermore, an object of an item may be represented by spreading a large number of rods on the floor surface and raising and lowering the rods in a direction perpendicular to the floor surface to change the floor surface into a wave shape. The means for displaying the item object on the studio 10 may be a combination of these devices.

・ Return items may be registered in the database 26, and the return items may be thrown to the performer at the next live. Such items are non-sale items that cannot be purchased by the user. When the item and another user's item compete with each other, control may be performed so that the performer wears the item with priority. The non-sold item may be a decoration, an effect, or a background image.

A specific action by a performer or user (for example, an action of throwing an object or in which direction and with which strength) is detected by the detection part of the smart watch 50 or the detection part of the smart device terminal 60a. It is not limited to what determines (detects) based on. For example, the determination may be made by calculating an inter-frame difference or a motion vector based on the video acquired by the camera.

As an example, instead of using an acceleration sensor or a gyro sensor to detect an operation performed by the user A, B, or C when presenting an item to the performers A, B, or C, for example, the operation is performed as follows. A camera having a video shooting function such as a web camera or a video camera is installed in front of the user. The camera here is, for example, a web camera provided integrally with a laptop personal computer, and as an example, a web camera or a video camera connected to a desktop personal computer. Also, as an example, a camera built in a smart device terminal. Then, the

user terminals

40, 60, 70, the server 20, or other devices calculate motion data for throwing the user's object based on the inter-frame difference of the frames constituting the video data, and based on the motion data Detects throwing motion. Alternatively, the motion vector of the object from the reference frame is detected, and the motion of throwing the user's object is detected. Thereafter, the track of the item and the item are displayed on the item position on the floor of the studio 10 and the display surface of the

user terminals

40, 60, and 70 so that the performers A, B, and C can recognize them.

In addition, the actions that the performers A, B, and C acquire the items that the users A, B, and C present to the performers A, B, and C are detected using the image analysis using the inter-frame difference and the motion vector described above. Also good. As an example, the above-described image analysis detects the actions of the performers A, B, and C squatting and bending when picking up items, and touching the item or touching the item position where the item is displayed. Thereafter, the performers A, B, and C can perform processing for attaching items and adding effects to the performers A, B, and C.

Furthermore, the operation of wearing items acquired by the performers A, B, and C may be detected using the image analysis or the like. As an example, the movement of the item to the heads of the performers A, B, and C can be detected by the image analysis.

Furthermore, an operation in which performers A, B, and C return items to users A, B, and C may be detected using the image analysis process. As an example, performers A, B, and C can be detected in the studio 10 by the image analysis.

That is, the actions performed by the performers A, B, and C can be detected by the image analysis process without using depth information, and the actions performed by the users A, B, and C can be detected by the image analysis process. .

・ As operation data, not all acceleration data, angle data, and angular velocity data, but at least acceleration data as motion data may be used. This is because the flying distance of the thrown item can be calculated from the acceleration data.

In the studio 10, the studio monitor 17 may be omitted. In this case, the ID object 76a of the user ID of the user who threw the item object may be displayed on the projector 16.

When there are too many items for the performers A, B, and C, the object of the item is displayed too much on the display screen of the studio monitor 17 and the

user terminals

40, 60, and 70. Similarly, the object of an item will be displayed too much by the projector 16 on the floor surface of the studio 10. In such a case, when the number of participating user terminals exceeds the threshold, the server 20 randomly extracts user terminals, and the object of the item from the extracted user terminals is displayed in the studio monitor 17 or the

user terminals

40, 60, 70. Is displayed on the display screen. Further, the server 20 displays the object of the item from the extracted user terminal on the floor surface of the studio 10 by the projector 16.

-Items as gifts from the users A, B, C to the performers A, B, C and return items from the performers A, B, C to the users A, B, C are, for example, from the users A, B, C. For performers A, B, and C, there is something that is simply “sent”. Also, as an example, “users A, B, and C” give a feeling that the users A, B, and C appreciate, congratulate, and support the performers A, B, and C. There is something. Further, as an example, there is one that “gives” the items (owned items) purchased by the users A, B, C to the performers A, B, C from the users A, B, C to the performers A, B, C. . Furthermore, as an example, there is one that “passes” items (ownership) purchased by users A, B, and C to performers A, B, and C.

As items, items presented to the performers A, B, C from the users A, B, C are simply displayed on the display surface of the

user terminals

40, 60, 70 and the display surface of the studio monitor 17. Also good. Such an item can be selected from the item selection object 72 shown in FIG. 5A, or image data that makes the performer's stage gorgeous by the user regardless of the item selection object 72, etc. It may be an item. Such items that are simply displayed on the display surface of the

user terminals

40, 60, and 70 or the display surface of the studio monitor 17 are charged when the user purchases an item, for example. In the case of a self-made item, it may be free.

The items of the wearing equipment and the effect items worn by the performers A, B, and C such as the headband may be further charged when the performers A, B, and C actually acquire the items. That is, you may make it charge twice, when the user A, B, C purchases an item and when the performers A, B, C acquire an item. In addition, you may make it charge only when performers A, B, and C acquire an item.

As a return from the performers A, B, and C to the users A, B, and C, it may be a simple display or production that can be seen only by the user who received the return. In this case, as an example, the operation of returning from the performers A, B, and C to the users A, B, and C may not be performed. In such a case, a real object such as a sign ball may not be mailed to the users A, B, and C.

The item may be a simple program including image data and moving image data created by the users A, B, and C using software. As an example of the simple program, there is an effect program such as an effect that makes the stage of the performer including the movement of the object gorgeous.

DESCRIPTION OF SYMBOLS 1 ... Live delivery system, 2 ... Network, 10 ... Studio, 11 ... Playback apparatus, 12 ... Speaker, 13 ... Microphone, 14 ... RGB camera, 15 ... Depth camera, 16 ... Projector, 17 ... Studio monitor, 18 ... Item, DESCRIPTION OF SYMBOLS 20 ... Server, 21 ... Audio IF, 22 ... RGB camera IF, 23 ... Depth camera IF, 24 ... Projector IF, 25 ... Display IF, 26 ... Database, 27 ... Data storage part, 28 ... Network IF, 29 ... Main memory , 30 ... control unit, 40 ... user terminal, 41 ... audio IF, 42 ... display IF, 43 ... network IF, 44 ... communication IF, 45 ... data storage unit, 46 ... operation IF, 47 ... main memory, 48 ... control 49, display unit, 50 ... smart watch, 51 ... sensor, 52 ... communication IF, 53 ... Data storage unit, 54 ... main memory, 55 ... control unit, 60 ... user terminal, 60a ... smart device terminal, 61 ... audio IF, 62 ... display IF, 63 ... operation IF, 64 ... sensor, 65 ... network IF, 66 ... Data storage unit, 67 ... Main memory, 68 ... Control unit, 69 ... Display unit, 70 ... User terminal, 71 ... Live video, 72 ... Item selection object, 72a ... Object, 72b ... Object, 72c ... Object, 72d ... object, 73 ... performer selection object, 73a ... first teaching object, 74 ... performer determination object, 74a ... second teaching object, 75 ... falling object, 76 ... acquisition object, 76a ... ID object, 77 ... return object, 78 ... Receiving object, 81 ... Effect Object, 82 ... box object.

Claims

A display device control unit for displaying on the display device the video of real space where the performer exists as a target of live distribution;
An acquisition unit for acquiring three-dimensional position information of the real space;
A detection unit for detecting a user action for a user to present an item to the performer;
Based on the three-dimensional position information acquired by the acquisition unit and user action information of the user action detected by the detection unit, the item position where the item should be arranged in the real space is calculated, and the calculated item A display control system comprising: an item display control unit for displaying the position on the real space so that the performer can recognize the position.
The display control system according to claim 1, wherein the user operation is detected by a smart device terminal held by the user.
The display control system according to claim 2, wherein the user action is an action in which the user throws an object.
The display control system according to claim 3, wherein the user action information is movement data of the user.
The display device control unit calculates an item object position where an item object corresponding to the item position is to be arranged in the video, and the video object and the item object are displayed at the item object position. The display control system according to any one of claims 1 to 4, wherein the display is displayed on the display device.
The display control system according to claim 5, wherein the display device control unit displays the item object on the video in real time.
When the detection unit determines that the performer is located within the range of the item presented to the performer from the user, the display device control unit causes the display device to display the item object in association with the performer. The display control system according to claim 5 or 6.
The display control system according to claim 7, wherein the range of the item is determined based on the three-dimensional position information acquired by the acquisition unit, user action information detected by the detection unit, and the type of the item.
The said display apparatus control part provides the said return item to the said user, when it determines with the said actor having performed the operation | movement for giving the return item to the said user, The any one of Claims 1 thru | or 8 is given. Display control system.
The display device control unit gives the return item to the user when it is determined that the user has performed a receiving operation, and provides the return item to the user when it is determined that the user has not performed a receiving operation. The display control system according to claim 9.
The display device control unit displays the display related to the return only on the display device of the user terminal that is the partner of the return item, and does not display it on the display device of the user terminal of other users. Display control system.
The display control system according to claim 1, wherein the item is an item purchased by the user.
Live video of the real space where the performer exists,
Obtaining the three-dimensional position information of the real space,
Detecting a user action for a user to present an item to the performer;
Based on the acquired three-dimensional position information and user action information of the detected user action, an item position where the item is to be arranged in the real space is calculated, and the performer recognizes the calculated item position. A display control method for displaying in the real space so that it can be performed.