JP2024071861A - Two-way live streaming system - Google Patents

Abstract

[Problem] When holding a live performance at a venue and simultaneously streaming the performance, the performers at the venue can only see a limited number of spectators or no spectators, making it difficult to motivate them to perform. Furthermore, the monitors at the venue cannot display all the viewers, and there is a possibility that some viewers may have problems that may disrupt the live performance. [Means for solving the problem] The video of the viewers is displayed on the monitors at the live venue, and the displayed viewer video is switched in a specified manner. Also, a two-way live streaming system is provided that can switch the display as described above or stop the output of viewer video that may disrupt the live performance by issuing a command to block the video signal of the viewer video. [Selected Figure] Figure 1

Description

The present invention relates to a distribution system that allows two-way interaction between the live performance venue and viewers watching the live performance from outside the venue.

In order to prevent the spread of COVID-19, which began around the end of 2019, restrictions were put on live performances, concerts, plays, lectures, and other gatherings where large numbers of people gather in one place. Singers and others held live performances without an audience and streamed them online. Since then, infection control measures have been implemented and restrictions are being relaxed, but the capacity of venues has decreased compared to before the COVID-19 outbreak due to the need to keep an appropriate distance between spectators (specifically, keeping spectators at least 1.5 meters apart, sitting one or two seats apart, keeping a distance between the stage and the audience seats, or separating them with transparent boards such as acrylic). For fans who cannot attend, live performances and other events are streamed online, and events are still being held in a mixed format of participation at the venue and participation through a viewing device such as a PC. Not only as a measure against infection, but also for fans who live in remote areas and find it difficult to go to the venue, or who cannot buy tickets to attend the venue due to the venue's capacity limit, internet streaming is becoming established as a new way to enjoy live performances.

Patent Document 1 discloses a method of live broadcasting without an audience, in which microphones and speakers at a live venue are connected via the internet to user terminals used by viewers to watch the performance. The method outputs the real-time audio responses of viewers from the user terminals to speakers at the live venue as analog audio signals, and notifies the live performers of the audience's responses in real time.

JP 2021-182696 A

However, the technology described in Patent Document 1 transmits only audio of the audience's reactions to the live venue, and does not disclose any video. As a result, there is a possibility that the performers will not be able to see the audience's reactions in real time during the live performance. With a no-audience live performance, all the performers can see is the empty auditorium seats, and even though there is audio from the audience, it is difficult for the performers to feel the reactions to their own performance, which can be said to lack excitement.

In general, remote live broadcasts distributed over the Internet involve displaying images from cameras on smartphones or tablet devices used by viewers located away from the venue (such as the viewer's home) on monitors at the venue. When there are many viewers watching live footage distributed over the Internet, there is an issue that it is not possible to display everyone on the monitors at the venue. In addition, it is unclear how to deal with viewers who cause trouble in some way (such as shouting or making strange noises, or emitting images or sounds that violate public order and morals) and who cause a nuisance to the progress of the live performance or to other audience members or viewers. There is also an issue of it being unclear how to deal with well-intentioned viewers who unintentionally cause feedback through the microphones or speakers at the live venue.

The present invention provides a two-way live distribution system that can select viewer video and, if not all viewers can be displayed on the monitor, can switch between viewer videos and block the video of inappropriate viewers.

In order to solve the above problems related to the two-way live distribution system, the present application provides:
As a first invention,
A monitor (A) is placed at the live venue and outputs the viewer's video from the venue terminal (C) described below;
A camera (B) is placed at the live venue to capture live footage and transmit it to a venue terminal (C) (not limited to a single terminal) described later;
a venue terminal (C) that is placed in a live venue and connected to the monitor (A) and the imaging device (B);
a live video transmission server (E) that transmits the live video from the venue terminal (C) via a network to a viewing terminal (D) that is used by a viewer to view the live video;
A viewer video transmission server (F) (which may be the same as the live video transmission server (E)) that transmits viewer video from the viewing terminal (D) to a venue terminal (C) at the live venue;
The present invention provides a two-way live distribution system comprising:

As the second invention, based on the first invention,
The venue terminal (C) provides a two-way live distribution system having a viewer video selection unit (G) that selects a viewer video to be output to the monitor (A) from among a plurality of viewer videos received from a viewing terminal (D) via a viewer video transmission server (F).

As the third invention, based on the second invention,
The hall terminal (C) has a viewer identification information storage unit (H) that stores viewer identification information for identifying a viewer (including the case of a viewing terminal (D)),
The viewer video selection unit (G) provides an interactive live distribution system configured to select a viewer video using viewer identification information.

As the fourth invention, based on the third invention,
The viewer identification information storage unit (H) provides an interactive live distribution system having an authority information storage means (J) that stores viewer identification information in association with authority information, which is information indicating the authority of the viewer identified by the viewer identification information.

As a fifth invention, based on any one of the second to fourth inventions,
The venue terminal (C) has a video signal cut-off command receiving unit (K) that receives a cut-off command from an operator of the venue terminal (C) to cut off a video signal of a viewer video from a viewing terminal (D) that is a transmission source of the viewer video output to the monitor (A),
The viewer video selection unit (G) provides a two-way live distribution system having a cut-off command execution means (L) for terminating the selection of the video signal of the viewer video for which a cut-off command has been received by the video signal cut-off command receiving unit (K).

Furthermore, the present invention provides an operating method for the two-way live distribution system of the first to fifth inventions, which is a computer, and an operating program that can be read by each of the methods. Each operating program may also be recorded on a recording medium.

The present invention, which has the above-mentioned main configuration, can provide a system.

Functional block diagram of the invention according to the first embodiment Functional block diagram 2 of the invention according to the first embodiment FIG. 1 is a flowchart showing the operation of the invention according to the first embodiment. Hardware diagram of the invention according to the first embodiment Functional block diagram of the invention according to embodiment 2 FIG. 11 is a flowchart showing the operation of the invention according to the second embodiment. Hardware diagram of the invention according to the second embodiment Functional block diagram of the invention according to embodiment 3 FIG. 11 is a flowchart showing the operation of the invention according to the third embodiment. Hardware diagram of the invention according to the third embodiment Functional block diagram of the invention according to the fourth embodiment FIG. 11 is a flowchart showing the operation of the invention according to the fourth embodiment. Hardware diagram of the invention according to the fourth embodiment Functional block diagram of the invention according to embodiment 5 FIG. 11 is a flowchart showing the operation of the invention according to the fifth embodiment. Hardware diagram of the invention according to the fifth embodiment Overall configuration of the present invention Example 1 of a screen of a viewing terminal (D) of the interactive live distribution system of the present invention Example 2 of a screen of the viewing terminal (D) of the interactive live distribution system of the present invention Example 3 of a screen of the viewing terminal (D) of the interactive live distribution system of the present invention Example 4 of a screen of the viewing terminal (D) of the two-way live distribution system of the present invention Monitor (A) screen example 1 of the two-way live distribution system of the present invention Monitor (A) screen example 2 of the two-way live distribution system of the present invention Monitor (A) screen example 3 of the two-way live distribution system of the present invention Monitor (A) screen example 4 of the two-way live distribution system of the present invention

<Prerequisites for this patent application>

<Prerequisites for the Description of All Embodiments>
<Hardware that can configure the present invention>
The present invention is, in principle, an invention that utilizes a computer, but at least a part of it is realized by software, hardware, or a combination of software and hardware. In this case, the software uses hardware resources to perform various calculations and realizes various functions through the required data and information. It can be said that information processing by software is specifically realized using hardware resources.

The hardware that realizes all or part of the constituent elements of the present invention is composed of the basic components of a computer, such as a CPU, memory, a bus, an input/output device, various peripheral devices, and a user interface. The various peripheral devices may include a storage device, an Internet interface, an Internet device, a LAN device, a Wi-Fi (registered trademark) device, a display, a display interface, a keyboard, a mouse, a speaker, a microphone, a camera, a video camera, a video, a video switcher, a video capture device, an audio mixer, a television, a CD device, a DVD device, a Blu-ray device, a USB memory, a USB memory interface, a removable hard disk, a general hard disk, a projector device, an SSD, a telephone, a fax, a copy machine, a printing device, a movie editing device, and various sensor devices.

The system does not necessarily have to be made up of a single housing, but may be made up of multiple housings connected through communication. The communication may be a LAN, a WAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared communication, or ultrasonic communication, and some of the systems may be installed across national borders.

<Fulfillment of the law of nature in all embodiments of the present invention>

The present invention functions through the cooperation of a computer and software. In this invention, live video and/or audio and viewer video and/or audio are transmitted by communication between the venue terminal (C), monitor (A), and filming device (B) installed at the live venue, the live video transmission server (E), the viewer video transmission server (F), and the viewing terminal (D) used by viewers watching at a location other than the live venue, via a general Internet line or a dedicated line. Since it includes processing unique to ICT, it is a so-called business model patent. From this perspective, if the present invention is judged based on the resources such as computers described in the claims and specification and the common general technical knowledge related to those matters, the present invention utilizes the laws of nature.

<Applicant's understanding of the significance of utilizing the laws of nature required by patent law>

The use of the laws of nature required by the Patent Act is required to ensure that the invention can be used industrially and usefully, based on the purpose of the law, from the viewpoint that the invention must have industrial applicability and contribute to the development of industry. In other words, it is required that the invention be industrially useful, that is, that the effect of the invention declared at the time of application can be reproduced with a certain degree of certainty by implementing the invention. From this perspective, the use of the laws of nature is interpreted as the function of each of the invention-specific matters (invention constituent elements) that constitute the invention to achieve the effect of the invention being achieved by using the laws of nature. Furthermore, the effect of an invention is sufficient if it has the potential to provide a certain usefulness to users who use the invention, and should not be viewed from the perspective of how the user feels or thinks about that usefulness. Therefore, even if the effect obtained by the performers performing at the live venue, the distributor who distributes the live video and/or audio, the viewers who watch the distributed live video on a viewing terminal, and the technician (who may be the same as the distributor) who receives the viewer video from the viewing terminal, etc., from this two-way live streaming system is a psychological effect (such as a sense of security), the effect itself is not subject to the judgment of the presence or absence of the required use of the laws of nature.

＜Hardware configuration＞

Figure 4 is a diagram showing an example of the hardware configuration in this embodiment 1. The hardware configuration of the two-way live distribution system, which is the computer in all embodiments of the present invention, will be described with reference to Figure 4.

As shown in this diagram, the computer is composed of a chipset, CPU, non-volatile memory, main memory, various buses, BIOS, various interfaces such as USB and LAN, a real-time clock, etc., which are configured on a motherboard. These cooperate with an operating system, device drivers (for various devices such as various interfaces such as USB, cameras, microphones, speakers or headphones, displays, video cameras, videos, video switchers, video capture devices, audio mixers, etc.), various programs, etc., and also cooperate with various external input/output devices (keyboards, mice, USBs, communication devices (LANs, etc.), monitors, microphones, cameras, etc.) connected to the computer. The various programs and data that constitute the present invention are configured to efficiently utilize these hardware resources to execute various processes.

≪Chipset≫

A "chipset" is a set of large-scale integrated circuits (LSIs) mounted on a computer's motherboard that integrates a communication function between the CPU's external bus and a standard bus that connects memory and peripheral devices, in other words a bridge function. There are cases where a two-chipset configuration is used, and cases where a one-chipset configuration is used. The north bridge is located closer to the CPU and main memory, and the south bridge is located farther away and serves as an interface with the relatively slow external I/O. The following description will be given of a two-chipset configuration, but the CPU and north bridge can be integrated into a single chip equivalent to the south bridge, and similar effects can be obtained by integrating the CPU with both chipsets.

(North Bridge)
The north bridge includes a CPU interface, a memory controller, and a graphics interface. Most of the functions of a conventional north bridge may be performed by the CPU. The north bridge is connected to the memory slot of the main memory via a memory bus, and is connected to the graphics card slot of the graphics card via a high-speed graphics bus (AGP, PCI Express).

(South Bridge)
The south bridge is connected to the PCI interface (PCI slot) via a PCI bus, and is responsible for I/O functions and sound functions with the ATA (SATA) interface, USB interface, Ethernet interface, etc. Incorporating circuits that support PS/2 ports, floppy disk drives, serial ports, parallel ports, and ISA buses, which do not require or are impossible to operate at high speed, will hinder the speed of the chipset itself, so they may be separated from the south bridge chip and placed in a separate LSI called a super I/O chip. A bus is used to connect the CPU (MPU) to peripheral devices and various control units. The bus is connected by the chipset. The memory bus used to connect to the main memory may instead adopt a channel structure to increase speed. A serial bus or a parallel bus can be used as the bus. While a serial bus transfers data one bit at a time, a parallel bus transmits the original data itself or multiple bits extracted from the original data as a single block and transmits it simultaneously over multiple communication paths. A dedicated line for the clock signal is set up in parallel with the data line to synchronize the data demodulation on the receiving side. It is also used as a bus to connect the CPU (chipset) to external devices, and includes GPIB, IDE/(Parallel) ATA, SCSI, PCI, etc. Since there is a limit to how fast it can be made, in PCI Express, an improved version of PCI, and Serial ATA, an improved version of Parallel ATA, the data line can be a serial bus.

≪CPU≫

The CPU sequentially reads, interprets, and executes a sequence of instructions called a program stored in the main memory, outputting information consisting of signals to the main memory. The CPU functions as the center of calculations within the computer. The CPU is composed of a CPU core, which is the center of calculations, and its peripheral parts, and includes registers, cache memory, an internal bus connecting the cache memory and the CPU core, a DMA controller, a timer, and an interface with a connection bus to the north bridge. A single CPU (chip) may have multiple CPU cores. In addition to the CPU, processing may also be performed by a graphic interface (GPU) or FPU. Although the embodiment is described as a two-core type, this is not limiting. Programs can also be built into the CPU.

≪Non-volatile memory≫

(HDD)

The basic structure of a hard disk drive consists of a magnetic disk, a magnetic head, and an arm on which the magnetic head is mounted. SATA (previously ATA) can be used as the external interface. A high-performance controller, such as SCSI, is used to support communication between hard disk drives. For example, when copying a file to another hard disk drive, the controller can read the sectors and transfer them to the other hard disk drive for writing. At this time, the host CPU's memory is not accessed. Therefore, there is no increase in the CPU load.

(Flash memory)
Currently, two types of flash memory are commonly used: NAND flash memory and NOR flash memory. Although there are pros and cons regarding read/write speed, NAND flash memory is more advantageous for high integration and is used for data storage applications. Compared to hard disk drives, NOR flash memory is smaller because it has no moving parts and does not generate vibration or noise during operation. However, the cost per capacity has not yet fallen to the point where it can replace hard disk drives. If the amount of data to be stored is not large, the device can be made smaller and it has the advantage of being more resistant to shocks.

≪Main memory≫

The CPU directly accesses and executes various programs in the main memory. The main memory is a volatile memory and uses DRAM. Programs in the main memory are expanded from the non-volatile memory to the main memory upon receiving a program start command. The CPU then executes the program according to various execution commands and execution procedures within the program.

≪Operating System (OS)≫

The operating system is used to manage the resources available to applications on the computer, to manage various device drivers, and to manage the computer itself (the hardware). Small computers sometimes use firmware as the operating system.

≪BIOS≫

The BIOS causes the CPU to execute procedures for starting up the computer hardware and running the operating system, and is most typically the hardware that the CPU reads first when it receives a computer startup command. The address of the operating system stored on the disk (non-volatile memory) is written here, and the operating system is sequentially deployed in the main memory by the BIOS deployed in the CPU and becomes operational. The BIOS also has a check function for checking the presence or absence of various devices connected to the bus. The results of the check are stored in the main memory and are made available to the operating system as appropriate. The BIOS may be configured to check external devices, etc. The computer may be configured to start up using a successor UEFI (Unified Extensible Firmware Interface) that has similar functions instead of the BIOS. The above is the same for all embodiments.

As shown in FIG. 4, the present invention can basically be configured with a general-purpose computer program and various devices. The computer basically operates by loading a program recorded in non-volatile memory into main memory, and then executing processing using the main memory, CPU, and various devices. Communication with devices is performed via an interface connected to a bus line. Possible interfaces include a display interface, a keyboard, and a communication buffer. Below, an embodiment of the present invention will be described with reference to illustrated examples.

<Overview of embodiment 1> Mainly claim 1
The interactive live streaming system of embodiment 1 is composed of a monitor (A), a filming device (B), a venue terminal (C), and a viewing terminal (D) used by viewers for viewing, all placed at a live venue, a live video transmission server (E) that transmits live video from the live venue to the viewing terminal (D), and a viewer video server (F) that transmits viewer video from the viewing terminal (D) to the venue terminal (C).

<Functional Configuration of First Embodiment>
Fig. 1 is a diagram showing an example of the configuration of a two-way live distribution system according to the present embodiment. As shown in the figure, the two-way live distribution system (0100) according to the first embodiment includes a monitor (A) (0101), a camera (B) (0102), a venue terminal (C) (0103), a viewing terminal (D) (0104), a live video transmission server (E) (0105), and a viewer video transmission server (F) (0106). In Fig. 1 and Figs. 2, 5, 8, 11, and 14 which are similar to Fig. 1 in the following explanation, the parts surrounded by dotted lines are installed at the live venue, and the parts surrounded by solid lines and indicated by a four-digit code consisting of the figure number +00 are the two-way live system of the present invention.

Each of the functional blocks constituting the interactive live streaming system described below can be realized by hardware, software, or both. Specifically, if a computer is used, the functional blocks include hardware components such as a CPU, main memory, GPU, image memory, graphics board, bus, or secondary storage device (hard disk, non-volatile memory, storage media such as CDs and DVDs, and drives for reading these media, etc.), input devices such as operation buttons used to input information, keyboards, mice, touch panels, electronic pens used solely for touching touch panels, joysticks or joystick-like pointer position input devices, input devices such as microphones and cameras, output devices such as monitors and speakers, communication devices, devices for synthesizing, processing, selecting and switching audio and video signals, and other external peripheral devices. , and interfaces for external peripheral devices, GPS receiving interfaces, GPS computing devices, gyro sensors, acceleration sensors, rotation detection sensors, signal processing devices for these sensors, cameras, image file processing circuits, speakers, microphones, audio file processing circuits, communication interfaces, barcode readers, electronic card readers, POS terminals, biometric authentication devices such as face authentication devices, encryption devices, fingerprint authentication devices, palm print authentication devices, and retina authentication devices, video cameras, videos, video switchers, video capture devices, audio mixers, driver programs for controlling these hardware devices, and other application programs. In particular, smartphones, tablet devices, mobile phones, smart watches, personal computers, data center server devices, wired and wireless networks, and interfaces are used.

The CPU performs calculations according to the programs deployed in the main memory, processing and storing data input from input devices and other interfaces and held in memory and hardware, and generating commands to control the hardware and software. Here, the above programs may be realized as multiple modularized programs, or two or more programs may be combined into one program.

The present invention can also be partially configured as software. Furthermore, storage media on which such software is recorded are naturally included within the technical scope of the present invention (this is not limited to this embodiment, but is the same throughout this specification).

The above functional blocks are merely examples for implementing the present invention, and functions may be omitted or new functions may be added as appropriate to the extent that does not contradict the problems to be overcome by the present invention and its effects. The same applies to the following embodiments.

<Description of Configuration of First Embodiment>
<Embodiment 1 Monitor (A) (0101)>
The "monitor (A)" (0101) is placed at the live concert venue and is configured to output viewer images from the venue terminal (C) described below.

<Embodiment 1: Monitor (A): Live venue>
A "live venue" includes a small venue with a capacity of 100 people or less, an outdoor stage with a capacity of 10,000 people, a sports stadium such as a martial arts hall or baseball stadium with a capacity of tens of thousands, and a horse racing track. One or more performers who play or sing music, give a lecture, perform a play, or play a sport (including martial arts such as sumo) are located on a "stage" (generally a place that is one step higher so that the audience can easily see, or conversely, the lowest part of a bowl-shaped or stepped venue) that is separated from the place where the later-described imaging device (B) is installed, which captures the video including the live audio to be distributed to the audience or viewers. A live performance is when the audience in the venue watches the performers on the stage play or sing, give a lecture, or perform a play (in the case of sports, the competition is held in a competition space such as a court or ring) according to the schedule of the performance, and live distribution is when the video of the live performance is transmitted in real time to viewers outside the venue.

<Embodiment 1: Monitor (A): Display>
The monitor (A) may be a single 50-inch diagonal display using a liquid crystal panel or an organic EL panel, a combination of four or nine displays to operate as one large display, a combination of LEDs to form a large display, or a projector to project images onto a screen or wall. One or all of the walls of a live venue may be configured to function as a monitor (A). Alternatively, a panel of about 32 to 40 inches may be assigned to each audience member and placed at the audience seating position, so that the performers on the stage performing the live performance may see the audience actually sitting in the audience seats. At least one of these monitors (A) is used to display the viewer's video output from the venue terminal (C) described below.

The main purpose of the viewer image displayed on monitor (A) is to be shown to the performers on stage, but if there is an audience in the venue, it is preferable to install it so that it can be seen by the audience there as well, to show that there are many viewers outside the venue as well. If monitor (A) is the equivalent of a 40-inch widescreen monitor, this is the maximum size that can be used to distinguish the size of about 4 to 5 people vertically and 5 to 6 people horizontally. If monitor (A) is full HD (1920 horizontal x 1080 vertical), the resolution of one viewer's image will be about 320 horizontal x 216 vertical.

<Embodiment 1: Monitor (A): Viewer Image>
"Viewer video" is basically real-time video (including a delay of a few seconds; hereinafter in this specification, real-time is used to mean a delay of a few seconds as well) of a viewer viewing live video transmitted from a viewing terminal (D). Viewer video includes not only video signals but also audio signals. Viewer video may not only be real-time video of the viewer, but may also be still images, video, audio only, text data, etc. stored in the viewing terminal (D).
On the viewing terminal (D),
a viewer image candidate storage unit that stores text data, image data, audio data, etc., which are not images of viewers who are viewing live images;
a transmission viewer image selection unit that allows the viewer to select a viewer image to be transmitted to the live venue from among the viewer's image taken by a built-in or external camera of the viewing terminal (D) and other data (such as text data, image data, and audio data) stored in the viewer image candidate storage unit of the viewing terminal (D) while viewing the live video;
a viewer image transmitting unit for transmitting the selected viewer image to a viewer image transmitting server (F);
This can be achieved by configuring the device to have the following:
The viewer can transmit the text data, image data, and audio data that he/she has created while watching the live video. This is effective when the viewer does not want to reveal his/her real face or appearance to others and wants to display another image (such as a portrait of himself/herself, a deformed face, an image of the viewer's pet such as a cat or dog, or an image of the viewer's car) on the monitor (A) instead of him/herself.

The viewer video transmitted from the viewing terminal (D) to the viewer video transmission server (F) is information transmitted over an Internet line or the like, and the viewer video is contained in the payload of the packet transmitted and received over the Internet line, and the payload or header can be configured to store viewer identification information described in embodiment 3 below that identifies the viewer appearing in the viewer video. Authorization information associated with the viewer identification information described in another embodiment 4 below may also be included. The header or payload of the packet also stores the address of the venue terminal (C), which is the final destination.

<Embodiment 1: Monitor (A): Example of a viewer's image other than the viewer's own image>
As another example other than the viewer's video and/or audio signals, a moving image of a hand or arm may be transmitted as the viewer's video, and actions such as waving at the start of a live performance or clapping at intervals or at the end of a performance may be displayed to convey goodwill, praise, or gratitude to the performers. As an example of a still image, an image of a flower or bouquet may be transmitted and displayed on the monitor (A) at the start or end of a live performance to convey actions or gratitude. As an example of text data, comments to the performers or the organizer of the live performance may be transmitted in text format as the viewer's video. In the case of text, it may be displayed only in a specified viewer video display section of the monitor (A) at the live performance venue, or it may be displayed on the entire monitor (A) and superimposed on other viewer videos. A viewer text data output selection means may be provided in the viewer video selection unit (G) of the venue terminal (C) to select the output method to the monitor (A).

<Embodiment 1: Monitor (A): Example of viewer video: Audio signal>
The viewer video may be a video signal and an audio signal transmitted from the viewing terminal (D) as described above, or may be only an audio signal, so the monitor (A) may be equipped with not only a display device for displaying the video but also a speaker for outputting audio. Alternatively, the monitor (A) may also be equipped with a speaker for monitoring only the audio signal. The audio signal may be converted into text by a voice recognition program, which is a known technology, and displayed on the monitor (A), which is a display device. The audio signal contained in the viewer video may be converted into text by a viewer video/audio signal text data conversion means provided in the viewer video selection unit (G) of the hall terminal (C), and the display method of the monitor (A) may be selected and displayed by a viewer text data output selection means.

<Embodiment 1: Imaging device (B) (0102)>
The "camera device (B)" (0102) is arranged at a live performance venue and is configured to capture live images and transmit them to a venue terminal (C) (not limited to a singular terminal) described below.

<Embodiment 1: Imaging device (B): Arrangement>
At least one camera (B) is provided. In the case of a group consisting of multiple performers, it is advisable to combine multiple cameras (B) such as a camera (B) that takes a picture of the whole group, a camera (B) that takes close-up pictures of each performer, and other cameras (B) that have a fixed angle of shooting and do not move from a preset location, and a camera (B) that moves to a suitable location and changes the shooting angle. The camera (B) can be moved by sliding it on wheels attached under the support rack of the camera (B), or the cameraman can carry the handheld camera (B) himself, or it can be installed at the end of a crane, or it can be installed on a drone and moved freely around the venue to take pictures.

In addition, viewers outside the venue, or those far from the stage where the performers are, can watch the video from the angle they want on their viewing terminal (D).
On the viewing terminal (D),
A live video camera selection unit that selects a camera (B) that is shooting at an angle that the viewer wants to see;
a live video acquisition unit that receives and acquires live video transmitted from a live video transmission server (E);
A live video output unit that outputs the live video to a display connected to (or built into) the viewing terminal (D),
Furthermore, the live image acquisition unit may be configured to include a live image acquisition means for each image capture device that acquires live images captured by the selected image capture device (B).

<Embodiment 1: Imaging device (B): camera, resolution>
The images may be taken using a general consumer video camera, a smartphone, a digital single-lens reflex camera capable of taking videos, or a professional device such as a television camera for broadcasting or a camera for filming. The higher the resolution of the video image to be taken, such as 720 or 1080, rather than 480, the finer the image can be taken. 4K (2160 vertical), which is becoming more common as a television receiver, has better resolution, but when distributed over an Internet line, the amount of data per hour becomes enormous. Therefore, it is advisable to select an appropriate resolution based on the trade-off between the bandwidth of the line used for distribution and the image quality obtained. Alternatively, the viewer may be allowed to select the resolution using the viewing software on the viewer's side depending on the viewer's environment.

<Embodiment 1: Imaging device (B): Live video>
"Live footage" refers to footage captured by a camera (B) of performers playing music, singing, playing in a park, performing in a theater, dancing, or playing sports (including martial arts such as sumo) at a venue where a live performance is held, and includes video and audio signals. The present invention is effective even if the distribution of live footage is limited to audio signals.

<Embodiment 1: Venue Terminal (C) (0103)>
The "venue terminal (C)" (0103) is placed at the live venue and is configured to be connected to the monitor (A) and the image capture device (B) (not limited to a single one). The venue terminal (C) transmits live images captured at the live venue, receives viewer images, and outputs them to the monitor (A) at the live venue. The detailed flow is as follows.

The venue terminal (C) (0103) shown in FIG. 1 receives live video from a camera (B) (0102) installed at the live venue, and transmits the live video to a live video transmission server (E) (0105). The live video transmission server (E) (0105) transmits the received live video to a viewing terminal (D) (0104). The viewing terminal (D) (0104) displays the live video on a display connected to it, and the viewer watches the live video displayed on the display. The viewing terminal (D) (0104) transmits viewer video of the viewer watching the live video to a viewer video transmission server (F) (0106) (which may be the same as the live video transmission server (E) (0105)). The viewer video transmission server (F) (0106) transmits the received viewer video to the venue terminal (C) (0103). The venue terminal (C) (0103) outputs the received viewer video to a monitor (A) (0101) installed at the live venue. One or more viewer videos output by the venue terminal (C) to the monitor (A) may be transmitted to a live video transmission server (E) (0105) so that the viewer can view them on a viewing terminal (D).

<Embodiment 1: Venue Terminal (C): Configuration of Venue Terminal (C)>
1, the venue terminal (C) (0103) is configured to handle both the live video captured by the imaging device (B) (0102) and the viewer video received from the viewer video transmission server (F) (0106) in one device, but multiple venue terminals that handle both signals may be provided. Alternatively, instead of handling both videos in one device, the venue terminals may be divided into a venue terminal (C) dedicated to transmitting the live video to the live video transmission server (E) and a venue terminal (C) dedicated to receiving the viewer video from the viewer video transmission server (F), with at least one of each provided.

<Embodiment 1: Venue terminal (C): Arrangement of viewer video when outputting to monitor (A)>
The venue terminal (C) outputs the viewer video received from the viewer video transmission server (F) to the monitor (A) arranged at the live venue. As explained above, it is possible to output multiple viewer videos to the monitor (A). When multiple viewer videos are arranged on the monitor (A) in tiles (grids) of the same size, for example, they must have the same aspect ratio and, if possible, the same resolution. If the aspect ratios are different, even if the resolution is adjusted automatically when arranging, gaps will occur vertically or horizontally when multiple viewer videos are arranged and output on the monitor (A). When the aspect ratio is forcibly matched, the viewer video will be trimmed or transformed by stretching it vertically or horizontally. When a specific viewer video among the viewer videos arranged in tiles is enlarged and displayed, it can be arranged well by making the size equal to n times the size of the other viewer videos, for example, four or nine times the size of the other viewer videos. The screen of one monitor (A) may be treated as a screen showing a room, for example, and the viewer videos may be displayed overlapping appropriately as if they were people in the room. In order to arrange the viewer video so as to suit the display on the monitor (A), the venue terminal (C) may be configured to have a viewer video arrangement unit that arranges at least one viewer video so as to suit the display on the monitor (A). Alternatively, a video signal processing device having the viewer video arrangement unit may be provided between the venue terminal (C) and the monitor (A) separately. The viewer video arrangement unit may have a function that can change the output pattern format, for example, arranging 16 viewer videos of 4×4 and outputting them to the monitor (A), and then outputting them as 9 viewer videos of 3×3 when outputting next. The viewer video arrangement unit may also be provided with a viewer video arrangement change means that changes the arrangement of the viewer video output to the monitor (A). This is used when a viewer video displayed at the edge of the monitor (A) is moved to the center of the monitor (A). For example, when the viewer video is output as 4×4 in the above example, the viewer video being output to the monitor (A) may be changed to 3×3, or the number of viewer videos to be output may be increased.

<Embodiment 1: Venue terminal (C): Viewer image: Change of background>
Since the location where each viewer watches the live video varies from viewer to viewer, the viewer video, which is basically a video of the viewer watching live, shows various aspects of the viewer's surroundings and background. Therefore, if the video is output in a tiled manner to the monitor (A), the viewer's background in the viewer video will appear chaotic and distorted in color. Either the venue terminal (C) that outputs to the monitor (A) or the viewing terminal (D) that transmits the viewer video can be provided with a viewer video background replacement processing unit that erases areas other than the viewer in the viewer video or replaces them with a background specified by the live event organizer. If the replacement processing is performed at the viewing terminal (D), the viewer may choose to perform the processing. The viewer may select from several types of backgrounds provided in advance by the live event organizer or this system, and output the viewer video to the monitor (A) with the same background. If the processing is performed at the venue terminal (C), the processing is performed after the viewer video to be output to the monitor (A) is selected. The background to be unified can be changed for each unit output to the monitor (A).

<Embodiment 1: Venue terminal (C): Other devices: video mixer, video switcher>
When multiple image capture devices (B) are installed in a live venue, normally not all live images captured by all of the image capture devices (B) are distributed, but instead the organizer or live venue staff selects and switches between the live images that they deem appropriate at any given time, and transmits them to the live image transmission server (E). For this reason, the venue terminal (C) has a function such as a video switcher or video mixer that is convenient for image switching, or if such a device is separately provided between the venue terminal (C) and the live image transmission server (E), live distribution can be performed well. The venue terminal (C) is not limited to a single type of device (PC, etc.), but may be composed of a server device and a control terminal (PC, etc.) that communicate with the image capture devices (B), monitors (A), live image transmission server (E), viewer image transmission server (F), etc., or may further be configured as the venue terminal (C) including other devices such as the video switcher and video mixer.

<Embodiment 1: Venue terminal (C): Angle selection (selection of imaging device (B))>
The live images photographed by a plurality of photographing devices (B) can be distributed over one line, or a distribution line can be assigned to each photographing device (B), and the viewing terminal (D) can be configured so that the viewer can select which line to use for viewing (which photographing device (B)'s live image to view).
To do this, the viewing terminal (D)
A live video shooting device selection unit that selects a plurality of shooting devices (B) that are placed at different locations in the live venue and shoot live video from different angles;
A live video acquisition unit that acquires live video,
Furthermore, the live image acquisition unit can be configured to have a live image acquisition means for each imaging device that acquires live images at a selected angle.

<Embodiment 1: Venue terminal (C): 3D live video>
The system can be configured such that the images of the performers taken from different angles by the multiple image capture devices (B) are acquired, a stereoscopic image of the performers and/or at least a part of the live venue is generated, and the viewer can view the generated stereoscopic live video of the live venue from any position desired by the viewer (the organizer may set a range restriction in advance, such as prohibiting viewing from a position equivalent to the stage). For this purpose, a stereoscopic live video generation unit is provided that acquires images of the performers taken from different angles by the multiple image capture devices (B) and generates a stereoscopic image of the performers and/or at least a part of the live venue. The stereoscopic live video generation unit may be provided in the venue terminal (C) or the live video transmission server (E), or a dedicated server device may be provided between the live video transmission server (E) and the venue terminal (C) to perform the processing. Since the venue terminal (C) also has other processing and the load of the stereoscopic video generation processing is heavy, it is preferable to provide the live video transmission server (E) or a dedicated server device.

<Embodiment 1: Venue terminal (C): 3D live video: Configuration example (dedicated server device installation example)>
The following will explain an example of a configuration in which a dedicated server device is provided for utilizing the 3D live video. Note that the 3D live video generating unit may be provided in the live video transmission server (E) or the venue terminal (C) as described above.
a stereoscopic live video generating unit that acquires images of the performers captured from different angles by a plurality of image capturing devices (B) and generates a stereoscopic image of the performers and/or at least a portion of the live venue;
a stereoscopic live video storage unit for storing the generated stereoscopic live video;
a stereoscopic live image transmitting unit that transmits the generated stereoscopic live image to a live image transmitting server (E) as a live image;
in a dedicated server device,
A live video shooting device selection unit that acquires the viewer's selection to view the stereoscopic live video (selection is made by pressing a stereoscopic live video selection button (display of 3D, etc.) displayed on the viewing screen (dedicated application or web browser) of the viewing terminal (D));
a live image acquisition unit that acquires the generated three-dimensional live image as a live image;
a live image output unit that outputs the acquired three-dimensional live image as a live image;
a stereoscopic live video viewing position acquisition unit that acquires stereoscopic live video viewing position information, which is position information indicating from what position and at what angle a viewer wants to view the live video in a live venue;
a stereoscopic live video viewing position information output unit that outputs the acquired stereoscopic live video viewing position information to a stereoscopic live video generating unit of the dedicated server device;
in a viewing terminal (D),
Furthermore, the three-dimensional live video generating unit of the dedicated server device is configured to generate and output a three-dimensional live video based on the received three-dimensional live position information, so that the three-dimensional live video generated at the viewer's desired angle from the viewer's desired position in the live venue is output to a display connected to (or built-in) the viewing terminal (D), and the viewer can view the three-dimensional live video he or she desires on the viewing terminal (D). When viewing the generated three-dimensional live video, VR goggles or AR glasses may be used to obtain a more realistic feeling. When reflecting the viewer in the live venue in the three-dimensional live video (when such authority information has been obtained: an example of embodiment 4), an image prepared in advance or an avatar image obtained by motion capturing the viewer while viewing may be transmitted as the viewer video.

<Embodiment 1: Venue terminal (C): Live video: Video that is not actual video>
The performers who play, sing, play in the park, perform in a play, dance, play a sport (including martial arts such as sumo) at the live venue may be non-existent performers (CG characters such as virtual idols), or real and non-existent performers may be displayed together to work together. Instead of the non-existent performers, a CG character tracing the movements of the real performer may be displayed, and the audio may be sent to the live video transmission server (E) as is or after conversion using a voice changer (e.g., changing a male voice to a female voice, or raising or lowering the pitch). In these cases, the video (including the audio) of the CG character tracing the non-existent or real performer is not displayed at the live venue, but is synthesized with the live video captured by the imaging device (B) at the venue terminal (C) and then sent to the live video transmission server (E). Alternatively, the image projected on a screen on the stage of a live concert venue and the voice of the non-existent performer or the CG character output from the speakers in the venue may be captured by a camera (B) and a microphone and transmitted. The image of the non-existent performer or the CG character tracing the real performer can be realized by known technology, and the image can be combined with the real performer or the live concert venue by known technology.

<Embodiment 1: Venue terminal (C): Private viewer video (text data)>
A plurality of venue terminals (C) may be prepared, and at least one of the venue terminals (C) may be configured not to output the viewer video to the monitor (A). (The other venue terminals (C) are configured to output the viewer video to the monitor (A).) In FIG. 2, two venue terminals (C) are installed (not limited to two), one (0203a) is connected to the monitor (A) (0201), and the other venue terminal (C) (0203b) is not connected to the monitor (A). When a viewer wants to send a text message directly to a performer, the viewer selects the venue terminal (C) (0203b) that is not connected to the monitor (A) that is seen by many people in the venue and transmits the text. This can be achieved by configuring the viewing terminal (D) to have a viewer video destination selection unit that directly specifies the destination venue terminal (C).
Or to the viewing terminal (D),
a transmission viewer image selection unit that selects a viewer image (including a type of viewer image) to be transmitted;
a viewer image transmitting unit for transmitting the selected viewer image to a viewer image transmitting server (F);
having
In addition, the viewer video transmission section
The system can also be configured to have a private viewer image destination selection means that automatically selects a venue terminal (C) (0203b) that is not connected to the monitor (A) as the destination when a text message that the viewer wishes to convey directly to the performer is selected (not shown) as the viewer image.

<Embodiment 1: Venue terminal (C): Flow of private viewer video (text data)>
After the viewer transmits the text message that he/she wants to directly convey to the performer from the viewing terminal (D) as a viewer video, the flow is as follows. The viewer video transmission server (F) (0206) that receives the viewer video distributes the destination venue terminal (C) according to the destination included in the viewer video (the viewer video transmission server (F) may have a viewer video destination selection unit that selects the venue terminal (C) from the destination included in the viewer video). The text message that is to be directly conveyed to the performer in private is transmitted to the venue terminal (C) (0203b). The venue terminal (C) (0203b) that receives the viewer video that is the text message to be directly conveyed to the performer outputs it privately to, for example, a monitor that monitors the live content that only the performer can see. As another example, if each performer wears an earphone for a wireless receiver, the operator of the venue terminal (C) or the like will read the text message orally only to the target performer using the earphone. If it is OK for others to read it, the moderator or organizer may read it out at times such as when speakers change or during breaks, or during specially set time slots.

<Embodiment 1: Viewing Terminal (D) (0104)>
The "viewing terminal (D)" (0104) is configured to receive and view live video from the venue terminal (C) via the network from the live video transmission server (E).

<Embodiment 1: Viewing terminal (D): aspect>
As the viewing terminal (D) used by the viewer for viewing, it is considered that a smartphone, a tablet terminal, or a PC (preferably with a camera) is often used. Alternatively, it can be viewed using a smart TV. Alternatively, the image of the smartphone or tablet terminal may be mirrored to be displayed on a TV or monitor, or may be displayed on a TV screen or the like via a signal conversion device. Each viewing terminal (D) has a display unit such as a display that displays live images, a speaker or earphone terminal for outputting the sound included in the live images, and a Bluetooth (registered trademark) communication device (not shown) for connecting speakers, earphones, etc.

The viewing terminal (D) has a communication device (not shown) for receiving the live video distribution and transmitting it to the viewer video transmission server (F) as described below. Communication is performed via wireless communication lines for mobile phones such as 4G or 5G, Wi-Fi (registered trademark), or via an Internet line connection via a wired LAN or WAN depending on the specifications of the viewing terminal (D). Furthermore, the viewing terminal (D) has a camera and a microphone for acquiring viewer video, which is video and audio captured of the viewers watching and reacting to the live video, as described below. The camera and microphone do not have to be built into the viewing terminal (D) but may be external. Built-in integrated types are preferable as they are easier to handle.

<Embodiment 1: Viewing terminal (D): Use at a live venue>
Not only viewers outside the venue (such as at home in a remote location), but also spectators in the venue may use the viewing terminal (D) to view the video. If the venue is not small but large enough to accommodate tens of thousands of people, spectators seated far from the stage cannot clearly see the performers' faces with their eyes. For artistic effects and direction, the performers' figures may be projected on a large monitor installed behind the stage, but it is not clear whether the performers, of whom the audience is a fan, will be displayed on the large monitor. Spectators in the venue may use the viewing terminal (D) to view live video of the performers they want to see.

<Embodiment 1: Viewing terminal (D): Example of viewing live video>
18 to 21 are examples of screens when a viewer uses a viewing terminal (D) to view live video. The present invention is not limited to the following examples. The following describes an example in which a dedicated app that runs on a viewing terminal (D) in the form of a tablet terminal or smartphone is used, but it is also possible to access a website provided by this system via the Internet using a browser, purchase tickets and make payments on the site, and view live video. FIG. 18 is the top screen of this system. Four buttons are arranged: "Performance Schedule,""Login,""NewRegistration," and "Member Information." When using this system for the first time, "New Registration" is performed.

<Embodiment 1: Viewing terminal (D): Application use: New registration>
Although the new registration screen itself is not shown, the name, age, contact information (address, telephone number, email address), and desired password are registered, similar to the member registration of a general membership site. In addition, the name of the artist who is a fan is entered, and if the member is a fan club member, the member information is registered. These are examples of the viewer's attribute information. When registration begins, a confirmation email may be sent to the mobile phone SMS or email address, and the email may be configured to guide the viewer to the registration page. Once everything is registered, a membership number is issued. The membership number is an example of viewer identification information, which will be described in embodiment 3 below. A server device for storing member information may be provided separately, and a viewer identification information storage unit for storing viewer identification information may be provided in the server device. The viewer identification information stored in the viewer identification information storage unit of the server device is stored as viewer identification information in at least the viewer identification information storage unit (H) described later of the venue terminal (C) of the live performance that the viewer plans to watch. In addition, the viewer identification information storage unit (H) may be configured to have a viewer attribute information storage means for storing the viewer's attribute information. It is preferable to register the information device used at the time of registration or the device that received the confirmation email as the viewing terminal (D) for watching the live video. In this case, it is advisable to automatically acquire device-specific information, such as the communication cookie used at the time of registration or the MAC address of the information device, into the system and store it in association with the membership number (including the viewer's attribute information). The viewer identification information storage unit (H) in the embodiment described below can be provided with a viewing terminal identification information storage means, which is information for identifying the viewing terminal (D). After acquiring the membership number and becoming a member, the next step is to select and purchase a viewing ticket.

<Embodiment 1: Viewing terminal (D): Use of app: Purchase of ticket>
When the "Performance Schedule" button is pressed, the screen moves to a list of performance schedules as shown in FIG. 19. In FIG. 19, the performance schedule for one week from July 11 to 17 is shown. Although FIG. 19 shows a case where only one performance is held per day, multiple performances may be displayed. In that case, the performance schedule for one week is described over multiple pages. In FIG. 19, ticket A for the Hibiya Music Festival on July 12 is selected for purchase (the display of the corresponding part changes, and the text color also changes to white). There are two types of tickets for the Hibiya Music Festival. Although only the price is shown, ticket B is a general viewing ticket, and ticket A is a ticket that has the right to interact for a short time with the artist of which the viewer is a fan through the viewing terminal (D) during viewing. Such differences in viewing methods depending on the viewer are one example of authority information that will be described in the following embodiment 4. In the example of Figure 19, the viewer identification information of a viewer who purchased an A ticket for the Hibiya Music Festival to be held on Tuesday, July 12th, and the authority information indicated by the A ticket (the right to interact with the artist for a short period of time) are associated and stored in the authority information storage means (J) within the viewer identification information storage unit (H) of this system.

<Embodiment 1: Viewing terminal (D): App viewing: Live video viewing>
FIG. 20 is a viewing screen of an actual performance. A small screen showing the state of the live venue and the placement of the cameras, which are the shooting devices (B), is popped up in the lower right (an example of the live video shooting device selection section of the viewing terminal (D)). The stage section is located at the top of the small screen, and the audience seats are located at the bottom. Camera 1 in front of the stage is displayed in black, indicating that the image displayed on the main screen of the viewing terminal (D) is the image of camera 1 (an example in which the live video of camera 1 is acquired by the live video acquisition means for each shooting device possessed by the live video acquisition section of the viewing terminal (D) and the live video output section outputs the live video). "1" is displayed in the upper left of the main screen to indicate that it is the image of camera 1. The other cameras 2 to 8 are positioned as shown in the small screen, with camera 2 from the left of the stage, camera 3 from the right of the stage, and cameras 4 to 8 placed near each of the five artists. These live images are captured by cameras 1 to 8, which are imaging devices (B) arranged at the live venue, and transmitted to a venue terminal (C), which transmits them to a live image transmission server (E), which transmits them to the viewer's viewing terminal (D), where they are displayed.

Below the main screen displaying the live video of camera 1 of the viewing terminal (D), sub-windows for cameras 2 to 8 are arranged. If the line bandwidth is narrow when live video of all cameras is transmitted over one line, the display of the main screen is prioritized, and the sub-windows below may be hidden or thinned out. If you want to see a screen other than camera 1, you can select a camera with a favorite angle from the small screen at the bottom right and display it on the main screen (live video shooting device selection unit). The videos from camera 1 to camera 8 may be collected at the venue terminal (C) and then transmitted over one line, or may be transmitted to the live video transmission server (E) over separate lines. The number of the camera you want to view is selected from the viewing terminal (live video shooting device selection unit). When transmitted over one line, the live video acquisition unit of the viewing terminal (D) receives the live video, and the live video of the shooting device (B) selected by the live video acquisition means for each shooting device possessed by the live video acquisition unit is displayed on the main screen of the viewing device (D). When transmission is performed over a separate line for each camera, the live video acquisition means for each imaging device selects the line corresponding to the selected camera, and the live video acquired by the live video acquisition unit of the viewing terminal (D) is displayed on the main screen of the viewing terminal (D).

When a drone-mounted camera can be selected as the imaging device (B) in the live video imaging device selection section of the viewing terminal (D), the viewing terminal (D) may be provided with a mobile imaging device instruction input section that instructs the drone on its movement and attitude (direction), and the venue terminal (C) may be configured to have a mobile imaging device instruction acquisition section that acquires instructions for the drone, and a mobile imaging device instruction output section that outputs the instructions to the drone. When a drone is selected, it is desirable to configure it so that the area in the venue that it can move around in is limited.

Three-dimensional live video was explained when explaining the venue terminal (C). It is also possible to watch a live performance in three-dimensional live video reproduced as a three-dimensional model. In this case, as explained above, you can move to any location within the live venue, and watch from close to the stage or your favorite performer from the wings of the stage. An example of the three-dimensional live video viewing position acquisition unit of the viewing terminal (D) is the small screen in the bottom right of Figure 20.

<Embodiment 1: Viewing terminal (D): Use of application: Example of interaction with artists during live performance>
FIG. 21 shows a viewer who has purchased an A ticket interacting with the artist, who is a fan, for a short time through the viewing terminal (D). The authority information stored in the authority information storage means (J) to be described in the fourth embodiment in association with the viewer identification information indicates that the viewer has the authority to directly interact with the artist, who is a fan, for a short time during the live performance. For example, at a predetermined timing during the live performance, the start of the interaction is directly announced (not shown) on the display of the viewing terminal (D), and when the viewer accepts the announcement (not shown), the artist is displayed in close-up on the main screen as shown in FIG. 21, and the viewer is displayed on the monitor screen of the viewer video in the lower right. The viewer video selection unit may be configured to select from candidate data (images, videos, text, etc.) for the viewer video stored in the viewer video candidate storage unit and transmit the selected data to the live venue. The viewer video is transmitted from the viewing terminal (D) to the viewer video transmission server (F), and the viewer video transmission server (F) transmits the received video to the venue terminal (C). The venue terminal (C) that receives the viewer image acquires the authority information of the A ticket associated with the viewer identification information from the viewer identification information of the viewer who transmitted the viewer image, and displays the received viewer image as an image of the viewer A, who is the communication partner, on the monitor (A) in front of the artist at the live venue. In this way, the viewer uses the viewing terminal (D) to enjoy the distributed live image.

<Embodiment 1: Viewing terminal (D): Use of application: Example of interaction between viewers during live performance>
It can also be configured so that the viewer video output to the monitor (A) at the live venue can be transmitted via a live video transmission server (E) and viewed on a viewing terminal (D). It is also possible to have interactions with other viewers through the same viewer video output to the monitor (A). An example of a configuration for this purpose is shown below.
The venue terminal (C)
a live venue viewer image transmission unit that transmits the viewer image output to the monitor (A) at the live venue to a live image transmission server (E) in parallel;
The viewing terminal (D)
a live venue viewer image acquisition unit that acquires a live venue viewer image that is the same viewer image as that output to a monitor (A) at the live venue;
a live venue viewer video output unit that outputs the acquired live venue viewer video to a separate window or the like on the display of the viewing terminal (D);
By having the above, the viewer can view the viewer video displayed on the monitor (A) at the live venue.
Furthermore, on the viewing terminal (D),
a live venue viewer image selection unit for selecting a viewer image by touching the displayed viewer image on the viewing terminal (D) or by moving a cursor;
a live venue viewer identification information acquisition unit that acquires viewer identification information associated with the selected viewer video;
an exchange application unit for sending messages to viewers identified by the acquired viewer identification information and applying for chat or direct dialogue;
By providing this function, it is possible to configure the system so that viewers can interact with other viewers who are watching the same live broadcast.

If the request is accepted by the other party, the system can be configured to have a viewer identification information storage unit (H) that stores viewer interaction information, which is information indicating that the two parties are in an exchange relationship. Once an exchange relationship is established, the two parties can exchange information while watching the live video by text chat or video communication directly through the viewing terminal, making the live viewing more enjoyable. It is preferable to register the person with whom you exchanged information during a live viewing session in the viewing terminal or live streaming app (said viewer interaction information storage unit) so that you can search for whether the other party is watching the next time another live streaming session (whether the performer is the same or different) is performed. If the other party is also watching, you can send a message, chat, or apply for a direct conversation, or start a direct chat, even if the viewer video is not output to the monitor (A). In addition to viewers who you met during live viewing, you can also register your original friends and acquaintances (you can refer to data such as a contact list in the viewing terminal (D)) and make it possible to send messages, chat, or apply for a direct conversation.

<Embodiment 1: Viewing terminal (D): Use of application: Viewing on monitor (A) on viewing terminal (D)>
When the viewer watches the same viewer video as that output to the monitor (A) at the live venue on the viewing terminal (D), if the viewer's own viewer video is displayed on the monitor (A), it is preferable that the viewer's own viewer video is displayed on the viewing terminal (D) so that the viewer can easily notice it. For example, the area in which the viewer's own viewer video is displayed may be surrounded by a thick frame, or the corners of the area may be filled in slightly to display "self". In addition, even when the viewer video is not displayed on the viewing terminal (D) as the same viewer video as that output to the monitor (A) at the live venue, it may be configured to display a ● mark or text on the edge or corner of the display connected to the viewing terminal (D) displaying the live video, whether or not the viewer's own viewer video has been output to the monitor (A) at the time of viewing. A thick line may be displayed on the bottom edge of the display to indicate whether or not the viewer's own viewer video has been output to the monitor (A) at the live venue from the start of the live performance. One example is to have the left end of the display represent the start time of the live performance and the right end represent the planned end time of the live performance, with a thick line connecting them and a circle with a larger diameter than the thick line being placed at the current time, and the period during which the viewer's image was output to monitor (A), including the current time and the left side of the circle which represents the past, being shown as a red line, and the period during which it was not output being shown as a white line, and to the right of the circle, the future period during which the viewer's image will not be output being shown as a dark grey line, and the period during which the viewer's image is scheduled to be output being shown as a dark red line.

<Embodiment 1: Viewing terminal (D): Charging during live performance>
It may also be possible to charge the performer or the organizer of the live event during the live viewing.
On the viewing terminal (D),
a charge payment method information acquisition unit for inputting information on a payment method such as electronic money or a credit card for paying a charge;
a billing amount information acquisition unit that acquires billing amount information, which is information regarding the payment of a billing amount that the user wishes to pay;
a billing amount information transmission unit that transmits billing amount information, which is information on a billing amount associated with billing payment method information;
This can be achieved by providing a charge amount information receiving unit in the venue terminal (C) that receives charge amount information. The charge payment method may be input each time, or a charge payment method information storage unit that stores information on a payment method that has been registered in advance or input once in the present system may be provided in the viewing terminal (D), the viewer video transmission server (F), the venue terminal (C), or a separate server managed (or entrusted with management) by the system operator. Alternatively, a charge payment method information acquisition unit may acquire information on a payment method such as electronic money, credit card, or prepaid card that is to be deposited in advance in the present system, and a charge amount information acquisition unit may accept input of the amount to be deposited in advance, and a charge amount information storage unit that stores charge amount information may be provided in the venue terminal (C) or a separate server managed (or entrusted with management) by the system operator.

By configuring in this way, the charge can be made as a so-called tip, for example, when a fan is moved by the performance of a comedian, or to support the organizer who has put on a live show. A charge button is placed on the edge of the viewing screen, and when the button is pressed, payment is made from electronic money (or a method that can be paid online, such as electronic money or credit card linked as a payment method) that has been registered in advance in this system (payment method information can be entered each time a charge is made). Separate buttons can be placed for the performer and the organizer. A fixed amount can be charged each time the button is pressed, or the button can be configured to allow the user to select an amount or enter an amount directly when the button is pressed. If the charge amount (tipping amount) during a live show is displayed at the live venue, it is expected to have the effect of raising the motivation of the performer and the organizer. When a charge is made, the name of the payer (real name or nickname, etc., registered in the system) can be displayed at the live venue for a certain period of time, and the names of the top high-payer payers can also be displayed along with their ranking.

<Embodiment 1: Live Video Transmission Server (E) (0105)>
The "live video transmission server (E)" (0105) is configured to transmit live video from the venue terminal (C) via a network to a viewing terminal (D) used by a viewer to view the live video. In addition to the live video, the venue terminal (C) can also transmit the viewer video to the live transmission server (E) in parallel with outputting the viewer video to the monitor (A) placed at the live venue. The live video transmission server (E) receives the viewer video transmitted from the venue terminal (C) (the same viewer video as the viewer video output to the monitor (A) at the live venue) and transmits it to the viewing terminal (D). The live video transmission server (E) may be installed at the live venue, or may be installed at a location managed by a server management company outside the live venue or a company that is in the business of live video distribution. It may be the same as the viewer video transmission server (F) that transmits the viewer video to the venue terminal (C) described below.

<Embodiment 1: Live video transmission server (E): other device, distribution line>
As mentioned above, it is preferable to equip the venue terminal (C) with functions such as a video switcher or video mixer, or to provide these devices separately, but the live video transmission server (E) may also be equipped with functions such as a video switcher or video mixer. If it is provided as a separate device, it is preferable to install it between the venue terminal (C) and the live video transmission server (E). Although it is called a video switcher or video mixer, it can handle not only video signals but also audio signals at the same time, and can perform switching, etc.

When distributing live video, it is advisable to use a low-latency, wide-bandwidth line so that there is as little time lag as possible between the actual progress of the live broadcast and the viewer's viewing on the viewing terminal (D), and so that there is sufficient bandwidth to prevent the live video being distributed and viewed from stopping, becoming distorted, or being choppy, such as with dropped frames.

<Embodiment 1: Viewer Video Transmission Server (F) (0106)>
The "viewer video transmission server (F)" (0106) is configured to transmit the viewer video from the viewing terminal (D) to the venue terminal (C) at the live venue. Note that there is nothing to prevent the viewer video transmission server (F) (0106) from being the same as the live video transmission server (E). The viewer video transmission server (F) may be installed at the live venue like the live video transmission server (E), or may be installed at a location managed by a server management company outside the live venue or a company engaged in live video distribution.

<Embodiment 1: Viewer Video Transmission Server (F): Server Capability>
The viewer video of the viewer who is watching outside the live venue is transmitted from the viewing terminal (D) to the viewer video transmission server (F). Since many viewers are outside the live venue, many accesses to the server occur simultaneously during the distribution of the live video. When selling viewing tickets, which are the right to watch the live video distribution, the number of tickets sold should be set according to the accessible capacity of the server, or the server capacity should be increased (by increasing the number of units, etc.) according to the change in the number of tickets sold to prevent a shortage.

<Embodiment 1: Viewer video transmission server (F): Delay of viewer video to live venue>
The viewing terminal (D) transmits viewer images (including audio) that are the reactions of the viewers to the received live video. The transmitted viewer images are output to the monitor (A) at the live venue with a slight delay from the performance and audio of the performers at the venue. This delay time is due to the processing time in the devices involved in the transmission and reception of the live video and the viewer images on the path, such as the shooting device (B), venue terminal (C), live video transmission server (E), viewing terminal (D), viewer video transmission server (F), venue terminal (C), and monitor (A), and the time lag when the viewer watches and reacts to the received live video. If the delay between the viewer images and the actual live performance at the live venue is large, it is disappointing. For example, when a performer makes a statement or takes an action that makes the audience in the venue laugh, the audience in the venue will laugh, but after the laughter in the venue dies down, the audience outside the venue may roar from the viewer images displayed on the monitor (A) in the quiet atmosphere of the venue as the next song begins to play. Even in the case of no audience, there is a concern that the motivation of the performers at the live venue will decrease if they receive a significantly different response from the viewers. Even in a venue with an audience, it takes several seconds for laughter to peak and then die down, so if the laughter of the viewer video overlaps with the laughter of the live video within a few seconds, it can be said to be within an acceptable range. Therefore, it is desirable that the delay in the round trip path between the transmission of the live video to the viewing terminal (D) and the reception of the viewer video from the viewing terminal (D) be within a range that humans would tolerate (less than 3 seconds, preferably 2 seconds or less).

<Embodiment 1: Viewer Video Transmission Server (F): Shaping of Viewer Video>
The viewer video transmission server (F) transmits the received viewer video to the venue terminal (C). The viewer video has a wide variety of resolutions (horizontal and vertical sizes of moving images) depending on the specifications of the viewing terminal (D), and is not suitable for display on the monitor (A) as is. The format must be unified before outputting from the venue terminal (C) to the monitor (A) at the live venue. In order to reduce the processing load on the venue terminal (C), it is better to have the viewer video transmission server (F) perform format unification processing of the viewer video (the venue terminal (C) may also process it by increasing its capacity). In order to unify the format of the viewer video, the viewer video transmission server (F) can be configured to have a viewer video format conversion unit that converts the format of the viewer video transmitted from the viewing terminal (D) into a format suitable for outputting to the monitor (A) at the live venue.

<Embodiment 1: Interactive live distribution system (0100)>
The "interactive live streaming system" (0100) is composed of a monitor (A) (0101), a filming device (B) (0102), a venue terminal (C) (0103), and a viewing terminal (D) (0104) used by viewers for viewing, a live video transmission server (E) (0105) that transmits live video from the live venue to the viewing terminal (D) (0104), and a viewer video server (F) (0106) that transmits viewer video from the viewing terminal (D) to the venue terminal (C) (0103).

Figure 17 shows a schematic diagram of the overall configuration of a two-way live distribution system. Through the Internet, the venue terminal (C) (1703A, B) at the live venue, the live video transmission server (E) (1705), the viewing terminal (D) (1704A-C) used by users A-C to view the live video, and the viewer video transmission server (F) (1706) to which the viewer video is sent from the viewing terminal (D) are connected by wire or wirelessly. At the live venue, the venue terminal (C) (1703A) is connected to a shooting device (B) (1702A) and a monitor (A) (1701A). The venue terminal (C) may be composed of not only one device but also multiple devices. For example, the venue terminal (C) (1703B) is composed of a control terminal and a server device, and the monitor (A) (1701B) and the shooting device (B) (1702B) are connected to the server device that constitutes the venue terminal (C) (1703B). Also, in one live venue or live event, the number of each is not limited to one each, such as one venue terminal (C), one monitor (A), and one image capture device (B). The venue terminal (C) (1703A), monitor (A) (1701A), and image capture device (B) (1702A), as well as the venue terminal (C) (1703B), monitor (A) (1701B), and image capture device (B) (1702B) shown in FIG. 17 may all be installed in one live venue and configured to work together to distribute one live performance in two directions.

The venue terminal (C) may also be configured by additionally connecting a video switcher or mixer to switch or superimpose video and audio signals and add subtitles. Furthermore, at least one venue terminal (C) may be provided for exclusive use, for transmitting live video to the live video transmission server (E) and for receiving viewer video from the viewer video transmission server (F). The number of venue terminals (C), viewing terminals (D), live video transmission servers (E), and viewer video transmission servers (F) are also not limited to those shown in FIG. 17.

<Processing flow of embodiment 1>
3 is a flowchart of the operation process of the two-way live streaming system which is the computer of embodiment 1. As shown in this figure, the two-way live streaming system of embodiment 1 has a venue terminal preparation step (c) (S0301), a shooting step (b) (S0302), a live video transmission step (e) (S0303), a viewer video transmission step (f) (S0304), and a monitor step (a) (S0305).

The operation method of the two-way live distribution system, which is a computer, is as follows:
The venue terminal preparation step (c) (S0301) performs processing for connecting the monitor (A) and the image capture device (B) to a venue terminal (C) that are arranged at a live performance venue;
The photographing step (b) (S0302) is a process in which a photographing device (B) is installed at a live performance venue, photographs live images, and transmits the images to a venue terminal (C) (not limited to a single terminal) described later.
A live video transmission step (e) (S0303) performs a process of transmitting the live video from the venue terminal (C) via a network from a live video transmission server (E) to a viewing terminal (D) used by a viewer to view the live video;
A viewer video transmission step (f) (S0304) performs a process of transmitting the viewer video from the viewing terminal (D) to a venue terminal (C) at the live venue from a viewer video transmission server (F) (which may be the same as the live video transmission server (E));
The monitor step (a) (S0305) performs processing for outputting viewer video from a venue terminal (C) (described later) arranged in the live performance venue to a monitor (A).
This is an operating method for causing a two-way live distribution system, which is a computer, to execute such a series of processes.

<First embodiment: Description of hardware>
4 is a diagram for explaining hardware configured by the main board of the venue terminal (C) of the two-way live distribution system, which is the computer of the first embodiment, and its peripheral devices. The live video transmission server (E), the viewer video transmission server (F), the viewing terminal (D), and the like are also configured similarly to a PC. The hardware of the viewing terminal (D) may be configured in such a way that a CPU and a chipset are integrated into one chip, as in a smartphone or tablet terminal. It is configured by a "CPU", a "chipset" consisting of a north bridge and a south bridge, a "non-volatile memory", a "main memory", an "I/O controller", a "USB, IEEE1394, LAN terminal, etc.", a "BIOS", a "PCI slot", a "real-time clock", and the like.

The various programs and data (information) stored in the non-volatile memory are expanded into the main memory when the system is started, and the programs are configured to perform calculations using the data in sequence by the CPU upon receiving execution commands.

In other words, in addition to the OS (operating system) and device drivers, the non-volatile memory contains a venue terminal preparation program (c), a shooting program (b), a live video transmission program (e), a viewer video transmission program (f), and a monitor program (a), and stores viewer video and live video as data. When the system is started up on the computer, these are expanded into the main memory, and when a start command is received, the CPU sequentially performs calculations using the programs and data.

This embodiment allows live video such as live video and audio to be distributed to viewers who are not at the live venue, and viewer video such as video and audio of viewers reacting to the live video in real time can be output to a monitor at the live venue, allowing the viewers' reactions to be delivered to the performers playing at the venue.

<Overview of the Second Embodiment> Mainly Claim 2
The second embodiment is based on the interactive live streaming system of the first embodiment, and is characterized by further having a viewer video selection unit (G) that selects, from the multiple received viewer videos, a viewer video to be output to a monitor at the live venue.

<Functional Configuration of Second Embodiment>
5 is a diagram showing an example of the configuration of an interactive live streaming system (0500) of the second embodiment based on the first embodiment. As shown in the figure, the interactive live streaming system (0500) of the second embodiment is configured to further include a viewer video selection unit (G) (0507) in addition to the components of the first embodiment.

<Description of Configuration of Second Embodiment>
<Embodiment 2: Viewer video selection unit (G) (0507)>
The "viewer video selection unit (G)" (0507) is configured in the venue terminal (C) (0503) to select a viewer video to be output to the monitor (A) from among multiple viewer videos received from the viewing terminal (D) via the viewer video transmission server (F).

As mentioned above, the "viewer video" is basically a real-time video of the viewer watching the live video transmitted from the viewing terminal (D). The viewer video includes not only a video signal but also an audio signal. When selecting with the viewer video selection unit (G), the output form to the monitor (A) can be selected from four options: video signal + audio signal, video signal only, audio signal only, and no output. It is possible to operate in such a way that only video or video + audio signal is used for everything, or the audio signal is the audio signal transmitted from the viewer and all viewer videos are output to the live venue, but the video signal is selected with the viewer video selection unit (G). It is also possible that the viewer video is not the viewer's own video while watching the live video, but video, images, audio data, and text data stored in the viewing terminal (D) is sent and received by the venue terminal. Video and images can be selected by the viewer video selection unit (G) along with other viewer videos and output to the monitor (A). If some viewers transmit audio data, the viewer video containing only audio data is output to the speakers at the live venue, and additional viewer video including video output is selected so that there is no blank space on the display of monitor (A). In the case of text data, it is output to monitor (A) as an image. In the following explanation of this embodiment 2, the selection of video signal + audio signal, or video signal only, is used as an example.

<Embodiment 2: Viewer Video Selection Unit (G): Example of Output to Monitor (A)>
When there are many viewers watching live video from outside the live venue using the viewing terminal (D), if all the viewers' videos are output to the monitor (A), the videos may be displayed so small that it is difficult for the performers on stage to distinguish the facial expressions of each person. Resolution: If the viewer videos are output at a resolution of 40 x 30 on the monitor (A) with a horizontal resolution of 1920 x vertical resolution, 1728 people can be displayed, but the facial features will be blurred, and as described above, the individual faces will not be distinguished and the expressions will not be clear. Or, when the video of the number of viewers that cannot be displayed is received, it is necessary to output a part selected from the received viewer videos to the monitor (A). As described in the explanation of the monitor (A) in the first embodiment, when displaying at a size that can be seen by the performers on stage, it will be difficult to display about 30 people on one display. If the performer is popular, it is considered that the number of viewers will not be able to be displayed on three or four displays.

<Embodiment 2: Viewer Video Selection Unit (G): Selection of Viewer Video: Manual or Automatic>
When a viewer video needs to be selected, an operator can manually select a viewer video and replace it with the viewer video output to the monitor (A) in the live venue. When performed manually, the operator must be present to perform the selection and replacement work, which is time-consuming, and when there are many viewers, it is possible that the viewers will make a mistake in their selection (selecting the same viewer video multiple times, or failing to select the viewer video that should be selected). Therefore, it is also possible to configure the system to automatically select and replace the viewer video output to the monitor (A). In that case, it is possible to replace the viewer videos displayed one after the other depending on the time. For example, the order in which the viewer videos arrive at the venue terminal (C) is stored, and the videos are selected in order of the earliest arrival for the number of people that can be displayed on the display at one time, and output to the monitor (A).

<Embodiment 2: Viewer video selection unit (G): Configuration example for automatically selecting viewer video by time>
The viewer video selection unit (G) of the venue terminal (C) further
A viewer image storage means for storing the acquired viewer image;
a viewer video arrival order information acquiring means for acquiring viewer video arrival order information which is information indicating the order in which the viewer videos were acquired;
A viewer video arrival order information storage means for storing viewer video arrival order information;
a viewer image switching time information acquisition means for acquiring viewer image switching time information which is information indicating a timing for switching a viewer image,
The viewer video selection unit (G) can be configured to select a viewer video stored in association with viewer arrival order information, which is information indicating the order in which the viewer videos arrived at the venue terminal (C), at each viewer video switching time.

<Processing flow of embodiment 2>
6 is a flowchart of the operation process of the two-way live streaming system of embodiment 2 based on embodiment 1. As shown in this figure, the two-way live streaming system of embodiment 2 has a venue terminal preparation step (c) (S0601), a shooting step (b) (S0602), a live video transmission step (e) (S0603), a viewer video transmission step (f) (S0604), a viewer video selection step (g) (S0605), and a monitor step (a) (S0606).

The operation method of the two-way live distribution system, which is a computer, is as follows:
The venue terminal preparation step (c) (S0601) performs processing for connecting the monitor (A) and the image capture device (B) to the venue terminal (C) that are arranged at the live performance venue;
The photographing step (b) (S0602) is a process in which a photographing device (B) is installed at a live performance venue, photographs live images, and transmits the images to a venue terminal (C) (not limited to a single terminal) described later.
A live video transmission step (e) (S0603) performs a process of transmitting the live video from the venue terminal (C) via a network from the live video transmission server (E) to a viewing terminal (D) used by a viewer to view the live video;
A viewer video transmission step (f) (S0604) performs a process of transmitting the viewer video from the viewing terminal (D) to a venue terminal (C) at the live venue from a viewer video transmission server (F) (which may be the same as the live video transmission server (E));
The viewer image selection step (g) (S0605) of the hall terminal (C) selects a viewer image to be output to the monitor (A) from among a plurality of viewer images received from the viewing terminal (D) from the viewer image transmission server (F),
The monitor step (a) (S0606) performs processing for outputting viewer video from a venue terminal (C) (described later) arranged in the live performance venue to a monitor (A).
This is an operating method for causing a two-way live distribution system, which is a computer, to execute such a series of processes.

<Description of Hardware in Second Embodiment>
7 is a diagram for explaining hardware configured by the main board and its peripheral devices of the two-way live distribution system, which is a computer of embodiment 2 based on embodiment 1. It is configured by a "CPU", a "chipset" consisting of a north bridge and a south bridge, a "non-volatile memory", a "main memory", an "I/O controller", "USB, IEEE1394, LAN terminal, etc.", a "BIOS", a "PCI slot", a "real-time clock", etc.

The various programs and data (information) stored in the non-volatile memory are expanded into the main memory when the system is started, and the programs are configured to perform calculations using the data in sequence by the CPU upon receiving execution commands.

In other words, in addition to the OS (operating system) and device drivers, the non-volatile memory contains a venue terminal preparation program (c), a shooting program (b), a live video transmission program (e), a viewer video transmission program (f), a viewer video selection program (g), and a monitor program (a), and stores viewer video and live video as data. When the system is started up on the computer, these are expanded into the main memory, and when a start-up command is received, the CPU sequentially performs calculations using the programs and data.

In this embodiment, viewer videos can be selected from multiple received viewer videos, narrowed down to the number that can be displayed on the venue monitor at one time, and sequentially replaced.

<Outline of embodiment 3> Mainly claim 3
The two-way live distribution system of the third embodiment based on the second embodiment has a viewer identification information storage unit (H) and a viewer video selection unit (G) configured to select a viewer video by using the viewer identification information.

<Functional Configuration of Third Embodiment>
8 shows a functional block diagram of an interactive live streaming system according to embodiment 3 based on embodiment 2. The interactive live streaming system (0800) of embodiment 3 further includes a viewer identification information storage unit (H) (0808) in addition to the configuration of embodiment 2. Since everything except the viewer identification information storage unit (H) (0808) has already been explained, only the viewer identification information storage unit (H) (0808) and viewer video selection unit (G) (0807) according to embodiment 3 will be explained below.

<Description of the configuration of embodiment 3>
<Third embodiment: Viewer identification information storage unit (H) (0808)>
The 'viewer identification information storage unit (H)' (0808) is configured in the hall terminal (C) (0803) so as to store viewer identification information for identifying the viewer (including the case of the viewing terminal (D)).

<Embodiment 3: Viewer Identification Information Storage Unit (H): Viewer Identification Information>
The "viewer identification information" may be a ticket number when purchasing a live viewing ticket as a right to view a live performance, or a reception number when purchasing a ticket on the website of the live performance organizer or a ticket sales site. If the ticket is purchased through a fan club, the viewer may have a fan club membership number, a membership number when registering as a member on a ticket sales site that is often used for purchasing tickets (as in the example in the first embodiment), or an email address or SMS address registered as a contact. Alternatively, the viewer may use viewing terminal identification information, which is information for identifying the viewing terminal (D) used by the viewer for viewing, as the viewer identification information. The viewing terminal identification information may be the MAC address or serial number (manufacturing number) of the viewing terminal (D). The viewer identification information may be configured to hold information such as the name, age, sex, address, email address, hobbies and preferences, family structure, and fan club membership number of the purchaser registered on the website when purchasing the ticket as viewer attribute information, which is information on the viewer's attributes, in association with the viewer identification information.

<Embodiment 3: Viewer Identification Information Storage Unit (H): Attaching Viewer Identification Information to Viewer Video>
The viewer terminal (D) can transmit a viewer video with viewer identification information attached. If the viewer identification information is configured to be attached, the viewer video can be selected by the viewer video selection unit (G) of the hall terminal (C) using the viewer identification information as a clue.

The viewer video arriving at the viewer video transmission server (F) from the viewing terminal (D) is information transmitted via an Internet line or the like, and the viewer video is stored in the payload of the packet transmitted and received via the Internet line, and the payload or header can be configured to store viewer identification information that identifies the viewer appearing in the viewer video.

<Embodiment 3: Viewer Identification Information Storage Unit (H): Viewer Identification Information Attached to Viewer Video>
The viewer identification information attached to the viewer video, which is the viewer identification information attached to the viewer video, does not have to be the same as the viewer identification information held by the viewer identification information holding unit (H) of the venue terminal (C). For example, a configuration will be described in which the viewing terminal identification information of the viewing terminal (D) used for viewing is used as an example of information different from the viewer identification information held by the venue terminal (C). The viewer identification information attached to the viewer video may be other information such as a MAC address or IP address that identifies the viewing terminal, or a live viewing ticket number. It is sufficient that the information is uniquely associated with the viewer identification information held by the viewer identification information holding unit of the venue terminal (C).

<Embodiment 3: Viewer Identification Information Storage Unit (H): Configuration for Attaching Viewer Identification Information to Attach Viewer Video>
The viewer identification information storage unit (H) of the hall terminal (C) further
a viewer image attached viewer identification information holding means for holding viewer image attached viewer identification information attached to the viewer image and the viewer's own viewer identification information (e.g., membership number, email address, etc.) in association with each other;
The viewing terminal (D) can be configured to have a viewer image transmitting section which attaches either the viewer identification information or its own viewing terminal identification information to the viewer image and transmits it to the viewer image transmitting server (F).

<Third embodiment: Viewer video selection unit (G) (0807)>
The "viewer video selection unit (G)" (0807) is configured to select a viewer video by using the viewer identification information. That is, the viewer video selection unit (G) (0807) of the venue terminal (C) is configured to select a viewer video to be output to the monitor (A) from among a plurality of viewer videos received from the viewing terminal (D) of the viewer video transmission server (F) by using the viewer identification information.

<Embodiment 3: Viewer video selection unit (G): Configuration example of automatic selection of viewer video by time>
In the second embodiment, an example in which the viewer video selection unit (G) of the hall terminal (C) automatically switches the viewer video every hour has been described. When viewer identification information for identifying the viewer who transmitted the viewer video is also used,
A viewer image arrival order information holding means of a viewer image selection unit (G) included in the hall terminal (C) holds viewer image arrival order information indicating the order in which the viewer images arrived at the hall terminal (C) in association with viewer identification information,
The viewer video selection unit (G) can be configured to select viewer videos of viewers identified by viewer identification information associated with the viewer video arrival order information in accordance with the order indicated by the viewer video arrival order information for output to the monitor (A).

<Embodiment 3: Viewer Video Selection Unit (G): Selection Example Using Viewer Attribute Information>
In addition to the order of arrival, if there are multiple monitors (A), the viewer video to be output to each monitor can be selected based on the viewer attribute information stored in association with the viewer identification information. For example, the viewer video can be selected based on the viewer's address, broadly dividing them into those living in Western Japan and those living in Eastern Japan, dividing them into regions (such as Hokkaido, Tohoku, Hokuriku, Kanto, Chubu, Kansai, Chugoku, Shikoku, and Kyushu/Okinawa), dividing them into men and women, dividing them into generations, and dividing them into fans of a specific artist. The viewer video of the fan of each performer can be selected and output to the monitor (A). Also, for example, in the case of a group consisting of multiple performers, when one member performs alone, the viewer video of the viewer who is a fan of the member can be output to the monitor (A) to show the member's support, which is expected to increase the motivation of the member and to perform better.

<Embodiment 3: Viewer Video Selection Unit (G): Response when Viewer Video Output is Partially Interrupted>
In addition, when the viewer video output to the monitor (A) is interrupted due to the viewer's interruption of viewing due to circumstances on the viewer's side, or when some of the viewer videos output to the monitor (A) are interrupted due to problems with the line or equipment, it is preferable to select from the viewer videos that were not output to the monitor (A) in order to quickly replace the corresponding viewer video. It is easy to notice that the viewer video output to the monitor (A) is not moving, the display is distorted, or the part where the corresponding viewer video was output is displayed in black, which is one of the factors that lowers the motivation of the performers on stage. When all the viewer videos are output to the monitor (A) and there is no new viewer video to be added, it is preferable to enlarge at least a part of the viewer video that has already been output. If it becomes impossible to display everyone by enlarging it, the viewer video to stop outputting to the monitor (A) is selected.

<Processing flow of embodiment 3>
9 is a flowchart of the operation process of the interactive live streaming system of embodiment 3 based on embodiment 2. As shown in this figure, the interactive live streaming system of embodiment 3 includes a viewer identification information retaining step (h) (S0901), a venue terminal preparing step (c) (S0902), a shooting step (b) (S0903), a live video transmitting step (e) (S0904), a viewer video transmitting step (f) (S0905), a viewer video selecting step (g) (S0906), and a monitoring step (a) (S0907).

The operation method of the two-way live distribution system, which is a computer, is as follows:
The viewer identification information holding step (h) (S0901) of the hall terminal (C) performs a process of holding viewer identification information for identifying a viewer (including the case of the viewing terminal (D)),
The venue terminal preparation step (c) (S0902) performs processing for connecting the monitor (A) and the image capture device (B) to a venue terminal (C) that are arranged at the live performance venue;
The photographing step (b) (S0903) is a process in which a photographing device (B) is installed at a live performance venue, photographs live images, and transmits the images to a venue terminal (C) (not limited to a single terminal) described later.
A live video transmission step (e) (S0904) performs a process of transmitting the live video from the venue terminal (C) via a network from the live video transmission server (E) to a viewing terminal (D) used by a viewer to view the live video;
A viewer video transmission step (f) (S0905) performs a process of transmitting the viewer video from the viewing terminal (D) to a venue terminal (C) at the live venue from a viewer video transmission server (F) (which may be the same as the live video transmission server (E));
The viewer video selection step (g) (S0906) of the hall terminal (C) performs a process of selecting a viewer video to be output to the monitor (A) from among a plurality of viewer videos received from the viewing terminal (D) from the viewer video transmission server (F) using viewer identification information,
The monitor step (a) (S0907) performs processing for outputting viewer video from a venue terminal (C) (described later) arranged in the live performance venue to a monitor (A).
This is an operating method for causing a two-way live distribution system, which is a computer, to execute such a series of processes.

Third Embodiment Hardware Description
10 is a diagram for explaining hardware configured by the main board and its peripheral devices of an interactive live distribution system which is a computer of embodiment 3 based on embodiment 2. It is configured by a "CPU", a "chipset" consisting of a north bridge and a south bridge, a "non-volatile memory", a "main memory", an "I/O controller", "USB, IEEE1394, LAN terminal, etc.", a "BIOS", a "PCI slot", a "real-time clock", etc.

The various programs and data (information) stored in the non-volatile memory are expanded into the main memory when the system is started, and the programs are configured to perform calculations using the data in sequence by the CPU upon receiving execution commands.

In other words, the non-volatile memory contains, in addition to the OS (operating system) and device drivers, a venue terminal preparation program (c), a shooting program (b), a live video transmission program (e), a viewer video transmission program (f), a viewer video selection program (g), a monitor program (a), and a viewer identification information retention program (h), and stores viewer video, live video, and viewer identification information as data. When the system is started up on the computer, these are expanded into the main memory, and when a start-up command is received, the CPU sequentially performs calculations using the programs and data.

In this embodiment, the viewer identification information can be used to select a viewer video of a viewer identified by the viewer identification information. If the system is configured to also retain viewer attribute information, the viewer video can be selected based on the viewer identification information associated with the attribute information. The output of the viewer video to the monitor (A) in the venue can also contribute to the performance effects in the venue.

<Outline of embodiment 4> Mainly claim 4
The two-way live distribution system of the fourth embodiment, which is based on the third embodiment, is configured to retain viewer identification information for identifying a viewer and authority information, which is information indicating the authority of the viewer, in association with each other.

<Functional Configuration of Fourth Embodiment>
11 shows a functional block diagram of an interactive live streaming system according to embodiment 4 based on embodiment 3. The interactive live streaming system (1100) of embodiment 4 further includes an authority information storage means (J) (1109) in a viewer identification information storage unit (H) (1108) in addition to the configuration of embodiment 3.

<Fourth embodiment: Description of configuration>
<Fourth embodiment: Authority information holding means (J) (1109)>
The "authority information storage means (J)" (1109) is configured within the viewer identification information storage unit (H) (1108) so as to store viewer identification information in association with authority information, which is information indicating the authority of the viewer identified by the viewer identification information.

<Fourth embodiment: Authority information holding unit (J): Example of authority information>
The "authority information" indicating the authority of the viewer may be, for example, the authority for the video signal + audio signal of the viewer video to be output to monitor (A), the authority for only the video signal of the viewer video to be output to monitor (A), the authority for only the audio signal of the viewer video to be output to monitor (A), the authority for neither the video signal nor the audio signal of the viewer video to be output to monitor (A) (text may be allowed to be transmitted, or images provided in advance by the live event organizer, such as images of applause or waving hands, may be selected and transmitted), the authority to only watch the distributed live video (as mentioned above, text data cannot be sent), the authority to distribute the live video over the line with the widest bandwidth among the lines in use, etc. Possible rights include the right to view live images in a resolution (resolution is not limited to these examples), such as SD (pixel count: 720 x 480), HD (pixel count: 1920 x 1080), 4K (pixel count: 3840 x 2160), etc. (if the right to high resolution is given, the right to view in low resolution is also given), the right to select one of the multiple image capture devices (B) installed in the venue and view the live images captured by that image capture device (B), the right to view the stereoscopic live images from any viewpoint in the three-dimensional reproduction space, the right to output the viewer's image to the monitor (A) for a long period of time, the right to display the viewer's image larger on the monitor (A) than other viewer images, etc. Also, in a configuration in which a button or icon representing the viewer's emotion is pressed on the viewing terminal (D) and the result is displayed on the monitor (A) in the venue, the right to input an action such as pressing a button or icon representing emotion into the present system is also conceivable. It is also possible to provide the authority to view live footage corresponding to the position of each seat to which the ticket purchased by the viewer corresponds by creating an image for each seat by placing a camera device (B) at each seat or by thinning out and interpolating the footage between the camera devices (B) placed at each seat. It is also possible to provide the authority to enable a specific viewer to have a conversation or other interaction with the performers at the live venue via the viewing terminal (D) (example of the first embodiment).

<Fourth embodiment: Authority information holding unit (J): other examples of authority information>
Other examples of "authority information" include the authority for a viewer to record, record audio, or capture still images of the live video being distributed (either one of them, or a combination of one or more of them). If the viewing terminal (D) has a large capacity non-volatile memory, the data may be recorded in the non-volatile memory of the viewing terminal (D), and if there is no non-volatile memory with sufficient capacity, an external memory may be connected and recorded. In the case where the viewer records and keeps the data on hand, it is desirable to attach viewer identification information or viewing terminal identification information of the viewing terminal (D) to the recorded data as key information for viewing, and to configure so that the stored data cannot be viewed unless it is played back by the viewer himself or the viewing terminal (D) that recorded it. In the unlikely event that the recorded data is copied from the viewer, the source can be traced and the data cannot be played back or viewed by anyone other than the viewer who recorded it, making it easy to protect the copyright of the live video. The data may be recorded on a cloud server managed by the live organizer or a company commissioned by the live organizer. If the data cannot be downloaded from the cloud server and is viewed by connecting to it, it is possible to prevent unauthorized use of the recorded data by anyone other than the viewer who recorded it. Furthermore, instead of being a recording right for viewers, the right may be a right to freely view live footage stored by the live event organizer on a cloud server or the like after the live event has ended.

<Embodiment 4: Authority Information Storage Unit (J): Expansion of Authority Information During Live Viewing>
The authority information of each viewer can be configured so that the viewer can extend the authority by charging a fee using a means of payment available on the Internet, such as electronic money, credit card payment, or inputting the number of a gift card or prepaid card, during live viewing. This is convenient when a viewer with no recording authority wants to record while watching live. In addition, it is preferable that the viewer can check the remaining number of the various authorities described above that can be changed to a higher (or extended) authority than the viewer's own authority from the viewing terminal (D) within the range allowed by the equipment capabilities of the live distribution, and can change the authority by charging. For example, when the authority is extended to be able to view a higher quality video, if all the viewers extend the authority, it may exceed the processing capacity of the distribution line and equipment prepared by the live organizer, causing the distribution video to be distorted or unable to be distributed.

<Embodiment 4: Authorization information storage unit (J): Configuration example of extending authority information during live viewing>
The following configuration example is conceivable for extending the above-mentioned authority information.
The viewing terminal (D) includes a charge payment method information acquisition unit for acquiring information on a payment method by which a viewer pays a charge in order to extend his/her authority;
a charge amount information acquisition unit that acquires charge amount information, which is information on the payment of an amount required for the extension, in association with the viewer identification information and the authority identification information that identifies the extended authority desired by the viewer;
a charge information transmission unit that transmits the charge information to a viewer video transmission server (F) for sending the charge information to a venue terminal (C);
The viewer identification information storage unit (H) of the hall terminal (D)
The system may further comprise an authority information expansion means for changing the authority information held in the authority information holding means to expand the authority based on the authority identification information and viewer identification information associated with the received billing amount information.

Below, we will explain again the main configuration requirements when using permission information.

<Embodiment 4: Viewer Identification Information Storage Unit (H) (1108)>
The "viewer identification information storage unit (H)" (1108) is configured to store viewer identification information for identifying a viewer (including the case of a viewing terminal (D)) and to have the authority information storage means (J) (1109). Various examples of authority information have been described above. A viewer selects and purchases authority for live viewing, for example, in the form of a ticket type (price may vary depending on the authority type). Information identifying a ticket, such as a ticket number, a transaction number of a website when purchasing a ticket, a membership number of the same site, and the name of the viewer are stored as viewer identification information in the viewer identification information storage unit (H) of the venue terminal (C), and authority information, which is information indicating the corresponding authority in association with the viewer identification information, is stored in the authority information storage means (J) in the viewer identification information storage unit (H).

<Embodiment 4: Live Video Transmission Server (E) (1105)>
The "live video transmission server (E)" (1105) is configured to transmit live video from the venue terminal (C) via a network to a viewing terminal (D) used by a viewer identified by the viewer identification information to view the live video, according to the authority information associated with the viewer identification information. The live video to be transmitted to the viewer is selected and transmitted depending on what live video the viewer has the authority to view. For example, if viewer A has the authority to view SD quality live video and another viewer B has the authority to view HD quality live video, the live video to be transmitted to each viewer A or B is transmitted based on the viewer identification information and authority information held in the viewer identification information holding unit (H) of the venue terminal (C) and the authority information holding means (J) provided therein.

<Embodiment 4: Live Video Transmission Server (E): Example of Transmission of Different Live Video for Each Authority Information>
The following configuration example is possible for transmitting live video based on authority information.
The venue terminal (C)
a live video type identification information acquisition unit that acquires live video type identification information, which is information for identifying a type of live video to be transmitted to a viewer identified by the viewer identification information, in association with the viewer identification information based on authority information associated with the viewer identification information stored in the authority information storage means (J);
The live video transmission server (E) can be configured to have a live video type identification information sending unit that sends the live video type identification information to a live video transmission server (E), and the live video transmission server (E) can be configured to have a live video distribution destination selection unit that sends live video of the type identified by the live video type identification information to a viewer identified by the associated viewer identification information based on the received live video type identification information.

<Embodiment 4: Viewing Terminal (D) (1104)>
The "viewing terminal (D)" (1104) receives live video transmitted from the live video transmission server (E) via the network, and transmits the viewer video to the viewer video transmission server (F) (1105) (which may be the same as the live video transmission server (E)). As described above, the viewing terminal (D) is assumed to be a tablet terminal, a smartphone, a PC, or the like. The viewer video is not limited to the video of the viewer captured by the built-in camera of the viewing terminal (D) as described above, but may be the video signal + audio signal of the viewer while watching, the video signal only, the audio signal only, or a video, still image, audio, etc. stored in the viewing terminal (D), or text data. It is not limited to data prepared before the start of the live, and may be data created and stored during the live. Alternatively, the live organizer may select and transmit avatars, videos, still images, animations, audio, text, etc. that are prepared for use by the viewer. The data prepared in advance by the live organizer may be set so that even viewers without the viewer video transmission authority can use it during the live, or it may be used in a time-limited manner. Alternatively, it may be possible to allow only viewers who have the viewer video transmission authority to use it.

<Embodiment 4: Viewer Video Transmission Server (F) (1106)>
The "viewer video transmission server (F)" (1106) is configured to transmit the viewer video from the viewing terminal (D) to the venue terminal (C) at the live venue. Depending on the type of live viewing authority purchased by the viewer, the viewer video may be prepared when the viewing terminal (D) prepares for the viewer video transmission, or the viewer video transmission server (F) transmitted from the viewing terminal (D) may prepare for transmission to the venue terminal (C). The preparation of the viewer video means that if the viewer has the authority to transmit the video signal + audio signal of the viewer video, the video signal only is prepared if the viewer has the authority to transmit only the video signal, and the audio signal only is prepared if the viewer has the authority to transmit only the audio signal. If the viewer does not have the authority to transmit the viewer video, it is better to configure the upstream viewing terminal (D) not to transmit the viewer video to the viewer video transmission server (F) because this does not cause a load on the line. It is also possible to arrange multiple venue terminals (C) so that each is responsible for transmitting live video, viewer video (video signal + audio signal, or video signal only), viewer video (audio signal only), viewer video (text message), etc., and to configure the viewer video to be transmitted to a separate venue terminal (C) for each piece of authority information based on the authority information associated with the viewer identification information.

<Embodiment 4: Viewer video selection unit (G) (1107)>
The "viewer video selection unit (G)" (1107) is configured in the venue terminal (C) to select a viewer video to be output to the monitor (A) from among a plurality of viewer videos received from the viewing terminal (D) from the viewer video transmission server (F) using viewer identification information and based on authority information associated with the viewer identification information. When selecting a viewer video to be output to the monitor (A), the selection is made based on authority information associated with the viewer identification information attached to the viewer video. For example, in the case of a viewer who has authority to have the video output to the monitor (A) in the live venue for a long time, output in a large size, or output in a position that is easily visible and noticeable to the performers, the viewer video selection unit (G) selects the viewer video taking into consideration the authority of the viewer. The viewer video of the viewer who has authority to have the video output for a long time is left unselected even during periodic replacement, and other parts of the viewer video are replaced. A viewer who has the right to output the video signal of the viewer video cannot select many viewer videos due to the display restrictions of the monitor (A), but when selecting a viewer video who has the right to output only the audio signal of the viewer video, the video is output to a speaker, and more viewer videos (audio signals only) can be selected than the video signals.

<Fourth embodiment: Viewer video selection unit (G): Selection based on attribute information and authority information>
When viewer attribute information is also stored in association with the viewer identification information, the viewer information is selected based on the associated authority information and attribute information using the viewer identification information attached to the viewer video. When the viewer identification information attached to the viewer video is different from the viewer identification information stored in the viewer identification information storage unit (H) of the venue terminal (C), the viewer identification information attached to the viewer video is searched for corresponding viewer identification information in the viewer identification information storage unit (H). The viewer video is selected based on the authority information and attribute information associated with the viewer identification information stored in the viewer identification information storage unit. For example, when a singer from Niigata Prefecture is performing, a viewer video having attribute information of an address in Niigata Prefecture becomes a candidate and is output to the monitor (A) based on the authority information (authority to output a video signal and/or an audio signal to the monitor (A)). In addition to the case where only the viewer video of the viewer living in Niigata Prefecture is output to the monitor (A), a case where the viewer video of the viewer living in Niigata Prefecture is collected and output near the front of the singer can also be considered. Also, if the user has the right to output at least a video signal, it is output to the display of monitor (A), and if only an audio signal is received, it is output only to the speaker of monitor (A), and so on.

<Embodiment 4: Monitor (A) (1101)>
The "monitor (A)" (1101) is placed at the live venue and is connected to the venue terminal (C) so as to output the viewer video from the venue terminal (C). The monitor (A) (1101) is not limited to a display that outputs video signals as described above. Speakers that output audio signals are also included in the monitor (A), and displays that display text messages, electronic bulletin boards, LED displays, etc. can also be included in the monitor (A). The viewer video is selected according to the viewer's authority information, and the viewer video according to the authority is output to the monitor (A).

<Processing flow of embodiment 4>
12 is a flowchart of the operation process of an interactive live distribution system which is a computer of embodiment 4 based on embodiment 3. As shown in this figure, the interactive live distribution system of embodiment 4 has a viewer identification information holding step (h) (S1201), an authority information holding sub-step (j) (S1202), a venue terminal preparation step (c) (S1203), a shooting step (b) (S1204), a live video transmission step (e) (S1205), a viewer video transmission step (f) (S1206), a viewer video selection step (g) (S1207), and a monitor step (a) (S1208).

The operation method of the two-way live distribution system, which is a computer, is as follows:
The viewer identification information holding step (h) (S1201) of the hall terminal (C) performs a process of holding viewer identification information for identifying a viewer (including the case of the viewing terminal (D)),
The authority information holding sub-step (j) (S1202) in the viewer identification information holding step (h) performs a process of holding viewer identification information and authority information, which is information indicating the authority of a viewer identified by the viewer identification information, in association with each other;
The venue terminal preparation step (c) (S1203) performs processing for connecting the monitor (A) and the image capturing device (B) to a venue terminal (C) that are arranged at the live performance venue;
The photographing step (b) (S1204) is a process in which a photographing device (B) is installed at a live performance venue and photographs live images and transmits the images to a venue terminal (C) (not limited to a single terminal) described later.
A live video transmission step (e) (S1205) performs a process of transmitting the live video from the venue terminal (C) via a network from the live video transmission server (E) to a viewing terminal (D) used by a viewer to view the live video;
A viewer video transmission step (f) (S1206) performs a process of transmitting the viewer video from the viewing terminal (D) to a venue terminal (C) at the live venue from a viewer video transmission server (F) (which may be the same as the live video transmission server (E));
The viewer image selection step (g) (S1207) of the hall terminal (C) selects a viewer image to be output to the monitor (A) from among a plurality of viewer images received from the viewing terminal (D) from the viewer image transmission server (F),
The monitor step (a) (S1208) performs processing for outputting viewer video from a venue terminal (C) (described later) arranged in the live performance venue to a monitor (A).
This is an operating method for causing a two-way live distribution system, which is a computer, to execute such a series of processes.

<Fourth embodiment: Description of hardware>
13 is a diagram for explaining hardware configured by the main board and its peripheral devices of the two-way live distribution system which is a computer of the fourth embodiment based on the third embodiment. It is configured by a "CPU", a "chipset" consisting of a north bridge and a south bridge, a "non-volatile memory", a "main memory", an "I/O controller", "USB, IEEE1394, LAN terminal, etc.", a "BIOS", a "PCI slot", a "real-time clock", etc.

The various programs and data (information) stored in the non-volatile memory are expanded into the main memory when the system is started, and the programs are configured to perform calculations using the data in sequence by the CPU upon receiving execution commands.

In other words, the non-volatile memory contains, in addition to the OS (operating system) and device drivers, a venue terminal preparation program (c), a shooting program (b), a live video transmission program (e), a viewer video transmission program (f), a viewer video selection program (g), a monitor program (a), a viewer identification information retention program (h), and an authority information retention sub-program (j), and stores viewer video, live video, viewer identification information, and authority information as data. When the system is started up on a computer, these are expanded into the main memory, and when a start-up command is received, the CPU sequentially performs calculations using the programs and data.

In this embodiment, by storing authority information, which is information indicating the viewer's authority, in association with viewer identification information, it is possible, for example, to display the viewer on the monitor (A) in the venue for a long period of time, or to display the viewer in a more conspicuous location on the monitor (A) to the performers, depending on the viewer's authority. Alternatively, it is possible to distinguish the viewing format of the received live video, for example, by providing different types of resolution when receiving live video distribution, or by allowing the viewer to select video from a different angle other than the standard distribution, depending on the viewer's authority.

<Outline of the 5th embodiment> Mainly claim 5
The two-way live distribution system of the fifth embodiment, which is based on any one of the second to fourth embodiments, is configured to further include a video signal cutoff command receiving unit (K) and a cutoff command executing means (L).

<Fifth embodiment: Functional configuration>
The two-way live streaming system (1400) of the fifth embodiment based on the second embodiment shown in Fig. 14 has a video signal cut-off command receiving unit (K) (1410) and a cut-off command executing means (L) (1411) in the viewer video selecting unit (G) (1407) in addition to the components of the second embodiment. The same effect can be obtained even if the system is based on either the third or fourth embodiment.

<Description of the configuration of embodiment 5>
<Fifth embodiment: Video signal cutoff command receiving unit (K) (1410)>
The “video signal cut-off command receiving unit (K)” (1410) is configured in the venue terminal (C) to receive a cut-off command, which is a command issued by an operator of the venue terminal (C) to cut off the video signal of the viewer video from the viewing terminal (D), which is the source of the viewer video being output to the monitor (A).

<Embodiment 5: Video signal cutoff command reception unit (K): cutoff command>
The "blocking command" is issued by the operator of the venue terminal (C) in order to switch the viewer video displayed on the monitor (A). The block command for the viewer video output to the monitor (A) may be issued for each viewer video. Conversely, it may be set in advance to automatically issue the block command. For example, it may be configured so that a block command is automatically issued periodically in order to switch between group A, which is a group of viewers identified by viewer identification information, and group B, which contains viewer identification information different from that of group A, and so on, every 10 minutes from now. The block command may also be issued not only for one viewer video, but also for multiple viewer videos. It may be issued as a block command to block the output to the monitor (A) for multiple viewer videos at once, or at least one block command may be issued periodically, such as every 30 seconds. The number of commands issued and the interval may be constant or not. Alternatively, it may be configured to be determined by AI or the like so that when a viewer video selected and output to the monitor (A) meets a specific condition, a command to block the corresponding viewer video is issued. For example, video of viewers who are not fans can be blocked off in order to selectively output video of fans according to the time of output or the order in which performers are appearing.

<Embodiment 5: Video signal cutoff command reception unit (K): cutoff command: audio signal>
The cut-off command is a command to cut off the video signal of the viewer video, but it may be configured to cut off the audio signal contained in the viewer video at the same time. Depending on the specifications of the viewing terminal (D) or the settings of the microphone and speaker in the live venue, the audio in the viewer video may cause feedback in the live venue. When feedback occurs, it is necessary to quickly cut off the relevant viewer video (especially the audio signal) in order to significantly impede the performance of the performers on stage. The viewer video is searched for based on the frequency of the audio signal causing feedback, and a cut-off command is issued to the viewer video causing the feedback. Regarding feedback, there is also a method of using a howling suppressor that suppresses feedback by setting a notch filter for the frequency at which feedback occurs.

<Embodiment 5: Video signal cut-off command reception unit (K): Cut-off command: Command based on quality of viewer's video>
In addition to howling, if a viewer is shown a viewer image (regardless of the form, such as video, image, still image, video, text, audio, etc.) that violates public order and morals and that the viewer would be reluctant to output to the monitor (A) in the viewer image, the blocking order should also be issued. Even if the viewer image in question has not been selected, it is preferable to remove it from the viewer identification information of the selection candidates waiting for selection. The violation of public order and morals mentioned above includes not only the viewer himself, but also the act of transmitting a viewer image that violates public order and morals to the viewer image transmission server (F). It may also include an act that violates the notices and terms and conditions of the live event organizer, etc., to which the viewer agreed when purchasing a ticket to receive the distribution of the live video.

<Embodiment 5: Video signal cutoff command reception unit (K): Cutoff command: Viewer video quality determination configuration example>
An example of a configuration for determining whether the viewer video is offensive to public order and morals will be described.
The venue terminal (C)
A viewer image judgment rule storage unit that stores rules for judging the quality of a viewer image;
The video signal may be configured to include a viewer video judgement section for judging the quality of the viewer video signal.

<Fifth embodiment: Shutdown command execution means (L) (1411)>
The 'cutoff command execution means (L)' (1411) is configured in the viewer video selection unit (G) (1407) so as to terminate the selection of the video signal of the viewer video for which the video signal cutoff command acceptance unit (K) has accepted the cutoff command.

The cut-off command issued by the operator of the venue terminal (C) is received and executed by the cut-off command execution means (L). For example, when the viewing terminal (D) used by a viewer viewing live video connects to this system to view the live video, it transmits information about the viewing terminal (D) (e.g., MAC address, IP address, etc.) and viewer information (e.g., viewer identification information, user attribute information, etc.) to this system. Then, the viewer identification information associated with the viewing terminal (D) that is the sender of the viewer video output to the monitor (A) is used to end the selection of the viewer video that is the target of the cut-off command (end the output to the monitor (A)). Simply ending the selection of the viewer video that is the target of the cut-off command will result in a portion of the corresponding part of the monitor (A) that is not displayed (e.g., a black display portion), so it is desirable to configure it so that another viewer video is selected instead and output to the monitor (A) at the same time as the selection is ended.

<Embodiment 5: Shutdown command execution unit (L): Shutdown command execution example for periodic display replacement>
An example of switching of viewer videos displayed on monitor (A) will be explained using Figures 22 and 23. Figure 22 shows an example of a display on one of monitors (A) displaying viewer videos in a venue. A total of 12 sets of viewers are displayed, 4 horizontally and 3 vertically. When the specified time arrives, a command to shut off the output of the viewer video currently displayed in Figure 22 is issued, and selection of the corresponding viewer video is terminated. In order to prevent the display on monitor (A) from being interrupted, video selection of the viewer videos of the other 12 sets of viewers is immediately started, resulting in the display shown in Figure 23.

<Embodiment 5: Shutdown command execution unit (L): Example of shutdown command execution due to violation of regulations>
An example of a case where the selection of a specific viewer video is cancelled for some reason will be described. This is an example of a live distribution held under the rule that the viewer video is displayed on the monitor (A) at the venue only when one person is viewing per ticket when selling tickets. In the display example of the monitor (A) in FIG. 22, two people are shown in the viewer video in the lower right section, which violates the rule. Therefore, the selection of the viewer video was cancelled (FIG. 24). The section on the monitor (A) is displayed in black because the output of the viewer video has been cancelled. As shown in FIG. 25, the selection of another viewer video is immediately started, and another viewer video is output to the section on the monitor (A), and the black display state is resolved. Note that the black display is shown in FIG. 24 for the sake of explanation, but it is desirable to switch as quickly as possible so that the audience and performers at the venue do not notice the black display.

<Processing flow of embodiment 5>
Fig. 15 is a flowchart of the operation process of the two-way live distribution system, which is a computer of the fifth embodiment based on the second embodiment. As shown in this figure, the two-way live distribution system of the fifth embodiment has a venue terminal preparation step (c) (S1501), a shooting step (b) (S1502), a live video transmission step (e) (S1503), a viewer video transmission step (f) (S1504), a video signal cut-off command reception step (k) (S1505), a viewer video selection step (g) (S1506), a cut-off command execution substep (l) (S1507) in the viewer video selection step (g), and a monitor step (a) (S1508). The same effect can be obtained even if either the third or fourth embodiment is used as a base.

The operation method of the two-way live distribution system, which is a computer, is as follows:
The venue terminal preparation step (c) (S1501) performs processing for connecting the monitor (A) and the image capture device (B) to a venue terminal (C) that are arranged at a live performance venue;
The photographing step (b) (S1502) is a process in which a photographing device (B) is installed at a live performance venue, photographs live images, and transmits the images to a venue terminal (C) (not limited to a single terminal) described later.
A live video transmission step (e) (S1503) performs a process of transmitting the live video from the venue terminal (C) via a network from the live video transmission server (E) to a viewing terminal (D) used by a viewer to view the live video;
A viewer video transmission step (f) (S1504) performs a process of transmitting the viewer video from the viewing terminal (D) to a venue terminal (C) at the live venue from a viewer video transmission server (F) (which may be the same as the live video transmission server (E));
The video signal cutoff command receiving step (k) (S1505) performs processing,
The viewer image selection step (g) (S1506) of the hall terminal (C) selects a viewer image to be output to the monitor (A) from among a plurality of viewer images received from the viewing terminal (D) from the viewer image transmission server (F),
The cut-off command execution sub-step (l) (S1507) in the viewer image selection step (g) performs processing,
The monitor step (a) (S1508) performs processing for outputting viewer video from a venue terminal (C) (described later) placed in the live venue to a monitor (A).
This is an operating method for causing a two-way live distribution system, which is a computer, to execute such a series of processes.

Fifth Embodiment: Description of Hardware
16 is a diagram for explaining hardware consisting of a main board and its peripheral devices of an interactive live distribution system, which is a computer of embodiment 5 based on embodiment 2. It is composed of a "CPU", a "chipset" consisting of a north bridge and a south bridge, a "non-volatile memory", a "main memory", an "I/O controller", "USB, IEEE1394, LAN terminal, etc.", a "BIOS", a "PCI slot", a "real-time clock", etc. The same effect can be obtained even if either embodiment 3 or 4 is used as a base.

The various programs and data (information) stored in the non-volatile memory are expanded into the main memory when the system is started, and the programs are configured to perform calculations using the data in sequence by the CPU upon receiving execution commands.

In other words, the non-volatile memory contains, in addition to the OS (operating system) and device drivers, a venue terminal preparation program (c), a shooting program (b), a live video transmission program (e), a viewer video transmission program (f), a viewer video selection program (g), a monitor program (a), a video signal cutoff command reception program (k), and a cutoff command execution subprogram (l), and also holds viewer video, live video, and cutoff commands as data. These are expanded in the main memory when the system is started up on the computer, and when a start-up command is received, the CPU sequentially performs calculations using the programs and data.

This embodiment makes it possible to block the viewer video being output to the monitor in the venue and terminate the video output.

<5. Effects>
In the two-way live distribution system having the above configuration, by outputting viewer video, which is video of viewers watching the live performance from outside the venue, to the monitor (A) at the venue, the real-time reactions of the viewers can be delivered to the performers on the stage at the venue not only by audio but also by video. The viewer video is expected to increase the motivation of the performers. By appropriately selecting the viewer video based on the viewer identification information, the viewers can be classified and output on the monitor (A). A cutoff command can be issued to stop the selection of the viewer video (stop the output to the monitor (A)) for some reason.

Two-way live distribution system...0100
Monitor (A)...0101
Photographing device (B) ... 0102
Venue terminal (C) ... 0103
Viewing terminal (D) ... 0104
Live video transmission server...0105
Viewer video transmission server (F) ... 0106

Claims (15)

A monitor (A) is placed at the live venue and outputs the viewer's video from the venue terminal (C) described below;
A camera (B) is placed at the live venue to capture live footage and transmit it to a venue terminal (C) (not limited to a single terminal) described later;
a venue terminal (C) that is placed in a live venue and connected to the monitor (A) and the imaging device (B);
a live video transmission server (E) that transmits the live video from the venue terminal (C) via a network to a viewing terminal (D) that is used by a viewer to view the live video;
A viewer video transmission server (F) (which may be the same as the live video transmission server (E)) that transmits viewer video from the viewing terminal (D) to a venue terminal (C) at the live venue;
A two-way live streaming system consisting of: The interactive live distribution system according to claim 1, wherein the venue terminal (C) has a viewer video selection unit (G) that selects a viewer video to be output to the monitor (A) from among multiple viewer videos received from the viewing terminal (D) from the viewer video transmission server (F). The hall terminal (C) has a viewer identification information storage unit (H) that stores viewer identification information for identifying a viewer (including the case of a viewing terminal (D)),
3. The interactive live distribution system according to claim 2, wherein the viewer video selection unit (G) is configured to select a viewer video using viewer identification information. The interactive live distribution system according to claim 3, wherein the viewer identification information storage unit (H) has an authority information storage means (J) that stores viewer identification information in association with authority information that is information indicating the authority of the viewer identified by the viewer identification information. The venue terminal (C) has a video signal cut-off command receiving unit (K) that receives a cut-off command from an operator of the venue terminal (C) to cut off a video signal of a viewer video from a viewing terminal (D) that is a transmission source of the viewer video output to the monitor (A),
The viewer video selection unit (G) has a cut-off command execution means (L) for terminating the selection of the video signal of the viewer video for which the cut-off command has been received by the video signal cut-off command receiving unit (K).
The interactive live distribution system according to any one of claims 2 to 4. A monitor step (a) of outputting a viewer image from a venue terminal (C) (described later) disposed in a live performance venue to a monitor (A);
A photographing step (b) in which a photographing device (B) is installed at a live venue and photographs live images and transmits the images to a venue terminal (C) (not limited to a single terminal) described later;
a venue terminal preparation step (c) for connecting the monitor (A) and the imaging device (B) to a venue terminal (C) disposed at a live performance venue;
a live video transmission step (e) of transmitting the live video from the venue terminal (C) via a network from a live video transmission server (E) to a viewing terminal (D) used by a viewer to view the live video;
a viewer video transmission step (f) of transmitting the viewer video from the viewer terminal (D) to a venue terminal (C) at the live venue from a viewer video transmission server (F) (which may be the same as the live video transmission server (E));
A method for operating a two-way live distribution system comprising: The method of operating the interactive live distribution system, which is a computer according to claim 6, includes a viewer video selection step (g) in which the venue terminal (C) selects a viewer video to be output to the monitor (A) from among multiple viewer videos received from the viewing terminal (D) from the viewer video transmission server (F). The hall terminal (C) has a viewer identification information storage step (h) for storing viewer identification information for identifying a viewer (including the case where the viewer is a viewing terminal (D)),
8. The method for operating a two-way live distribution system that is a computer according to claim 7, wherein said viewer video selection step (g) is configured to select a viewer video using viewer identification information. The method of operating the interactive live distribution system, which is a computer according to claim 8, wherein the viewer identification information holding step (h) includes an authority information holding substep (j) of associating and holding viewer identification information with authority information, which is information indicating the authority of the viewer identified by the viewer identification information. the venue terminal (C) has a video signal cut-off command receiving step (k) of receiving a cut-off command, which is a command to cut off a video signal of a viewer video from the viewing terminal (D) which is a transmission source of the viewer video output to the monitor (A), from an operator of the venue terminal (C);
The viewer video selection step (g) includes a cut-off command execution sub-step (l) of terminating the selection of the video signal of the viewer video for which a cut-off command has been received in the video signal cut-off command receiving step (k),
A method for operating a two-way live distribution system comprising the computer according to any one of claims 7 to 9. A monitor step (a) of outputting a viewer image from a venue terminal (C) (described later) disposed in a live performance venue to a monitor (A);
A photographing step (b) in which a photographing device (B) is installed at a live venue and photographs live images and transmits the images to a venue terminal (C) (not limited to a single terminal) described later;
a venue terminal preparation step (c) for connecting the monitor (A) and the imaging device (B) to a venue terminal (C) disposed at a live performance venue;
a live video transmission step (e) of transmitting the live video from the venue terminal (C) via a network from a live video transmission server (E) to a viewing terminal (D) used by a viewer to view the live video;
a viewer video transmission step (f) of transmitting the viewer video from the viewer terminal (D) to a venue terminal (C) at the live venue from a viewer video transmission server (F) (which may be the same as the live video transmission server (E));
A program that can be read and operated by a two-way live distribution system that is a computer comprising the steps of: A program that can be read and operated by the computer of claim 11, in which the venue terminal (C) has a viewer video selection step (g) for selecting a viewer video to be output to the monitor (A) from among multiple viewer videos received from the viewing terminal (D) from the viewer video transmission server (F). The hall terminal (C) has a viewer identification information storage step (h) for storing viewer identification information for identifying a viewer (including the case where the viewer is a viewing terminal (D)),
13. The program readable and operable by a computer that is an interactive live distribution system according to claim 12, wherein said viewer video selection step (g) is configured to select a viewer video by using viewer identification information. The program readable and operable for the interactive live distribution system, which is a computer according to claim 13, includes a viewer identification information holding step (h) and an authority information holding substep (j) for holding viewer identification information in association with authority information that is information indicating the authority of the viewer identified by the viewer identification information. the venue terminal (C) has a video signal cut-off command receiving step (k) of receiving a cut-off command, which is a command to cut off a video signal of a viewer video from the viewing terminal (D) which is a transmission source of the viewer video output to the monitor (A), from an operator of the venue terminal (C);
The viewer video selection step (g) includes a cut-off command execution sub-step (l) of terminating the selection of the video signal of the viewer video for which a cut-off command has been received in the video signal cut-off command receiving step (k),
15. A program readable and operable by the computer according to claim 12, comprising:

Images