JP2021174239A - Computer program, server device, and method - Google Patents

Abstract

To provide a computer program, a server device, a terminal device, and a display method which detect a position of a distributor (performer) in a physical space with improved accuracy.SOLUTION: The computer program relating to one embodiment is executed by at least one processor to allow the at least one processor to function to: acquire movement information about respective movements of a plurality of parts of a body of a performer existing in a physical space, which are output from a plurality of first sensors attached to the plurality of parts; acquire position information about a position of the body of the performer, which is output from at least one second sensor attached to the body of the performer; calculate respective positions of the plurality of parts in the physical space on the basis of the movement information; and calculate the position of the body in the physical space on the basis of the position information.SELECTED DRAWING: Figure 5

Description

The technology disclosed in this application relates to computer programs, server devices and methods for displaying images based on the behavior of a distributor (performer).

Conventionally, a motion capture system has been used to generate an image based on the movement of a distributor (performer). Japanese Unexamined Patent Publication No. 2017-511906 (Patent Document 1) discloses a motion capture system using an inertial sensor. The motion capture system disclosed in Patent Document 1 determines the position of each part in the body based on signals received from inertial sensors attached to a plurality of parts in the body of the distributor. The entire Patent Document 1 is incorporated in the present specification by citation.

Japanese Unexamined Patent Publication No. 2017-511906

However, in the motion capture system using the inertial sensor described above, when the position (coordinates) of the distributor in the physical space is calculated using the inertial sensor, the position calculated in this way is the passage of time. Due to the noise accumulated with it, it may differ significantly from the actual distributor position.

Accordingly, some embodiments disclosed in this application provide computer programs, server devices, terminal devices and display methods that detect the position of a distributor (performer) in physical space with improved accuracy.

The computer program according to the embodiment is described as "a plurality of computer programs output from a plurality of first sensors attached to a plurality of parts of the distributor's body existing in a physical space by being executed by at least one processor. The motion information regarding the motion of each of the parts of the body is acquired, the position information regarding the position of the body output from at least one second sensor attached to the body of the distributor is acquired, and the motion information is based on the motion information. It is possible to make the at least one processor function so as to calculate the position of each of the plurality of parts in the physical space and calculate the position of the body in the physical space based on the position information. " can.

The server device according to one aspect is "equipped with at least one processor, and the at least one processor outputs from a plurality of first sensors attached to a plurality of parts of the distributor's body existing in a physical space. The motion information regarding the motion of each of the plurality of parts is acquired, and the position information regarding the position of the body output from at least one second sensor attached to the body of the distributor is acquired, and the motion information is obtained. The position of each of the plurality of parts in the physical space is calculated based on the above, and the position of the body in the physical space is calculated based on the position information. "

A method according to one aspect is "a method executed by at least one processor that executes a computer-readable instruction, wherein the at least one processor executes the instruction to bring it into physical space. A first acquisition step of acquiring motion information regarding the operation of each of the plurality of parts output from a plurality of first sensors attached to a plurality of parts in the body of an existing distributor, and attachment to the body of the distributor. A second acquisition step of acquiring position information regarding the position of the body output from at least one second sensor, and calculating the position of each of the plurality of parts in the physical space based on the motion information. A second calculation step of calculating the position of the body in the physical space based on the position information is included. ”

FIG. 1 is a block diagram showing an example of a configuration of a communication system according to an embodiment. FIG. 2 is a table illustrating which device can execute each operation executed by the communication system 1 shown in FIG. FIG. 3 is a block diagram schematically showing an example of the hardware configuration of the terminal device 20 (server device 30) shown in FIG. FIG. 4 is a block diagram schematically showing an example of the functions of the terminal device 20 (server device 30) shown in FIG. FIG. 5 is a flow chart showing an example of the operation performed by the communication system 1 shown in FIG. FIG. 6 is a flow chart showing an example of an operation performed by the body position information calculation unit 150 (250) in the communication system 1 shown in FIG. FIG. 7 is a flow chart showing another example of the operation performed by the body position information calculation unit 150 (250) in the communication system 1 shown in FIG.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. The same reference numerals are attached to common components in the drawings. It should also be noted that the components represented in one drawing may be omitted in another for convenience of explanation. Furthermore, it should be noted that the attached drawings are not always drawn to the correct scale.

1. 1. Outline of Communication System In the communication system disclosed in the present application, the following operations are simply performed. First, a distributor (performer) in a physical space such as a studio attaches a plurality of first sensors (for example, inertial sensors) to a plurality of parts of the body. In addition, the distributor attaches at least one second sensor (eg, an optical sensor) to any part of the body (eg, the back). The "distribution device" receives operation information output from a plurality of first sensors and position information output from at least one second sensor, for example, for each unit time while the distributor is performing. Can be obtained. Here, the distribution device includes any device including, for example, the following device.
-Distributor's terminal device-Terminal device installed in a studio, etc. and / or a studio unit-Server device installed in a studio, etc.

(Aspect in which the distribution device generates a moving image)
Next, the distribution device can calculate the position of each part of the distributor in the physical space based on the acquired operation information. In addition, the distribution device can calculate the position of the distributor's body in the physical space based on the acquired position information. Further, the distribution device can convert the position of each part in the acquired physical space into the position of each part in the virtual space. Furthermore, the distribution device can convert the acquired body position in the physical space into the body position in the virtual space. Further, the distribution device can generate a moving image including an animation of the distributor's avatar and transmit it to the server device based on the position of each part and the position of the body in the acquired virtual space. This server device can distribute the moving image received from the terminal device of the distributor or the like to the terminal device of each viewer. As a result, the terminal device of each viewer can play back the moving image received from the server device.

(Aspect in which the server device generates a moving image)
In parallel with this, the distribution device can transmit the acquired operation information and location information to the server device. This server device can calculate the position of each part of the distributor in the physical space based on the acquired operation information. In addition, this server device can calculate the position of the distributor's body in the physical space based on the acquired position information. Further, this server device can convert the acquired position of each part in the physical space into the position of each part in the virtual space. Furthermore, the server device can convert the acquired body position in the physical space into the body position in the virtual space. Further, this server device can generate a video including an animation of the distributor's avatar and send it to each viewer's terminal device based on the position of each part and the position of the body in the acquired virtual space. can. As a result, the terminal device of each viewer can play back the moving image received from the server device.

(A mode in which each viewer's terminal device generates a moving image)
Alternatively, this server device can transmit the acquired operation information and position information to each viewer's terminal device. The terminal device of each viewer can calculate the position of each part of the distributor in the physical space based on the acquired operation information. In addition, each viewer's terminal device can calculate the position of the distributor's body in the physical space based on the acquired position information. Further, the terminal device of each viewer can convert the position of each part in the acquired physical space into the position of each part in the virtual space. Furthermore, each viewer's terminal device can convert the acquired body position in the physical space into the body position in the virtual space. Further, the terminal device of each viewer can generate and display a moving image including an animation of the distributor's avatar based on the position of each part and the position of the body in the acquired virtual space.

2. Example of communication system

FIG. 1 is a block diagram showing an example of a configuration of a communication system according to an embodiment. As shown in FIG. 1, the communication system 1 may include one or more terminal devices 20 connected to the communication network 10 and one or more server devices 30 connected to the communication network 10. can. In addition, in FIG. 1, three terminal devices 20A to 20C are exemplified as an example of the terminal device 20, and three server devices 30A to 30C are exemplified as an example of the server device 30. However, as the terminal device 20, one or more terminal devices 20 other than these may be connected to the communication network 10. Further, as the server device 30, one or more server devices 30 other than these can be connected to the communication network 10.

Further, the communication system 1 may include one or more studio units 40 connected to the communication network 10. Note that FIG. 1 illustrates two studio units 40A and 40B as examples of the studio unit 40. However, as the studio unit 40, one or more studio units 40 other than these may be connected to the communication network 10.

The communication network 10 can include, without limitation, a mobile phone network, a wireless LAN, a fixed telephone network, the Internet, an intranet and / or Ethernet (registered trademark) and the like.

FIG. 2 is a table illustrating which device can execute each operation executed by the communication system 1 shown in FIG.

First, focus on aspect (1). From (A) acquisition of operation information and position information to (D) generation of moving images can be executed by the distribution device. (E) The display of the moving image can be executed by the distribution device and the terminal device of each viewer.

Next, focus on aspect (2). In the route (2A), (A) acquisition of moving image information and location information to (E) display of moving image can be executed by the distribution device.

In the route (2B) that can be executed in parallel with the route (2A), (B) calculation of the position in the physical space to (D) generation of the moving image can be executed by the server device. (E) The display of the moving image can be executed by the terminal device of each viewer.

Next, focus on aspect (3). The operation performed on the route (3A) is the same as the operation performed on the route (2A) of the above-described aspect (2).

In the route (3B) that can be executed in parallel with the route (3A), (B) the calculation of the position in the physical space and (C) the calculation of the position in the virtual space can be executed by the server device. (D) video generation and (E) video display can be performed by each viewer's terminal device.

Next, focus on aspect (4). The operation performed on the route (4A) is the same as the operation performed on the route (2A) of the above-described aspect (2).

In the route (4B) which can be executed in parallel with the route (4A), (B) the calculation of the position in the physical space can be executed by the server device. From (C) calculation of the position in the virtual space to (E) display of the moving image can be executed by the terminal device of each viewer.

Next, focus on aspect (5). The operation performed on the route (5A) is the same as the operation performed on the route (2A) of the above-described aspect (2).

In the route (5B) that can be executed in parallel with the route (5A), (B) calculation of the position in the physical space to (E) display of the moving image can be executed by the terminal device of each viewer.

The distributor's terminal device 20, the terminal device 20 installed in the studio, and the studio unit 40 included in the "distribution device" execute a predetermined installed application (application for video distribution, etc.). Therefore, the above-described operation assigned in each of the cases (1) to (5) can be executed. Alternatively, the distributor's terminal device 20 included in the "distribution device", and the terminal device 20 and the studio unit 40 installed in the studio or the like can be executed from the server device 30 by executing the installed web browser. A web page can be received and displayed to perform similar actions.

In the present specification, the "predetermined application" or "specific application" may be one or more applications, or a combination of one or more applications and one or more middleware. It may be.

The terminal device 20 of each viewer also executes the installed predetermined application (application for video distribution, etc.) to execute the above-described operation assigned in each of the cases (1) to (5). Can be executed. Alternatively, the distributor's terminal device 20 included in the "distribution device", and the terminal device 20 and the studio unit 40 installed in the studio or the like can be executed from the server device 30 by executing the installed web browser. A web page can be received and displayed to perform similar actions.

The distributor's terminal device 20 and each viewer's terminal device 20 can include, but are not limited to, smartphones, tablets, mobile phones (feature phones), and / or personal computers.

The terminal device 20 and the studio unit 40 installed in a studio or the like can include, but are not limited to, a personal computer, a supercomputer, and / or a quantum computer and the like.

The server device 30 included in the "distribution device" executes the installed specific application and functions as an application server, thereby performing the above-described operations assigned in each of the cases (1) to (5). Can be executed. Further, the server device 30 included in the "distribution device" executes a specific installed application and functions as a web server, thereby transmitting to each terminal device 20 and / or a studio unit or the like via a web page. , A similar operation can be performed.

The server device 30 can include, but is not limited to, a personal computer, a supercomputer, and / or a quantum computer and the like.

3. 3. Hardware Configuration of Each Device Next, an example of the hardware configuration of each of the terminal device 20, the server device 30, and the studio unit 40 will be described.

3-1. Hardware Configuration of Terminal Device 20 A hardware configuration example of each terminal device 20 will be described with reference to FIG. FIG. 3 is a block diagram schematically showing an example of the hardware configuration of the terminal device 20 (server device 30) shown in FIG. In FIG. 3, reference numerals in parentheses are described in relation to each server device 30 as described later.

Each terminal device 20 can mainly include a central processing unit 21, a main storage device 22, an input / output interface device 23, an input device 24, an auxiliary storage device 25, and an output device 26. .. These devices are connected by a data bus and / or a control bus.

The central processing unit 21 is sometimes referred to as a "CPU", and can perform calculations on instructions and data stored in the main storage device 22 and store the results of the calculations in the main storage device 22. can. Further, the central processing unit 21 can control the input device 24, the auxiliary storage device 25, the output device 26, and the like via the input / output interface device 23. The terminal device 20 can include one or more such central processing units 21.

The main storage device 22 is sometimes referred to as a “memory”, and is an instruction received from an input device 24, an auxiliary storage device 25, a communication network 10 or the like (server device 30 or the like) via an input / output interface device 23. And data, and the calculation result of the central processing apparatus 21 can be stored. The main storage device 22 can include RAM (random access memory), ROM (read-only memory) and / or flash memory without limitation.

The auxiliary storage device 25 is a storage device having a larger capacity than the main storage device 22. The auxiliary storage device 25 stores instructions and data (computer programs) that constitute the specific application, web browser, and the like, and is controlled by the central processing unit 21 to store these instructions and data (computer programs). Can be transmitted to the main storage device 22 via the input / output interface device 23. The auxiliary storage device 25 can include, but is not limited to, a magnetic disk device and / or an optical disk device.

The input device 24 is a device that takes in data from the outside, and may include a touch panel, buttons, a keyboard, a mouse, and / or a sensor without limitation. As will be described later, the sensor includes, without limitation, a first-class sensor including one or more cameras and / or a second-class sensor including one or more microphones and the like. Can be done.

The output device 26 can include, but is not limited to, a display device, a touch panel, and / or a printer device.

In such a hardware configuration, the central processing unit 21 sequentially loads and loads the instructions and data (computer programs) that constitute a specific application stored in the auxiliary storage device 25 into the main storage device 22. Instructions and data can be calculated. As a result, the central processing device 21 controls the output device 26 via the input / output interface device 23, or another device (for example, the server device 30) via the input / output interface device 23 and the communication network 10. And other terminal devices 20 and the like), various information can be transmitted and received. As a result, the terminal device 20 can execute the above-described operation with reference to FIG. 2 by executing the installed specific application.

The terminal device 20 may include one or more microprocessors and / or a graphics processing unit (GPU) in place of the central processing unit 21 or together with the central processing unit 21.

3-2. Hardware Configuration of Server Device 30 A hardware configuration example of each server device 30 will be described with reference to FIG. The hardware configuration of each server device 30 can be, for example, the same as the hardware configuration of each terminal device 20 described above. Therefore, reference numerals for the components of each server device 30 are shown in parentheses in FIG.

Each server device 30 can mainly include a central processing unit 31, a main storage device 32, an input / output interface device 33, an input device 34, an auxiliary storage device 35, and an output device 36. .. These devices are connected by a data bus and / or a control bus.

The central processing device 31, the main storage device 32, the input / output interface device 33, the input device 34, the auxiliary storage device 35, and the output device 36 are included in the terminal devices 20 described above, respectively. It can be substantially the same as the device 22, the input / output interface device 23, the input device 24, the auxiliary storage device 25, and the output device 26.

In such a hardware configuration, the central processing unit 31 sequentially loads and loads the instructions and data (computer programs) that constitute a specific application stored in the auxiliary storage device 35 into the main storage device 32. Instructions and data can be calculated. As a result, the central processing device 31 controls the output device 36 via the input / output interface device 33, or another device (for example, each terminal device) via the input / output interface device 33 and the communication network 10. Various information can be sent and received with (20 etc.).

As a result, the server device 30 can execute a specific installed application and function as an application server and / or a web server. As a result, the server device 30 can execute the above-described operation with reference to FIG.

The server device 30 may include one or more microprocessors and / or a graphics processing unit (GPU) in place of the central processing unit 31 or together with the central processing unit 31.

3-3. Hardware Configuration of Studio Unit 40 The studio unit 40 can be mounted by an information processing device such as a personal computer. Although not shown, the studio unit 40 mainly includes a central processing unit, a main storage device, an input / output interface device, an input device, and an auxiliary device, similarly to the terminal device 20 and the server device 30 described above. A storage device and an output device can be included. These devices are connected by a data bus and / or a control bus.

The studio unit 40 can execute a specific installed application and function as an information processing device. As a result, the studio unit 40 can perform the above-described operation with reference to FIG.

The studio unit 40 may include one or more microprocessors and / or a graphics processing unit (GPU) in place of or in combination with the central processing unit.

4. Functions of Each Device Next, an example of the functions of each of the terminal device 20, the server device 30, and the studio unit 40 will be described.

4-1. Function of Terminal Device 20 FIG. 4 is a block diagram schematically showing an example of the function of the terminal device 20 (server device 30) shown in FIG. As shown in FIG. 4, the terminal device 20 includes a storage unit 100, a sensor unit 110, an operation information acquisition unit 120, a position information acquisition unit 130, a site position information calculation unit 140, and a body position information calculation unit 150. The conversion unit 160, the moving image generation unit 170, the display unit 180, the user interface unit 190, and the communication unit 200 can be included.

(1) Storage unit 100
It is possible to store various information including, for example, the following information required for image distribution and / or image reception.
-Operation information received from the first sensor (for example, inertial sensor) attached to the performer-Position information received from the second sensor (for example, optical sensor) attached to the performer-Part position information calculated based on the operation information -Body position information calculated based on position information-Part position information converted from coordinates in physical space to coordinates in virtual space-Converted from coordinates in physical space to coordinates in virtual space Body position information-Videos generated based on the part position information converted to coordinates in virtual space and the body position information converted to coordinates in virtual space, etc.

(2) Sensor unit 110
The sensor unit 110 can include sensors such as various types of cameras and / or microphones and at least one processor that processes the information acquired by such sensors. The sensor unit 110 can acquire data (image and / or voice, etc.) about the face of the performer facing the sensor unit 110, and further execute information processing on this data.

Specifically, for example, the sensor unit 110 can first acquire image data regarding the performer's face for each unit time using various types of cameras. Here, the unit time can be set / changed to an arbitrary length at an arbitrary timing via the user interface unit 190 by the user, the performer, or the like. Further, the sensor unit 110 can measure the position of each of the plurality of parts on the performer's face every unit time by using the image data acquired in this way. Here, the plurality of parts are, as described above, the right eye, the left eye, the right eyebrow, the left eyebrow, the nose, the mouth, the right ear, the left ear, the chin, the right cheek, the left cheek, and / or among these parts. A part or the like in any part of the above can be included without limitation. As a method for measuring the positions of a plurality of parts on the performer's face every unit time using the image data acquired by the camera, various methods well known to those skilled in the art can be used.

For example, in one embodiment, the sensor unit 110 can include, as a sensor, an RGB camera that captures visible light and a near-infrared camera that captures near-infrared light. As such a camera, for example, a True Depth camera of iPhone X (registered trademark) can be used. As the True Depth camera, the camera disclosed at https://developer.apple.com/documentation/arkit/arfaceanchor can be used. The matters described on this website are incorporated herein by reference in their entirety.

Regarding the RGB camera, the sensor unit 110 can generate data recorded for each unit time by associating the image acquired by the RGB camera with a time code (a code indicating the time when the image was acquired). This data can be, for example, an MPEG file.

Further, the sensor unit 110 generates data recorded for each unit time by associating a numerical value (for example, a floating point numerical value) indicating a predetermined number (for example, 51) depths acquired by the near-infrared camera with the time code. can do. This data can be, for example, a TSV file. Here, the TSV file is a file in a format in which a plurality of data are recorded by separating the data with tabs.

Regarding the near-infrared camera, specifically, first, the dot projector can emit an infrared laser including a dot pattern to the performer's face. Further, the near-infrared camera can capture the infrared dots projected and reflected on the performer's face and generate an image of the infrared dots captured in this way. The sensor unit 110 compares the image of the dot pattern radiated by the pre-registered dot projector with the image captured by the near-infrared camera, and compares each point (for example, 51 points and 51 points) in both images. The depth of each point (each part) can be calculated by using the deviation of the position in each part). Here, the depth of each point (each part) can be the distance between each point (each part) and the near-infrared camera. The sensor unit 110 can generate data recorded for each unit time by associating the numerical value indicating the depth calculated in this way with the time code as described above.

As a result, the sensor unit 110 acquires a moving image such as an MPEG file and the position (coordinates, etc.) of each part as data (measurement data) related to the performer's face for each unit time in association with the time code. be able to.

By using Argumented Faces, the sensor unit 110 can acquire the following information every unit time by using the image captured by the camera.
(1) Physical center position of the performer's skull,
(2) A face mesh that includes hundreds of vertices that make up the performer's face and is defined for the center position, and
(3) Positions (coordinates) of each of a plurality of parts (for example, right cheek, left cheek, and apex of nose) on the performer's face identified based on the above (1) and (2).

By using this technique, the sensor unit 110 can acquire the positions (coordinates) of each of the plurality of parts on the performer's face for each unit time.

The sensor unit 110 can acquire a voice signal by performing well-known signal processing on the voice data related to the utterance and / or singing of the performer output from the microphone or the like. This audio signal may be, for example, an MPEG file or the like.

(3) Operation information acquisition unit 120 and site position information calculation unit 140
A plurality of first sensors (for example, inertial sensors) (not shown) are attached to a plurality of parts of the performer's body in a physical space such as a studio. The multiple parts of the performer's body include, for example, the head, jaw, right shoulder, right elbow, right wrist, back of right hand, finger of right hand, left shoulder, left elbow, left wrist, back of left hand, finger of left hand, and right hip joint. , Right knee, right ankle, right instep, right toe, left hip, left knee, left ankle, left instep, left toe, back and hips, etc. can be selected from multiple sites. Each of these plurality of first sensors can include at least one of a three-dimensional accelerometer, a three-dimensional gyroscope, a three-dimensional magnetic field meter, and the like. Each of the plurality of first sensors can transmit a sensor signal (operation information) for each unit time. As a plurality of first sensors, for example, "MVN BIOMECH" (http://0c7.co.jp/products/xsens/products/mvn_biomech/) provided by Ixens can be used. .. The entire content disclosed on the above website is incorporated herein by reference in its entirety.

The operation information acquisition unit 120 can acquire sensor signals (operation information) transmitted by each of the plurality of first sensors.

The part position information calculation unit 140 is based on the sensor information of each of the plurality of first sensors acquired by the operation information acquisition unit 120, and the position information of each of the plurality of first sensors (that is, a plurality of parts in the body). (And orientation, etc.) can be calculated. As a method for acquiring the position information of the sensor based on the sensor information received from each first sensor, for example, Japanese Patent Application Laid-Open No. 2017-511906, Japanese Patent Application Laid-Open No. 2013-500812 and Japanese Patent No. 5388473 It is possible to use the well-known methods described. It should be noted that these documents are incorporated herein by reference in their entirety.

(4) Position information acquisition unit 130 and body position information calculation unit 150
At least one second sensor (for example, an optical sensor) (not shown) is attached to any part (for example, the back) of the performer's body in a physical space such as a studio. Further, two base stations (first base station and second base station) (not shown) may be installed near the performer in a physical space such as a studio.

The first base station and the second base station can be, for example, a multi-axis laser emitter. The first base station can, for example, emit a flashing light for synchronization and then scan the laser light around the vertical axis. The second base station can, for example, emit a flashing light for synchronization and then scan the laser light around the horizontal axis.

The second sensor can detect the incident of blinking light and laser light from the first base station and the second base station. Further, the second sensor is based on the time difference between the incident timing of the blinking light and the incident timing of the laser light, the light receiving time by the second sensor, the incident angle of the laser light detected by the second sensor, and the like. 2 The position information (and orientation, etc.) of the sensor can be detected. Furthermore, the second sensor can transmit the position information detected by the second sensor every unit time. The position information acquisition unit 130 can acquire the position information transmitted by the second sensor every unit time in this way.

As such a second sensor, a first base station, and a second base station, for example, "HTC VIVE" provided by HTC can be used. The entire content disclosed on the above website is incorporated herein by reference in its entirety.

The body position information calculation unit 150 can calculate the body position information based on the position information of the second sensor acquired by the position information acquisition unit 130. The method of calculating the position information of the body will be described later with reference to FIG.

(5) Conversion unit 160
The conversion unit 160 can convert the position information of each of the plurality of first sensors calculated by the site position information calculation unit 140 from the coordinates in the physical space to the coordinates in the virtual space. Further, the conversion unit 160 can convert the position information of the second sensor calculated by the body position information calculation unit 150 from the coordinates in the physical space to the coordinates in the virtual space. The physical space is the coordinates in the space where the performer exists (for example, a studio). The virtual space is the coordinates in the space where the performer's avatar exists.

(6) Video generator 170
For example, the moving image generation unit 170 can generate a moving image including an animation of an avatar that follows the movement of the performer's body by using the following information.
-Data (measurement data) related to the performer's body acquired by the sensor unit 110
-Position information of each of a plurality of first sensors (plural parts in the body) calculated by the site position information calculation unit 140-Body position information calculated by the body position information calculation unit 150

For example, the moving image generation unit 170 can generate a moving image in which the avatar performs the following movements, for example, following the movement of the performer's body.
・ Avatar keeps a straight face and blinks ・ Avatar keeps a straight face and looks down ・ Avatar moves mouth and eyes ・ Avatar moves in virtual space ・ Avatar moves multiple parts of the body To work

Such generation of moving images can be realized by the moving image generation unit 170 performing, for example, the following processing.
-Process of drawing the face of the avatar based on the measurement data (coordinates of each of the plurality of parts in the face) about the performer's body acquired by the sensor unit 110-The position of the body calculated by the body position information calculation unit 150. Processing to draw an avatar at the coordinates determined based on the information-Draw the part corresponding to the first sensor at the coordinates determined based on the position information of each first sensor calculated by the part position information calculation unit 140. Further, the generation of such a moving image can be realized by using various rendering techniques well known to those skilled in the art.

The file storing the moving image generated by the moving image generation unit 170 in this way can be stored in the storage unit 100.

(7) Display unit 180
The display unit 180 may include, for example, a touch panel and / or a display panel. Such a display unit 180 can reproduce and display a file in which a moving image generated by the moving image generation unit 170 and stored by the storage unit 100 is stored.

(8) User interface unit 190
The user interface unit 190 can include a touch panel, a mouse and / or a keyboard and the like. Such a user interface unit 190 can generate information indicating the content of the operation performed by the performer.

(9) Communication unit 200
The communication unit 200 can communicate various information required for the distribution of the moving image and / or the reception of the moving image with the server device 30 via the communication network 10. In particular, when the terminal device 20 operates as the terminal device 20 of the performer (distributor), the communication unit 200 can transmit the moving image stored in the storage unit 100 to the server device 30. When the terminal device 20 operates as the viewer's terminal device 20, the communication unit 200 can receive the moving image delivered by the distributor's terminal device 20 via the server device 30.

4-2. Function of Server Device 30 and Function of Studio Unit 40 An example of the function of the server device 30 will be described with reference to FIG. As the function of the server device 30, for example, at least a part of the functions of the terminal device 20 described above can be used. Therefore, reference numerals for the components of the server device 30 are shown in parentheses in FIG. Further, since the studio unit has the same configuration as the terminal device 20 or the server device 30 shown in FIG. 4, it is possible to perform the same operation as the terminal device 20 or the server device 30.

5. Operations Performed by Communication System 1 Next, an example of operations performed by the communication system 1 described above will be described with reference to FIG. FIG. 5 is a flow chart showing an example of the operation performed by the communication system 1 shown in FIG.

5-1. Overall Operation As shown in FIG. 5, first, in step (hereinafter referred to as “ST”) 402, the distribution device can acquire measurement data, operation information, and position information, for example, for each unit time. The distribution device may store the measurement data, the operation information, and / or the position information acquired in this way in the storage unit 100 (200). The measurement data can be acquired by the sensor unit 110 (210). The operation information and the position information can be acquired by the operation information acquisition unit 120 (220) and the position information acquisition unit 130 (230), respectively. The operation performed in ST402 can be included in the operation (A) shown in FIG.

In ST404, the distribution device (part position information calculation unit 140 (240)) can calculate the position information of each part in the physical space based on the operation information acquired in ST402. The operation performed in ST404 can be included in the operation (B) shown in FIG.

In ST406, the distribution device (body position information calculation unit 150 (250)) can calculate the position information of the body in the physical space based on the position information acquired in ST402. The specific processing performed in ST406 will be described later with reference to FIG. The operation performed in ST406 can be included in the operation (B) shown in FIG.

In ST408, the distribution device (part position information calculation unit 140 (240)) changes the position information of each part calculated in ST404 from the three-dimensional coordinates in the physical space to the three-dimensional coordinates in the virtual space. Can be converted. Since the size of the physical space in which the performer performs (for example, a studio) and the size of the virtual space in which the avatar is placed are not the same, they are virtual from the three-dimensional coordinates in such a physical space. Position information is converted to three-dimensional coordinates in a typical space (for example, a room of a karaoke box). The operation performed in ST408 may be included in the operation (C) shown in FIG.

In ST410, the distribution device (body position information calculation unit 150 (250)) converts the body position information calculated in ST406 from three-dimensional coordinates in physical space to three-dimensional coordinates in virtual space. can do. Since the size of the physical space in which the performer performs (for example, a studio) and the size of the virtual space in which the avatar is placed (for example, a karaoke box room) are not the same, such physical The position information is converted from the three-dimensional coordinates in the space to the three-dimensional coordinates in the virtual space. The operation performed in ST410 can be included in the operation (C) shown in FIG.

In ST412, the distribution device (the moving image generation unit 170 (270)) receives the measurement data acquired in ST402, the position information of each part converted into three-dimensional coordinates in the virtual space in ST408, and the virtual in ST410. It is possible to generate a moving image including an animation of an avatar based on the position information of the body converted into three-dimensional coordinates in a typical space and the information including the information. Further, the distribution device (communication unit 200 (300)) can transmit the moving image generated in this way to the server device 30. As a result, the terminal device 20 of each viewer can receive the moving image generated in ST414 via the server device 30. The operation performed in ST412 can be included in the operation (D) shown in FIG.

In ST414, the distribution device and the terminal device 20 of each viewer can play and display the moving image generated in ST414. The operation performed in ST414 may be included in the operation (E) shown in FIG.

In ST416, the distribution device can determine whether or not to continue the process. If it is determined that the process is to be continued, the process returns to ST402 described above. On the other hand, when it is determined that the processing is stopped, the processing ends.

The process in each step shown in FIG. 5 may be executed at least partially in parallel with the process in another step. For example, in ST402, in parallel with the distribution device acquiring measurement data, operation information, and position information every unit time, the distribution device executes the operations in each of ST404 to ST414, and each viewing The terminal device 20 of the user can execute the operation in ST414.

5-2. Specific Example of Processing Performed by Body Position Information Calculation Unit 150 (250) Next, refer to FIG. 6 for an example of specific processing performed by the body position information calculation unit 150 (250) in ST406 described above. explain. FIG. 6 is a flow chart showing an example of an operation performed by the body position information calculation unit 150 (250) in the communication system 1 shown in FIG.

Referring to FIG. 6, in ST502, the distribution device can execute preprocessing on the position information acquired in ST402 every first unit time. In the pre-processing, the process of converting the position information acquired in ST402 into the position information based on the predetermined origin (predetermined coordinates) and / or the position information acquired in ST402 is stored in a predetermined number of digits. Processing etc. may be included.

In ST504, the distribution device acquires the position information acquired in the previous first unit time (hereinafter referred to as "previous first unit time") and the current first unit time (hereinafter referred to as "current first unit time"). It is possible to calculate the amount of change between the obtained position information and the obtained position information.

In ST506, the distribution device can determine whether or not the amount of change calculated in ST504 exceeds the threshold value. When it is determined in ST506 that the amount of change is equal to or less than the threshold value, in ST508, the distribution device can determine the position information acquired in the current first unit time as effective position information. Such effective position information is used in ST410 described above. After this, the process returns to ST502 described above.

On the other hand, when it is determined in ST506 that the amount of change exceeds the threshold value, the process shifts to ST510. In this phenomenon, the amount of change exceeds the threshold value, the blinking light or other light emitted by the first base station and / or the second base station is reflected by some object and attached to the distributor. It is conceivable that this occurs due to (not limited to) incident on the second sensor.

In ST510, the distribution device can discard the position information acquired in the current first unit time. In ST512, the distribution device can perform the same processing as in ST502 described above. In ST514, the distribution device receives the position information acquired in the new first unit time before (the first unit time before different from the first unit time used so far) and the new current first unit time (this). It is possible to calculate the amount of change between the position information acquired in the current 1st unit time different from the current 1st unit time used up to now) and the current 1st unit time.

In ST516, the distribution device can determine whether or not the amount of change calculated in ST504 exceeds the threshold value. When it is determined in ST516 that the amount of change is less than or equal to the threshold value, in ST518, the distribution device validly uses the position information acquired in the current first unit time (same as the new current first unit time in ST514). It can be determined as position information. Such effective position information is used in ST410 described above. After this, the process returns to ST502 described above.

On the other hand, when it is determined in ST516 that the amount of change exceeds the threshold value, the process returns to ST510 described above.

According to the process shown in FIG. 6, when the amount of change between the position information acquired in the previous first unit time and the position information acquired in the current first unit time is equal to or less than the threshold value, the distribution device The position information acquired in the current first unit time can be used as effective position information.

On the other hand, when the amount of change exceeds the threshold value, the distribution device can discard the position information acquired in the current first unit time. After that, the distribution device makes such a new position until the amount of change between the position information acquired in the new previous first unit time and the position information acquired in the new current first unit time becomes equal to or less than the threshold value. Continue to discard the location information acquired in the current unit time. Further, when the amount of change becomes equal to or less than the threshold value, the distribution device can use the position information acquired in the current first unit time at such a time point as effective position information. As a result, if there is an error in the position information received from the second sensor due to light reflection or the like, the distribution device continues to discard such position information, and then acquires it from the second sensor. Appropriate location information can be used as valid location information.

In the example shown in FIG. 6, between the interval (ST502, ST512) in the time when the distribution device acquires the position information from the second sensor and the two position information used by the distribution device to acquire the change amount. The case where the intervals (ST504, ST514) are the same (every first unit time) has been described. However, as illustrated in FIG. 7, the interval (ST604, ST614) between the two position information used by the distribution device to acquire the amount of change is set at the time when the distribution device acquires the position information from the second sensor. It is also possible to make it longer than the interval (ST602, ST612). That is, in FIG. 7, the second unit time can be made longer than the first unit time. The operation in the example shown in FIG. 7 is the same as the operation in the example shown in FIG. 6 except that the first unit time and the second unit time are used in combination.

6. Modification example The position information of each part calculated by the distribution device in ST404 shown in FIG. 5 has a large error due to the accumulation of errors that occur over time, depending on the performance of the first sensor used. May be included. In order to deal with such a problem, the distribution device can correct the position information of each part calculated in ST404 at a predetermined timing (for example, periodically).

For example, the distribution device can correct the position information of each part calculated in ST404 based on the "valid position information" acquired in ST406. Specifically, in the first method, in the distribution device, the body position information acquired as valid position information in ST406 is, for example, (Δx, Δy) from the body position information acquired as valid position information last time. , Δz), the position information of each part acquired in ST404 is relatively shifted by (Δx, Δy, Δz) or (k1 · Δx, k2 · Δy, k3 · Δz). can do. Here, k1, k2, and k3 are arbitrary coefficients, respectively.

Further, in the examples shown in FIGS. 6 and 7, the first unit time and / or the second unit time may be a unit time that can be changed at an arbitrary timing.

Further, in the various embodiments described above, cases where an inertial sensor and an optical sensor are used as the first sensor and the second sensor, respectively, have been described. However, each of the different first and second sensors can be selected from a variety of well-known sensors, including optical (formula) sensors, inertial (formula) sensors, magnetic (formula) sensors and mechanical (formula) sensors. Can be done. Even in such a case, as the position information calculated using the position information from the second sensor, only the position information in which the amount of change per unit time is equal to or less than the threshold value is used as effective position information. As a result, among the position information calculated using the position information from the second sensor, the position information having an error exceeding a certain magnitude is discarded. Therefore, it is possible to detect the position of the distributor's body with improved accuracy. Further, based on the amount of change in the position information per unit time calculated using the position information from the second sensor, the operation information received from each of the plurality of first sensors (plural parts) is used. The position information calculated in the above can be corrected. As a result, the position information of the portion where each first sensor is attached can also be detected with improved accuracy.

Furthermore, in the examples shown in FIGS. 6 and 7, the distribution device executes the operations (A) to (E) shown in FIG. 2, and the terminal device 20 of each viewer executes the operation (E). (The case corresponding to the aspect (1) in FIG. 2) has been described. However, the operations (A) to (E) may be shared and executed by a plurality of devices as shown in the aspects (2) to (5) in FIG.

In this case, when two adjacent operations are executed by devices that are different from each other, the device that executes the operation that occurs first of these two operations obtains information or the like obtained by the operation, and the operation that occurs later. Must be sent to the device that runs.

For example, focusing on the route (2B) of the aspect (2), the distribution device transmits the operation information and the position information (measurement data) obtained by the operation (A) to the server device that executes the operation (B). There is a need. Similarly, the server device needs to transmit the moving image obtained by the operation (D) to the terminal device of each viewer.

As another example, focusing on the route (5B) of the aspect (5), the distribution device views the operation information and the position information (measurement data) obtained by the operation (A), for example, via the server device. It is necessary to send to the terminal device of the person.

In the route (3B) of the aspect (3), the route (4B) of the aspect (4), and the route (5B) of the aspect (5) shown in FIG. 2, the terminal device of each viewer delivers. Instead of displaying the video generated by the device, the video generated (drawn) by the viewer's terminal device itself can be displayed (in this method, the viewer's terminal device (client) renders (this method). It is sometimes called "client rendering" because it performs drawing). As a result, the terminal device of each viewer generally does not need to receive the moving image having a large capacity, so that the moving image including the animation of the distributor's avatar can be displayed at a higher speed. That is, it is possible to significantly suppress the time difference between the moving image displayed by the distribution device and the moving image displayed by the terminal device of each viewer.

7. Various Aspects The computer program according to the first aspect is described as "by being executed by at least one processor, from a plurality of first sensors attached to a plurality of parts of the distributor's body existing in a physical space. The output motion information regarding each movement of the plurality of parts is acquired, and the position information regarding the position of the body output from at least one second sensor attached to the body of the distributor is acquired, and the motion is described. The at least one processor is used to calculate the position of each of the plurality of parts in the physical space based on the information and calculate the position of the body in the physical space based on the position information. Can "make it work".

In the first aspect, the computer program according to the second aspect "determines whether or not the body position calculated based on the position information can be used as an effective position. One processor can work. "

In the second aspect, the computer program according to the third aspect "calculates the position of the body based on the position information every first unit time, and the amount of change in the position of the body in the first unit time. Is less than or equal to the threshold, the at least one processor may function to use the body position calculated based on the position information as an effective position during the first unit time. "

In the third aspect, the computer program according to the fourth aspect states, "When the amount of change in the position of the body in the first unit time exceeds the threshold value, the computer program is based on the position information in the first unit time. The at least one processor can be made to function so as to discard the calculated position of the body. "

In the fourth aspect, the computer program according to the fifth aspect states that "the change in the position of the body in the new first unit time occurs after the first unit time in which the amount of change in the position of the body exceeds the threshold value". The at least one processor is made to function so that when the amount returns below the threshold, the position of the body calculated based on the position information is used as an effective position in the new first unit time. " be able to.

In the second aspect, the computer program according to the sixth aspect "calculates the position of the body based on the position information every first unit time, and the second unit time longer than the first unit time. The at least one processor functions to use the body position calculated based on the position information as an effective position in the second unit time when the amount of change in the body position is less than or equal to the threshold value. You can "make it".

In the sixth aspect, the computer program according to the seventh aspect states, "When the amount of change in the position of the body in the second unit time exceeds the threshold value, the computer program is based on the position information in the second unit time. The at least one processor can be made to function so as to discard the calculated position of the body. "

In the seventh aspect, the computer program according to the eighth aspect states that "the change in the position of the body occurs after the second unit time when the amount of change in the position of the body exceeds the threshold value". The at least one processor is made to function so that when the amount returns below the threshold, the position of the body calculated based on the position information is used as an effective position in the new second unit time. " be able to.

The computer program according to the ninth aspect "can change the first unit time" in any one of the third aspect to the fifth aspect.

The computer program according to the tenth aspect "can change the first unit time and / or the second unit time" in any one of the sixth aspect to the eighth aspect.

In any of the third to tenth aspects, the computer program according to the eleventh aspect "sets the effective position of the body calculated based on the position information of the body in a virtual space. The at least one processor is made to function so as to convert into a position and convert each position of the plurality of parts calculated based on the operation information into each position of the plurality of parts in a virtual space. "be able to.

The computer program according to the twelfth aspect is the delivery in the eleventh aspect, "based on the position of the body in the virtual space and the position of each of the plurality of parts in the virtual space. The at least one processor can be made to function so as to generate a moving image containing an animation of a person's avatar. "

The computer program according to the thirteenth aspect is described in any one of the first to the twelfth aspects, wherein "the at least one processor is a central processing unit (CPU), a microprocessor and / or a graphics processing unit (1). GPU) can be included.

The computer program according to the fourteenth aspect is the at least one such that "the acquired operation information and / or the position information is stored in the storage unit" in any one of the first to thirteenth aspects. Two processors can work. "

The computer program according to the fifteenth aspect is described in any one of the first to the fourteenth aspects, wherein "the terminal device in which at least one processor is installed in the studio, the terminal device of the distributor, and the viewing user". Can be mounted on a terminal device and / or a server device.

The server device according to the sixteenth aspect is described as "a plurality of first sensors including at least one processor, wherein the at least one processor is attached to a plurality of parts of the distributor's body existing in a physical space. The motion information regarding the movement of each of the plurality of parts output from is acquired, and the position information regarding the position of the body output from at least one second sensor attached to the body of the distributor is acquired. It is possible to calculate the position of each of the plurality of parts in the physical space based on the motion information, and calculate the position of the body in the physical space based on the position information. "

The server device according to the seventeenth aspect can be "the at least one processor includes a central processing unit (CPU), a microprocessor and / or a graphics processing unit (GPU)" in the sixteenth aspect.

The server device according to the eighteenth aspect has the at least one processor so as to "store the acquired operation information and / or the position information in the storage unit" in the sixteenth aspect or the seventeenth aspect. Can "make it work".

A method according to a nineteenth aspect is "a method executed by at least one processor that executes a computer-readable instruction, wherein the at least one processor executes the instruction to be physical. A first acquisition step of acquiring motion information regarding the operation of each of the plurality of parts output from a plurality of first sensors attached to a plurality of parts of the distributor's body existing in space, and the distributor's body. A second acquisition step of acquiring position information regarding the position of the body output from at least one second sensor attached to the body, and the position of each of the plurality of parts in the physical space based on the motion information. A first calculation step for calculating the position of the body in the physical space based on the position information, and a second calculation step for calculating the position of the body in the physical space can be included. "

According to the twentieth aspect, in the nineteenth aspect, "each step is executed by the terminal device installed in the studio or the at least one processor mounted on the terminal device of the distributor". can.

The method according to the 21st aspect is that in the 19th aspect, "the first acquisition step and the second acquisition step are mounted on a terminal device installed in a studio or a terminal device of the distributor, at least one of the above. The at least one processor executed by one processor, wherein the first calculation step and the second calculation step are mounted on a terminal device installed in the studio or a server device connected to the terminal device of the distributor. Can be executed by.

The method according to the 22nd aspect is that in the 19th aspect, "the first acquisition step and the second acquisition step are mounted on a terminal device installed in a studio or a terminal device of the distributor, at least one of the above. The first calculation step and the second calculation step are executed by one processor, and are mounted on a terminal device installed in the studio or a terminal device connected to the distributor's terminal device via a server device. It can also be executed by the at least one processor. "

The method according to the 23rd aspect is described in the 19th aspect, "Each step is mounted on a terminal device installed in a studio or a terminal device connected to the distributor's terminal device via a server device. It can also be executed by the at least one processor. "

In any of the 19th to 23rd aspects, the method according to the 24th aspect is "the at least one processor is a central processing unit (CPU), a microprocessor and / or a graphics processing unit (GPU). ) Can be included.

The method according to the 25th aspect is the operation information and / or the operation information acquired by the at least one processor executing the instruction in any one of the 19th aspect to the 24th aspect. The position information can be stored in the storage unit. "

As described above, according to various aspects, it is possible to provide a computer program, a server device, a terminal device, and a display method for detecting the position of a distributor (performer) in a physical space with improved accuracy.

1 Communication system 10 Communication network 20 (20A to 20C) Terminal device 30 (30A to 30C) Server device 40 (40A, 40B) Studio unit 100 (200) Storage unit 110 (210) Sensor unit 120 (220) Operation information acquisition unit 130 (230) Position information acquisition unit 140 (240) Part position information calculation unit 150 (250) Body position information calculation unit 160 (260) Conversion unit 170 (270) Video generation unit 180 (280) Display unit 190 (290) User Interface part 200 (300) Communication part

Claims (25)

By running on at least one processor
The motion information regarding the motion of each of the plurality of parts output from the plurality of first sensors attached to the plurality of parts of the distributor's body existing in the physical space is acquired.
The position information regarding the position of the body output from at least one second sensor attached to the body of the distributor is acquired, and the position information is obtained.
Based on the motion information, the positions of the plurality of parts in the physical space are calculated.
The position of the body in the physical space is calculated based on the position information.
A computer program characterized in that the at least one processor functions as described above. It is determined whether or not the body position calculated based on the position information can be used as an effective position.
The computer program according to claim 1, wherein the at least one processor functions as described above. The position of the body is calculated based on the position information every first unit time, and the position of the body is calculated.
When the amount of change in the position of the body in the first unit time is equal to or less than the threshold value, the position of the body calculated based on the position information in the first unit time is used as an effective position.
The computer program according to claim 2, wherein the at least one processor functions as described above. When the amount of change in the position of the body in the first unit time exceeds the threshold value, the position of the body calculated based on the position information is discarded in the first unit time.
The computer program according to claim 3, wherein the at least one processor is made to function as described above. When the amount of change in the position of the body returns to the threshold or less in the new first unit time that occurs after the first unit time in which the amount of change in the position of the body exceeds the threshold value, the new first unit time Use the body position calculated based on the position information as an effective position.
The computer program according to claim 4, wherein the at least one processor is made to function as described above. The position of the body is calculated based on the position information every first unit time, and the position of the body is calculated.
When the amount of change in the position of the body in the second unit time longer than the first unit time is equal to or less than the threshold value, the position of the body calculated based on the position information in the second unit time is an effective position. To use as
The computer program according to claim 2, wherein the at least one processor functions as described above. When the amount of change in the position of the body in the second unit time exceeds the threshold value, the position of the body calculated based on the position information is discarded in the second unit time.
The computer program according to claim 6, wherein the at least one processor is made to function as described above. When the amount of change in the position of the body returns to the threshold or less in the new second unit time that occurs after the second unit time in which the amount of change in the position of the body exceeds the threshold value, the new second unit time Use the body position calculated based on the position information as an effective position.
The computer program according to claim 7, wherein the at least one processor is made to function as described above. The computer program according to any one of claims 3 to 5, wherein the first unit time can be changed. The computer program according to any one of claims 6 to 8, wherein the first unit time and / or the second unit time can be changed. The effective position of the body calculated based on the position information is converted into the position of the body in a virtual space.
The position of each of the plurality of parts calculated based on the operation information is converted into the position of each of the plurality of parts in the virtual space.
The computer program according to any one of claims 3 to 10, wherein the at least one processor functions as described above. A moving image including an animation of the performer's avatar is generated based on the position of the body in the virtual space and the position of each of the plurality of parts in the virtual space.
11. The computer program of claim 11, which causes the at least one processor to function as described above. The computer program according to any one of claims 1 to 12, wherein the at least one processor includes a central processing unit (CPU), a microprocessor and / or a graphics processing unit (GPU). The acquired operation information and / or the position information is stored in the storage unit.
The computer program according to any one of claims 1 to 13, which causes the at least one processor to function as described above. One of claims 1 to 14, wherein the at least one processor is mounted on a terminal device installed in a studio, a terminal device of the distributor, a terminal device of a viewing user, and / or a server device. The listed computer program. Equipped with at least one processor
The at least one processor
Obtaining motion information regarding the motion of each of the plurality of parts output from a plurality of first sensors attached to a plurality of parts of the performer's body existing in the physical space,
The position information regarding the position of the body output from at least one second sensor attached to the body of the distributor is acquired, and the position information is obtained.
Based on the motion information, the positions of the plurality of parts in the physical space are calculated.
The position of the body in the physical space is calculated based on the position information.
A server device characterized by that. The server device according to claim 16, wherein the at least one processor includes a central processing unit (CPU), a microprocessor and / or a graphics processing unit (GPU). The acquired operation information and / or the position information is stored in the storage unit.
The server device according to claim 16 or 17, wherein the at least one processor functions as described above. A method performed by at least one processor that executes computer-readable instructions.
When the at least one processor executes the instruction,
A first acquisition step of acquiring motion information regarding the operation of each of the plurality of parts output from a plurality of first sensors attached to a plurality of parts of the distributor's body existing in a physical space.
A second acquisition step of acquiring position information regarding the position of the body output from at least one second sensor attached to the body of the distributor.
A first calculation step of calculating the position of each of the plurality of parts in the physical space based on the operation information, and
A second calculation step of calculating the position of the body in the physical space based on the position information, and
A method characterized by including. 19. The method of claim 19, wherein each step is performed by a terminal device installed in a studio or at least one processor mounted on the distributor's terminal device. The first acquisition step and the second acquisition step are executed by the terminal device installed in the studio or the at least one processor mounted on the terminal device of the distributor.
A claim in which the first calculation step and the second calculation step are executed by the at least one processor mounted on a terminal device installed in the studio or a server device connected to the terminal device of the distributor. Item 19. The method according to item 19. The first acquisition step and the second acquisition step are executed by the terminal device installed in the studio or the at least one processor mounted on the terminal device of the distributor.
The first calculation step and the second calculation step are mounted on a terminal device installed in the studio or a terminal device connected to the terminal device of the distributor via a server device, at least one of the above. 19. The method of claim 19, which is performed by a processor. 19. The step is executed by at least one processor mounted on a terminal device installed in a studio or a terminal device connected to the distributor's terminal device via a server device. The method described. The method of any of claims 19 to 23, wherein the at least one processor comprises a central processing unit (CPU), a microprocessor and / or a graphics processing unit (GPU). The method according to any one of claims 19 to 24, wherein the at least one processor stores the acquired operation information and / or the position information in the storage unit by executing the instruction.

Images