JP2024029089A - Computer programs, methods and server devices - Google Patents

Abstract

The present invention provides a computer program, a server device, a terminal device, and a display method that display images based on a performer's actions using a new method. [Solution] A terminal device 20 or a server device 30 includes a storage unit that stores various information required for distributing images and/or receiving images, and acquires image data regarding the performer's body every unit time. a sensor section; a change amount acquisition section that obtains the amount of change per unit time in each of a plurality of parts of the performer's body based on the acquired measurement data regarding the performer's body; a reference value acquisition unit that acquires a reference value based on the amount of change per unit time of a plurality of parts of the body; An identification unit that identifies the body part; and a determination unit that determines any one specific movement associated with the at least one body part with the largest identified change amount per unit time as the movement performed by the performer. . [Selection diagram] Figure 3

Description

The technology disclosed in the present application relates to a computer program, method, and server device that display images based on the actions of a performer (user) in various applications.

A service called "Animoji" is known as a service that uses technology to control the facial expression of a virtual character displayed in an application based on the facial expression of a performer (Non-Patent Document 1). With this service, users can change the facial expressions of their avatars displayed in messenger applications by changing their facial expressions while looking at a smartphone equipped with a camera that detects facial deformation.

Furthermore, as another service, a service called "custom cast" is known (Non-Patent Document 2). In this service, the user assigns one of a number of prepared facial expressions to each of a plurality of flick directions on the smartphone screen. Furthermore, when distributing a video, the user can make the avatar displayed in the video express that facial expression by flicking the screen along the direction corresponding to the desired facial expression.

In addition, each of the above-mentioned non-patent documents 1 and 2 is incorporated into the present specification by reference.

"Using Animoji on iPhone X and later", [online], October 24, 2018, Apple Japan Co., Ltd., [searched on January 10, 2020], Internet (URL: https://support.apple.com /ja-jp/HT208190) "Custom Cast", [online], October 3, 2018, Dwango Co., Ltd., [searched on January 10, 2020], Internet (URL: https://customcast.jp/)

In applications that display virtual characters (such as avatars), it is sometimes desired that the characters express impressive facial expressions. Impressive facial expressions include, for example, the following three examples. The first example is an expression in which the shape of the face is unrealistically deformed like a cartoon. This facial expression includes, for example, a facial expression in which both eyes protrude from the face. A second example is an expression in which symbols, graphics and/or colors are added to the face. This facial expression includes, for example, a facial expression with tears streaming down its face, a facial expression with a bright red face, an angry facial expression with triangular eyes, and the like. The third example is a facial expression that expresses emotions including happiness, anger, sadness, and happiness. Impressive facial expressions are not limited to these examples.

However, first, the technique described in Patent Document 1 changes the facial expression of a virtual character to follow changes in the shape of the user's (performer's) face. Therefore, with the technique described in Patent Document 1, it is difficult to express the above-mentioned impressive facial expressions, which are difficult to actually express on a user's face, in the facial expressions of a virtual character.

Next, in the technique described in Patent Document 2, it is necessary to assign in advance an expression to be expressed by a virtual character to each of a plurality of flick directions. Therefore, the user (performer) needs to recognize all the prepared facial expressions. Furthermore, the total number of facial expressions that can be assigned to multiple flick directions and used at once is limited to less than 10, which is not sufficient.

Therefore, some embodiments disclosed in the present application provide a computer program, a server device, a terminal device, and a display method that display images based on a performer's actions using a new method.

According to one embodiment, a computer program "is executed by at least one processor to perform each of at least one specific motion and at least one region having a large amount of change among a plurality of regions in the performer's body." holding the associated information, identifying at least one region of the performer's body having a large amount of change per unit time using measurement data regarding the performer's body, and using the information to identify the at least one region; ``The processor is made to function so as to determine, as a detected operation, any one specific operation associated with at least one of the identified parts having the largest amount of change per unit time among the operations.'' can.

A computer program according to another aspect includes: ``by being executed by at least one processor, each of at least one specific action and at least one area having the largest amount of change among the plurality of areas in the performer's body; , and identify at least one region of the performer's body having a large amount of change per unit time using measurement data regarding the performer's body transmitted from the performer's terminal device; Using the information, determine, as a detected motion, any one of the at least one specific motion that is associated with the at least one region with the highest identified change amount per unit time. The processor may be operated to perform the following functions.

According to one aspect, a method is a method performed by at least one processor that executes computer-readable instructions, the processor executing the instructions to perform each of the at least one specified operation. Information that associates at least one region with the largest amount of change among a plurality of regions in the performer's body is stored, and the amount of change per unit time in the performer's body is calculated using measurement data regarding the performer's body. identify at least one region with the largest amount of change per unit time, and use the information to determine which one of the at least one specific motion is associated with the at least one region with the largest amount of change per unit time; One specific operation can be determined as a detection operation.

A server device according to one aspect is provided with the following: “The server device includes at least one processor, and the processor executes computer-readable instructions to perform each of the at least one specific action and the performance of multiple parts of the performer's body. At least one part of the performer's body that has a large amount of change per unit time is stored, and the at least one part of the performer's body that has a large amount of change per unit time is identified using measurement data regarding the performer's body. Then, using the information, among the at least one specific motion, any one of the identified motions associated with the at least one region with the largest change per unit time is selected as a detected motion. be able to decide.

FIG. 1 is a block diagram showing an example of the configuration of a communication system according to an embodiment. FIG. 2 is a block diagram schematically showing an example of the hardware configuration of the terminal device 20 (server device 30) shown in FIG. FIG. 3 is a block diagram schematically showing an example of the functions of the terminal device 20 (server device 30) shown in FIG. 1. FIG. 4 is a flow diagram showing an example of operations performed in the entire communication system 1 shown in FIG. FIG. 5 is a flow diagram illustrating an example of the operations related to the generation and transmission of a moving image among the operations illustrated in FIG. 4. FIG. 6 is a diagram schematically showing an example of correspondence information used in the communication system shown in FIG. 1. FIG. 7A is a diagram showing an example of a moving image displayed by the terminal device 20 etc. used in the communication system shown in FIG. 1. FIG. 7A is a diagram showing another example of a moving image displayed by the terminal device 20 or the like used in the communication system shown in FIG. 1. FIG. 7C is a diagram showing yet another example of a moving image displayed by the terminal device 20 or the like used in the communication system shown in FIG. 1.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. Note that common components in the drawings are given the same reference numerals. Also, it should be noted that components depicted in one drawing may be omitted from another drawing for convenience of explanation. Furthermore, it should be noted that the attached drawings are not necessarily drawn to scale.

1. Example of a Communication System In the communication system disclosed in this application, to put it simply, a terminal device or the like installed facing a distribution user (performer) transmits an image (video image) generated based on the operation of the distribution user (performer). and/or still images) can be transmitted to each viewing user's terminal device or the like via a server device or the like.

FIG. 1 is a block diagram showing an example of the configuration of a communication system according to an embodiment. As shown in FIG. 1, the communication system 1 may include one or more terminal devices 20 connected to the communication network 10 and one or more server devices 30 connected to the communication network 10. can. Note that, in FIG. 1, three terminal devices 20A to 20C are illustrated as examples of the terminal device 20, and three server devices 30A to 30C are illustrated as examples of the server device 30. However, as the terminal device 20, one or more terminal devices 20 other than these may be connected to the communication network 10. Further, as the server device 30, one or more server devices 30 other than these may be connected to the communication network 10.

Furthermore, the communication system 1 can include one or more studio units 40 connected to the communication network 10. Note that FIG. 1 illustrates two studio units 40A and 40B as examples of the studio unit 40. However, one or more studio units 40 other than these may be connected to the communication network 10.

In the "first aspect", in the communication system 1 shown in FIG. Data regarding the body of the opposing performer and/or audio data regarding speech and/or singing uttered by the performer can be obtained. Furthermore, this terminal device 20A can transmit an image of a virtual character generated according to the acquired data to the server device 30 (for example, the server device 30A) via the communication network 10. Further, the server device 30A sends the image of the virtual character received from the terminal device 20A to one or more other terminal devices 20 via the communication network 10 and to a predetermined application (an application for viewing a video, etc.). ) to deliver the image to the terminal device 20 that has sent the request for image delivery.

Note that in this specification, a "predetermined application" or a "specific application" may be one or more applications, or a combination of one or more applications and one or more middleware. It may be.

In the "second aspect", in the communication system 1 shown in FIG. Data regarding the body and/or audio data regarding speech and/or singing uttered by the performer may be obtained. Furthermore, this server device 30B transmits the image of the virtual character generated according to the acquired data to one or more terminal devices 20 via a communication network 10 and to a predetermined application (an application for viewing a video, etc.). ) to deliver the image to the terminal device 20 that has sent the request for image delivery.

In the "third aspect", in the communication system 1 shown in FIG. Audio data related to speech and/or singing uttered by a performer can be obtained. Furthermore, this studio unit 40 can generate an image of a virtual character generated according to this data and transmit it to the server device 30. Further, the server device 30 transmits the images acquired (received) from the studio unit 40 to one or more terminal devices 20 via the communication network 10 and executes a predetermined application (such as an application for viewing a video). The image can be distributed to the terminal device 20 that has sent the request for image distribution.

The communication network 10 may include, but is not limited to, a mobile phone network, a wireless LAN, a fixed telephone network, the Internet, an intranet, and/or an Ethernet (registered trademark).

The terminal device 20 performs operations such as acquiring data regarding the performer's body and/or audio data regarding speech and/or singing uttered by the performer by executing a specific installed application. be able to. Further, the terminal device 20 can perform operations such as transmitting an image of a virtual character generated according to the acquired data to the server device 30 via the communication network 10. Alternatively, the terminal device 20 can receive and display a web page from the server device 30 and perform similar operations by running an installed web browser.

The terminal device 20 is any terminal device capable of performing such operations, and may include, without limitation, a smartphone, a tablet, a mobile phone (feature phone), and/or a personal computer. can.

In the "first aspect", the server device 30 can function as an application server by executing a specific installed application. Thereby, the server device 30 receives images of virtual characters from each terminal device 20 via the communication network 10 and distributes the received images to each terminal device 20 via the communication network 10. It is possible to perform actions, etc. Alternatively, the server device 30 can perform similar operations via a web page sent to each terminal device 20 by executing a specific installed application and functioning as a web server.

In the "second aspect", the server device 30 can function as an application server by executing a specific installed application. Thereby, the server device 30 acquires data regarding the body of the performer in the studio or other location where the server device 30 is installed, and/or audio data regarding the speech and/or singing uttered by the performer. It is possible to perform actions, etc. Further, the server device 30 can perform operations such as distributing images of virtual characters generated according to the acquired data to each terminal device 20 via the communication network 10. Alternatively, the server device 30 can perform similar operations via a web page sent to each terminal device 20 by executing a specific installed application and functioning as a web server. Furthermore, the server device 30 can function as an application server by executing a specific installed application. As a result, the server device 30 receives facial expressions from the studio unit 40 installed in the studio or other locations according to data regarding the body of the performer in the studio or the like and/or audio data regarding speech and/or singing uttered by the performer. It is possible to perform operations such as acquiring (receiving) an image of a virtual character that has been changed. Furthermore, the server device 30 can perform operations such as distributing this image to each terminal device 20 via the communication network 10.

The studio unit 40 can function as an information processing device that executes a specific installed application. As a result, the studio unit 40 can acquire data regarding the body of the performer in the studio where the studio unit 40 is installed or other locations, and/or audio data regarding the speech and/or singing uttered by the performer. can. Furthermore, the studio unit 40 can transmit the image of the virtual character generated according to the acquired data to the server device 30 via the communication network 10.

2. Hardware Configuration of Each Device Next, an example of the hardware configuration of each of the terminal device 20, server device 30, and studio unit 40 will be described.
2-1. Hardware configuration of terminal device 20
An example of the hardware configuration of each terminal device 20 will be described with reference to FIG. 2. FIG. 2 is a block diagram schematically showing an example of the hardware configuration of the terminal device 20 (server device 30) shown in FIG. In addition, in FIG. 2, reference numerals in parentheses are described in relation to each server device 30, as will be described later.

As shown in FIG. 2, each terminal device 20 mainly includes a central processing unit 21, a main storage device 22, an input/output interface device 23, an input device 24, an auxiliary storage device 25, and an output device 26. and can include. These devices are connected to each other by a data bus and/or a control bus.

The central processing unit 21 is sometimes referred to as a "CPU" and can perform calculations on instructions and data stored in the main storage device 22 and store the results of the calculations in the main storage device 22. can. Furthermore, the central processing unit 21 can control the input device 24, the auxiliary storage device 25, the output device 26, etc. via the input/output interface device 23. Terminal device 20 may include one or more such central processing units 21 .

The main storage device 22 is sometimes referred to as a "memory" and receives instructions from the input device 24, the auxiliary storage device 25, the communication network 10, etc. (server device 30, etc.) via the input/output interface device 23. and data, as well as the calculation results of the central processing unit 21. The main storage device 22 can include, but is not limited to, RAM (random access memory), ROM (read only memory), and/or flash memory.

The auxiliary storage device 25 is a storage device that has a larger capacity than the main storage device 22. The auxiliary storage device 25 stores instructions and data (computer programs) constituting the specific application, web browser, etc., and is controlled by the central processing unit 21 to store these instructions and data (computer programs). can be sent to the main storage device 22 via the input/output interface device 23. The auxiliary storage device 25 can include, but is not limited to, a magnetic disk device and/or an optical disk device.

The input device 24 is a device that takes in data from the outside, and can include, but is not limited to, a touch panel, a button, a keyboard, a mouse, and/or a sensor. The sensors may include, but are not limited to, a first sensor including one or more cameras, etc., and/or a second sensor including one or more microphones, etc., as described below. .

The output device 26 may include, but is not limited to, a display device, a touch panel, and/or a printer device.

In such a hardware configuration, the central processing unit 21 sequentially loads instructions and data (computer programs) constituting a specific application stored in the auxiliary storage device 25 into the main storage device 22, and Can operate on instructions and data. Thereby, the central processing unit 21 controls the output device 26 via the input/output interface device 23, or also controls other devices (for example, the server device 30) via the input/output interface device 23 and the communication network 10. and other terminal devices 20, etc.).

Thereby, by executing the installed specific application, the terminal device 20 acquires data regarding the performer's body and/or audio data regarding the speech and/or singing uttered by the performer, and the acquired data An operation such as transmitting an image of a virtual character generated according to the above to the server device 30 via the communication network 10 (including various operations described in detail later) can be executed. Alternatively, the terminal device 20 can receive and display a web page from the server device 30 and perform similar operations by running an installed web browser.

Note that the terminal device 20 can include one or more microprocessors and/or graphics processing units (GPUs) instead of or together with the central processing unit 21.

2-2. Hardware Configuration of Server Apparatus 30 An example of the hardware configuration of each server apparatus 30 will be described with reference to FIG. 2 as well. The hardware configuration of each server device 30 can be, for example, the same as the hardware configuration of each terminal device 20 described above. Therefore, reference numerals for components included in each server device 30 are shown in parentheses in FIG. 2.

As shown in FIG. 2, each server device 30 mainly includes a central processing unit 31, a main storage device 32, an input/output interface device 33, an input device 34, an auxiliary storage device 35, and an output device 36. and can include. These devices are connected to each other by a data bus and/or a control bus.

The central processing unit 31, the main memory 32, the input/output interface device 33, the input device 34, the auxiliary storage 35, and the output device 36 are the central processing unit 21, the main memory included in each terminal device 20 described above, respectively. The device 22, the input/output interface device 23, the input device 24, the auxiliary storage device 25, and the output device 26 may be substantially the same.

In such a hardware configuration, the central processing unit 31 sequentially loads instructions and data (computer programs) constituting a specific application stored in the auxiliary storage device 35 into the main storage device 32, and Can operate on instructions and data. Thereby, the central processing unit 31 controls the output device 36 via the input/output interface device 33, or also controls other devices (for example, each terminal device) via the input/output interface device 33 and the communication network 10. 20, etc.), various information can be sent and received.

Thereby, in the "first aspect", the server device 30 can function as an application server by executing a specific installed application. Thereby, the server device 30 receives images of virtual characters from each terminal device 20 via the communication network 10 and distributes the received images to each terminal device 20 via the communication network 10. Operations, etc. (including various operations detailed below) can be performed. Alternatively, the server device 30 can function as a web server by running a specific installed application. Thereby, the server device 30 can perform similar operations via the web page sent to each terminal device 20.

Furthermore, in the “second aspect”, the server device 30 can function as an application server by executing a specific installed application. Thereby, the server device 30 performs an operation of acquiring data regarding the body of the performer in the studio or other location where the server device 30 is installed, and/or audio data regarding the speech and/or singing uttered by the performer. etc. can be executed. Furthermore, the server device 30 performs an operation such as distributing an image of a virtual character generated according to the acquired data to each terminal device 20 via the communication network 10 (including various operations described in detail later). can be executed. Alternatively, the server device 30 can function as a web server by running a specific installed application. Thereby, the server device 30 can perform similar operations via the web page sent to each terminal device 20.

Furthermore, in the "third aspect", the server device 30 can function as an application server by executing a specific installed application. As a result, the server device 30 can generate virtual data based on the data regarding the body of the performer located in the studio where the studio unit 40 is installed or other locations and/or the audio data regarding the speech and/or singing uttered by the performer. It is possible to perform operations such as acquiring (receiving) an image of a character from the studio unit 40 via the communication network 10. Furthermore, the server device 30 can also perform operations such as distributing this image to each terminal device 20 via the communication network 10 (including various operations described in detail later).

Note that the server device 30 can include one or more microprocessors and/or graphics processing units (GPUs) instead of or together with the central processing unit 31.

2-3. Hardware Configuration of Studio Unit 40 The studio unit 40 can be implemented by an information processing device such as a personal computer. Although not shown, the studio unit 40 mainly includes a central processing unit, a main storage device, an input/output interface device, an input device, and an auxiliary device, similar to the terminal device 20 and server device 30 described above. It can include a storage device and an output device. These devices are connected to each other by a data bus and/or a control bus.

The studio unit 40 can function as an information processing device by executing a specific installed application. As a result, the studio unit 40 can acquire data regarding the body of the performer in the studio where the studio unit 40 is installed or other locations, and/or audio data regarding the speech and/or singing uttered by the performer. can. Furthermore, the studio unit 40 can transmit the image of the virtual character generated according to the acquired data to the server device 30 via the communication network 10.

3. Functions of Each Device Next, an example of the functions each of the terminal device 20, server device 30, and studio unit 40 have will be described.
3-1. Functions of terminal device 20
An example of the functions of the terminal device 20 will be described with reference to FIG. 3. FIG. 3 is a block diagram schematically showing an example of the functions of the terminal device 20 (server device 30) shown in FIG. 1.

As shown in FIG. 3, the terminal device 20 includes a storage unit 100, a sensor unit 110, a change amount acquisition unit 120, an identification unit 140, a determination unit 150, an image generation unit 160, a display unit 170, A user interface section 180 and a communication section 190 may be included. The terminal device 20 can further include a reference value acquisition unit 130.

(1) Storage unit 100
The storage unit 100 can store various information required for image distribution and/or image reception. In particular, the storage unit 100 can store correspondence information. In the correspondence information, each of at least one predetermined specific motion is associated with at least one region having a large change amount among a plurality of regions on the performer's body.

Here, the at least one predetermined specific operation may include, for example, at least one of a plurality of operations illustrated below.
(A) At least one of various facial expressions that are difficult for the performer to actually express (hereinafter referred to as "special facial expressions") (B) Various movements using the body (making a heart shape with both hands) at least one of the following)

Note that the special expression shown in (A) above can include, for example, at least one of the expressions (A1) to (A3) shown below.
(A1) An expression in which the shape of the face is unrealistically deformed, like a cartoon. (A2) An expression in which symbols, figures, and/or colors are added to the face. (A3) An expression that expresses emotions, including happiness, anger, sorrow, and happiness.

In addition, multiple parts of the performer's body include the right eye, left eye, right eyebrow, left eyebrow, nose, mouth, right ear, left ear, chin, right cheek, left cheek, neck, right shoulder, left shoulder, right hand, left hand, It may include, without limitation, the breast and/or a portion of any of these areas. Here, the part of any part includes, for example, when the part is the right eye, the right end of the right eye, the left end of the right eye, the center of the right eye, the upper edge of the right eye, and/or the right eye. This may include the lower edges of the eyes, etc.

Furthermore, the correspondence information includes at least one predetermined specific motion, at least one region with the largest amount of change among the plurality of regions on the performer's body, and at least one region with the largest amount of change in the performer's body. It is also possible to associate the order based on the magnitude of the amount of change of the two parts.
Note that a specific example of the correspondence information will be described later.

(2) Sensor section 110
The sensor unit 110 may include sensors such as various types of cameras and/or microphones, and at least one processor to process information acquired by such sensors. The sensor section 110 can acquire data (images and/or sounds, etc.) regarding the body of the performer facing the sensor section 110, and further perform information processing on this data.

Specifically, for example, the sensor unit 110 can first acquire image data regarding the performer's body every unit time using various types of cameras. Here, the unit time can be set or changed to any length at any timing by the user, performer, etc. via the user interface section 180. Furthermore, the sensor unit 110 can measure the position of each of a plurality of parts of the performer's body every unit time using the image data acquired in this way. Here, the multiple parts include the right eye, left eye, right eyebrow, left eyebrow, nose, mouth, right ear, left ear, chin, right cheek, left cheek, neck, right shoulder, left shoulder, right hand, It may include, but is not limited to, the left hand, the chest, and/or a portion of any of these areas. Note that various methods well known to those skilled in the art can be used to measure the positions of multiple parts of the performer's body every unit time using image data acquired by a camera.

For example, in one embodiment, the sensor unit 110 may include, as sensors, an RGB camera that captures images of visible light, and a near-infrared camera that captures images of near-infrared rays. As such a camera, for example, the True Depth camera of iPhone X (registered trademark) can be used. Note that the camera disclosed at https://developer.apple.com/documentation/arkit/arfaceanchor can be used as the true depth camera. The material described on this website is incorporated herein by reference in its entirety.

Regarding the RGB camera, the sensor unit 110 can generate data recorded every unit time by associating an image acquired by the RGB camera with a time code (a code indicating the time at which the image was acquired). This data may be an MPEG file, for example.

Furthermore, the sensor unit 110 generates data recorded every unit time by associating a predetermined number (for example, 51) of numerical values (for example, floating point numerical values) indicating the depth acquired by the near-infrared camera with the above-mentioned time code. can do. This data may be, for example, a TSV file. Here, the TSV file is a file in which a plurality of pieces of data are recorded with data separated by tabs.

Regarding the near-infrared camera, specifically, first, a dot projector can emit an infrared laser including a dot pattern onto the performer's body. Further, a near-infrared camera can capture the infrared dots projected onto and reflected from the performer's body, and generate an image of the infrared dots thus captured. The sensor unit 110 compares the image of the dot pattern emitted by the dot projector registered in advance and the image captured by the near-infrared camera, and calculates each point (for example, 51 points, 51 points) in both images. The depth of each point (each part) can be calculated using the positional shift in each part (in each part). Here, the depth of each point (each part) may be the distance between each point (each part) and the near-infrared camera. The sensor unit 110 can generate data recorded every unit time by associating the numerical value indicating the depth calculated in this manner with the time code as described above.

Thereby, the sensor unit 110 acquires a moving image such as an MPEG file and the position (coordinates, etc.) of each part as data (measured data) regarding the performer's body for each unit time in association with the time code. be able to.

In another embodiment, the sensor unit 110 can utilize a technology called Argumented Faces. As Argumented Faces, information disclosed at https://developers.google.com/ar/develop/java/augmented-faces/ can be used. The information disclosed on this website is incorporated herein by reference in its entirety.

By using Argumented Faces, the sensor unit 110 can acquire the following information every unit time using an image captured by a camera.
(1) The physical center position of the performer's skull;
(2) a face mesh that includes hundreds of vertices that make up the performer's face and is defined relative to the center position; and
(3) The position (coordinates) of each of the multiple parts on the performer's face (e.g., right cheek, left cheek, apex of the nose) identified based on (1) and (2) above.

By using this technique, the sensor unit 110 can acquire the positions (coordinates) of each of a plurality of parts of the performer's upper body (face, etc.) every unit time.

Note that the sensor unit 110 can acquire an audio signal by performing well-known signal processing on the audio data regarding the speaker's speech and/or singing output from a microphone or the like. This audio signal may be, for example, an MPEG file.

(3) Change amount acquisition unit 120
The change amount acquisition unit 120 can acquire the amount of change per unit time of each of a plurality of parts of the performer's body based on the data (measurement data) regarding the performer's body acquired by the sensor unit 110. Specifically, for example, the change amount acquisition unit 120 uses the position (coordinates) acquired in the first unit time and the second position (coordinates) that occurs next (immediately after) the first unit time for a region called the right cheek, for example. It is possible to calculate the difference between the position (coordinates) obtained in the unit time of . Thereby, the change amount obtaining unit 120 can obtain the change amount of the right cheek region between the first unit time and the second unit time. That is, the change amount obtaining unit 120 can obtain the change amount per unit time of the right cheek. The change amount obtaining unit 120 can similarly obtain the change amount per unit time for other parts as well.

The amount of change per unit time of each region can be expressed, for example, by a floating point between 0 and 1.0.
Note that the unit time may be fixed, variable, or a combination thereof. Further, the unit time may be a time corresponding to one frame.

ここで、ｘ_ｉは、前記演者の身体における複数の部位のうちの第ｉ番目の部位の単位時間当たりの変化量である。Ｎは、部位の総数（２以上）である。この数式により算出される値は、２乗平均平方根（Ｒｏｏｔ Ｍｅａｎ Ｓｑｕａｒｅ）と称される。 (4) Reference value acquisition unit 130
The reference value acquisition unit 130 acquires a reference value based on the amount of change per unit time of a plurality of parts of the performer's body based on the data (measured data) regarding the performer's body acquired by the sensor unit 110. In one embodiment, the reference value R may be calculated using the following formula:

Here, x _i is the amount of change per unit time of the i-th part of the plurality of parts in the performer's body. N is the total number of parts (2 or more). The value calculated by this formula is called the root mean square.

The reference value indicates the amount of change per unit time in a plurality of parts of the performer's body. If the reference value is large, it can be estimated that the performer's body has performed one of at least one predetermined specific motion. On the other hand, if the reference value is small, it can be estimated that the performer's body is not performing any of the at least one predetermined specific motion.

In addition, the above reference value may be the root mean square itself shown in the above formula, or may be a value obtained by multiplying this root mean square by an arbitrary coefficient. .

なお、ここでも、ｘ_ｉは、前記演者の身体における複数の部位のうちの第ｉ番目の部位の単位時間当たりの変化量である。Ｎは、部位の総数（２以上）である。 Further, in another embodiment, the reference value R may be an average value of x _i as shown in the following formula.

Note that, also here, x _i is the amount of change per unit time of the i-th part of the plurality of parts in the performer's body. N is the total number of parts (2 or more).

(4) Identification unit 140
The identification unit 140 uses the amount of change per unit time of each of the plurality of parts of the performer's body acquired by the amount of change acquisition unit 120 to identify the top parts with the largest amount of change per unit time among the plurality of parts. At least one site can be identified.

In one embodiment, when an event in which the reference value calculated by the reference value acquisition unit 130 exceeds a threshold is detected, the identification unit 140 identifies at least one region with a large change per unit time. I can do it. At least one site with a large variation may include N sites.

(5) Determination unit 150
The determining unit 150 determines, in the above-mentioned correspondence information stored in the storage unit 100, any one specific action associated with the at least one region with the largest change per unit time identified by the identifying unit 140. , can be determined as an action (detected action) performed by the performer.

Furthermore, the storage unit 100 stores information about each of at least one predetermined specific motion, at least one region with the largest amount of change among the plurality of regions on the performer's body, and at least one region with the largest amount of change. When storing correspondence information that associates the order based on the magnitude of the amount of change of the two parts, the determining unit 150 can also perform the following operation. Specifically, the determining unit 150 determines that the above-mentioned correspondence information stored in the storage unit 100 matches at least one region with the largest change amount per unit time identified by the identifying unit 140, and that the above-mentioned change Any one specific motion that matches the order based on the magnitude of the amount can be determined as the motion (detected motion) performed by the performer.

Note that a specific example of the operation performed by the determining unit 150 will be described later.

(6) Image generation unit 160
The image generation unit 160 uses the data (measurement data) regarding the performer's body acquired by the sensor unit 110 to generate a video that includes an animation of a virtual character that follows the performer's body movements. I can do it. For example, the image generation unit 160 may generate a video in which the virtual character simply blinks while keeping a straight face, a video in which the virtual character simply looks down while maintaining a straight face, and , it is possible to generate videos, etc. in which a virtual character moves its mouth and eyes in accordance with the performer's facial movements. In order to generate such a moving image, the image generation unit 160 changes the position (coordinates) of a specific part (such as the eyes) based on the data (measurement data) regarding the performer's body acquired by the sensor unit 110. This can be achieved by, for example, detecting the change in the area and drawing this specific part based on such change. Moreover, generation of such a moving image can be realized by using various rendering techniques well known to those skilled in the art.

Furthermore, when the determining unit 150 determines that any one of the predetermined at least one specific actions has been executed (one of the specific actions is detected as a detected action), the image generating unit 160 ), the following operations can be performed: Specifically, the image generation unit 160 can generate a video including an animation of a virtual character that moves based on the determined detection motion. For example, the image generation unit 160 first determines how the virtual character changes and/or moves in association with each of the above-described specific actions (A1), (A2), (A3), and/or (B). It is possible to hold information in advance regarding whether to do so. Next, the image generation unit 160 uses information corresponding to the determined detection motion from among the information held in advance to generate a video including an animation in which the virtual character changes and/or moves. can do. Additionally, generation of such moving images can also be achieved using various rendering techniques well known to those skilled in the art.

The image generation unit 160 can cause the storage unit 100 to store a file (for example, a file such as an MPEG file) that stores the moving image generated in this way.
Note that in an embodiment based on VR (Virtual Reality), the image generation unit 160 generates an image (video) in which a virtual character that moves as described above is placed in a virtual space formed by CG or the like. be able to. In an embodiment based on AR (Augmented Reality) or MR (Mixed Reality), the image generation unit 160 can generate an image (video) in which a virtual character that moves as described above is placed in real space. can.

(7) Display section 170
The display unit 170 can include, for example, a touch panel and/or a display panel. Such a display unit 170 can play and display a file containing a moving image stored in the storage unit 100.

(8) User interface section 180
The user interface unit 180 may include a touch panel, a mouse, a keyboard, and the like. Such a user interface unit 180 can generate information indicating the content of the operation performed by the performer (user).

(9) Communication department 190
The communication unit 190 can communicate various information required for image distribution and/or image reception with the server device 30 via the communication network 10. In particular, when the terminal device 20 operates as the terminal device 20 of a performer (distribution user), the communication unit 190 can transmit the moving image stored in the storage unit 100 to the server device 30. When the terminal device 20 operates as the viewing user's terminal device 20, the communication unit 190 can receive the video distributed by the distribution user's terminal device 20 via the server device 30.

3-2. Functions of the server device 30 A specific example of the functions of the server device 30 will be described with reference to FIG. 3 as well. As the functions of the server device 30, for example, part of the functions of the terminal device 20 described above can be used. Therefore, reference symbols for components included in the server device 30 are shown in parentheses in FIG. 3.

First, in the above-mentioned "second aspect", the server device 30 has the storage section 200 to the communication section 290, respectively, except for the differences described below. may be the same as section 190.

The sensors included in the sensor unit 210 may be placed facing the performer in a space where the performer performs, such as in a studio where the server device 30 is installed or at another location. Similarly, a display, a touch panel, etc. included in the display unit 270 may also be placed facing the performer or near the performer in the space where the performer performs.

The communication unit 290 can distribute a file containing moving images stored in the storage unit 200 in association with each performer to the plurality of terminal devices 20 via the communication network 10. Each of the plurality of terminal devices 20 executes an installed predetermined application (for example, a video viewing application) and transmits a signal (request signal) requesting distribution of a desired video to the server device 30. can do. Thereby, each terminal device 20 can receive a desired moving image from the server device 30 that responded to this signal via the predetermined application.

Note that various information (files storing videos, etc.) stored in the storage unit 200 is stored in one or more other server devices (storage) 30 that can communicate with the server device 30 via the communication network 10. It may also be stored.

On the other hand, in the above-mentioned "first embodiment", the sensor section 210 to image generation section 260 used in the above-mentioned "second embodiment" can be used as options. In addition to operating as described above, the communication unit 290 stores in the storage unit 200 a file storing a moving image transmitted by each terminal device 20 and received from the communication network 10, and then transmits the file to a plurality of terminals. It can be distributed to the device 20.

On the other hand, in the "third aspect", the sensor unit 210 to image generation unit 260 used in the above-mentioned "second aspect" can be used as options. In addition to operating as described above, the communication unit 290 stores a file storing a moving image transmitted by the studio unit 40 and received from the communication network 10 in the storage unit 200, and then transmits the file to a plurality of terminal devices. It can be distributed to 20 people.

3-3. Functions of Studio Unit 40 The studio unit has a configuration similar to that of the terminal device 20 or server device 30 shown in FIG. However, the communication unit 190 (290) can transmit the moving image generated by the image generation unit 160 (260) and stored in the storage unit 100 (200) to the server device 30 via the communication network 10.

In particular, the sensor included in the sensor section 110 (210) may be placed facing the performer in a space where the performer performs, such as in a studio where the studio unit 40 is installed or in another location. Similarly, the display, touch panel, etc. included in the display unit 170 (270) may also be placed opposite to or near the performer in the space where the performer performs.

4. Operation of the entire communication system 1 Next, a specific example of the operation of the entire communication system 1 having the above-described configuration will be described with reference to FIG. FIG. 4 is a flow diagram showing an example of operations performed in the entire communication system 1 shown in FIG.

First, in step (hereinafter referred to as "ST") 402, in the case of the first aspect, the terminal device 20 (the image generation unit 160) creates a virtual character using data (measurement data) regarding the performer's body. It is possible to generate videos containing animations. In the case of the second aspect, (the image generation unit 260 of) the server device 30 can perform similar operations. In the case of the third aspect, the studio unit 40 can perform similar operations.

In ST404, in the case of the first aspect, (the communication unit 190 of) the terminal device 20 can transmit the generated video to the server device 30. In the case of the second aspect, (the communication unit 290 of) the server device 30 may not execute ST404 or may transmit the generated video to another server device 30. In the case of the third aspect, the studio unit 40 can transmit the generated video to the server device 30. Note that specific examples of operations executed in ST402 and ST404 will be described later with reference to FIG. 5 and the like.

In ST406, in the case of the first aspect, (the communication unit 290 of) the server device 30 can transmit the video received from the terminal device 20 to another terminal device 20. In the case of the second aspect, the server device 30 (or another server device 30) (the communication unit 290) can transmit the video received from the terminal device 20 to the other terminal device 20. In the case of the third aspect, (the communication unit 290 of) the server device 30 can transmit the video received from the studio unit 40 to another terminal device 20.

In ST408, in the case of the first aspect and the third aspect, the other terminal device 20 (the display unit 170 of) receives the video transmitted by the server device 30 and displays the display of the terminal device 20 or the terminal. It can be displayed on a display connected to the device 20 or the like. In the case of the second aspect, the other terminal device 20 (the display unit 170 of) receives the video transmitted by the server device 30 or another server device 30 and displays the display of the terminal device 20 or the like or the display unit 170 of the other terminal device 20. can be displayed on a display connected to the

In ST410, it is determined whether the operation is to be continued. If it is determined that the operation is to be continued, the process returns to ST402 described above. On the other hand, if it is determined that the operation is not to be continued, the process ends.

Note that FIG. 4 shows a state in which the operations shown in ST402 to ST408 are executed sequentially for the purpose of simplifying the explanation. However, in reality, ST402 to ST408 may be executed in parallel.

5. Operations related to video generation and transmission performed by terminal device 20 etc.Next , among the operations described with reference to FIG. 4, specific operations related to video generation and transmission performed by terminal device 20 in ST402 and ST404 An example will be described with reference to FIG. FIG. 5 is a flow diagram illustrating an example of the operations related to the generation and transmission of a moving image among the operations illustrated in FIG. 4.

Hereinafter, in order to simplify the explanation, attention will be paid to the case where the entity that generates the moving image is the terminal device 20 (that is, the case of the first aspect). However, the entity that generates the video may be the server device 30 (in the case of the second aspect). Furthermore, the entity that generates the moving image may be the studio unit 40 (in the case of the third aspect).

First, as a premise, in ST502, the terminal device 20 (storage unit 100) stores each of at least one predetermined specific motion and at least one region with the largest change amount among the plurality of regions in the performer's body. It holds correspondence information that associates and. The correspondence information stored in the storage unit 100 may be the correspondence information illustrated below.
- Compatibility information generated by the relevant terminal device 20 - Compatibility information generated by an expert, engineer, or other user and sent from the server device 30 etc. for a fee or free of charge - Directly from another user's terminal device 20 or correspondence information indirectly transmitted via the server device 30 or the like. Note that, as described later, the terminal device 20 can also update the correspondence information it holds.

FIG. 6 is a diagram schematically showing an example of correspondence information used in the communication system shown in FIG. 1. The correspondence information illustrated in FIG. 6 can associate each specific motion with, for example, six top regions with the largest amount of change per unit time. The top six sites can include, for example, the first to sixth sites. The first to sixth regions are arranged in order of increasing amount of change per unit time. That is, the amount of change per unit time in the first region is the largest, and the amount of change per unit time in the sixth region is the smallest.

The correspondence information shown in FIG. 6 can, for example, associate the specific action of "angry face" with the top six body parts with the largest amount of change per unit time. The first part to the sixth part are respectively "right end of the mouth", "left end of the mouth", "center of the right eye", "center of the left eye", "right end of the right eyebrow", and " It may be the left end of the left eyebrow.

Further, as another example, the correspondence information shown in FIG. 6 can associate the specific action "cross out" with the top six parts having the largest amount of change per unit time. The first part to the sixth part are "the right end of the right eye", "the left end of the left eye", "the right end of the right eyebrow", "the left end of the left eyebrow", and "the left end of the right eyebrow", respectively. and "the right end of the left eyebrow."

Note that in FIG. 6, the correspondence information associates, for example, six parts with each specific motion. However, the correspondence information can also associate different numbers of parts with a plurality of specific actions.

Returning to FIG. 5, in ST504, the terminal device 20 (image generation unit 160) generates an animation of a virtual character using data (measurement data) regarding the performer's body, as described in relation to ST402. Videos can be generated. An example of the generated video is shown in FIG. 7A. The video 700 may include a virtual character (sometimes referred to as an "avatar") 702 of a performer. A virtual character 702 that may be included in the moving image 700 may be placed in a virtual space made of CG or the like, or may be placed in a real space. In ST504, the virtual character 702 can change to follow the performer's body movements. For example, in response to the performer facing the terminal device 20 blinking, the virtual character 702 included in the video 700 also blinks. Furthermore, the terminal device 20 (the display unit 170) can display the video 700 generated in this way to present it to the performer.

Returning to FIG. 5, also in ST504, (the communication unit 190 of) the terminal device 20 can transmit such a moving image 700 to the server device 30. Thereby, the terminal devices 20 (of the display units 170) of other viewing users connected to the server device 30 can also display this video 700 on their display units.

Next, in ST506, the terminal device 20 (change amount acquisition unit 120) can acquire the change amount of each of the plurality of parts for each unit time. Next, in ST508, the terminal device 20 (reference value acquisition unit 130) uses the amount of change per unit time of each part acquired in ST506 to obtain a reference value based on the amount of change per unit time of the plurality of parts. It is possible to obtain R. Here, in one embodiment, the "plurality of parts" from which the reference value R is obtained may be the entirety of a plurality of predetermined parts of the performer's body. In another embodiment, the "multiple parts" from which the reference value R is obtained is a part of a plurality of predetermined parts (for example, two or more representative parts) of the performer's body. It can be.

Next, in ST510, the terminal device 20 (for example, the reference value acquisition unit 130) can estimate that the performer did not perform any specific action if the reference value R acquired in ST508 is less than or equal to the threshold. can. As a result, the process returns to ST504 described above.

On the other hand, in ST510, the terminal device 20 (for example, the reference value acquisition unit 130) can estimate that the performer has performed any specific action if the reference value R acquired in ST508 exceeds the threshold. . As a result, the process moves to ST512.

In ST512, the terminal device 20 (identification unit 140) uses, for example, the amount of change per unit time of the plurality of parts obtained in ST506 described above, to identify the top M parts with the largest amount of change among the plurality of parts. The parts of the body can be identified. Here, M is a natural number and can be arbitrarily set within a range of 1 or more and N or less. By setting M large (or small), the accuracy of the detection operation determined in ST516, which will be described later, may increase (or decrease).

Furthermore, in one embodiment, the terminal device 20 (the identification unit 140 of the terminal device 20) can identify the top M parts with the largest amount of change from among the predetermined M parts. Thereby, the terminal device 20 (identification unit 140) only needs to compare the amount of change of each of the predetermined M portions when identifying the top M portions with the largest amount of change. Therefore, the terminal device 20 (the identification unit 140 of the terminal device 20) can more quickly identify the top M parts with the largest amount of change.

Furthermore, in one embodiment, (the identification unit 140 of) the terminal device 20 can rearrange (sort) the M parts identified in this way into an order according to the magnitude of the amount of change.

Next, in ST514, terminal device 20 (determining unit 150 of It is possible to determine whether there is a "specific action associated with". Here, the "specific motions associated with the M identified parts" can include, for example, at least one of the following specific motions (1) to (4).
(1) A specific operation in which M parts that are completely the same as the identified M parts are associated in correspondence information.
(2) A specific operation in which L parts that are completely the same as the top L parts with the largest amount of change among the M parts identified are matched in the correspondence information (however, L is 1 or more and is a natural number less than or equal to M).
(3) The specific motion shown in (1) above, further in an order according to the magnitude of change in at least one top region among the M regions associated with the specific motion. is the specific motion that matches the order according to the magnitude of change of at least one top region among the M identified regions. Note that the order of at least one part associated with this specific action (3) is said to completely match or partially (approximately) match the order of the identified M parts. be able to.
(4) The specific motion shown in (2) above, further in an order according to the magnitude of change in at least one top region among the L regions associated with the specific motion. is the specific motion that matches the order according to the magnitude of change of at least one top region among the M identified regions. Note that the order of at least one part associated with this specific action (4) is said to completely match or partially (approximately) match the order of the identified M parts. be able to.

Specific operations (1) to (4) will be explained using specific examples.
Consider a case where the top M parts identified in ST512 are part 1, part 3, and part 2 (M=3). Here, the amount of change in each part becomes smaller in this order. That is, part 1 is the part with the largest amount of change, and part 2 is the part with the smallest amount of change.

Focus is on specific action (1).
Among the specific actions included in the correspondence information stored in the storage unit 100, for example, a specific action in which the following parts are associated in the following order may correspond to the specific action (1).
・Part 1, Part 3, Part 2
・Part 1, Part 2, Part 3
・Part 2, Part 3, Part 1
・Part 2, Part 1, Part 3
・Part 3, Part 1, Part 2
・Part 3, Part 2, Part 1
・Part 1, Part 3, Part 2, Part 10
・Part 1, part 3, part 2, part 21, etc.

Let's focus on specific action (2).
Among the specific actions included in the correspondence information stored in the storage unit 100, for example, a specific action in which the following parts are associated in the following order may correspond to the specific action (2).
・Part 1, part 3, part 2 (L=3)
・Part 1, part 3, part 2, part 12 (L=3)
・Part 3, Part 2, Part 1, Part 18 (L=3)
・Part 3, Part 16, Part 1, Part 2 (L=3)
・Part 2, part 1, part 3 (L=3)
・Part 2, part 34, part 1, part 18, part 3 (L=3)
・Part 89, part 3, part 2 (L=3)
・Part 1, part 3, part 8 (L=2)
・Part 1, part 3, part 8, part 12 (L=2)
・Part 1, part 55, part 24, part 3 (L=2)
・Part 22, part 1, part 9, part 3 (L=2)
・Part 1, part 9, part 12 (L=1)
・Part 1, part 8, part 14, part 22 (L=1)
・Part 15, part 25, part 1 (L=1), etc.

Let's focus on specific action (3).
Among the specific actions included in the correspondence information stored in the storage unit 100, for example, a specific action in which the following parts are associated in the following order may correspond to the specific action (3).
・Part 1, Part 3, Part 2
・Part 1, Part 3, Part 2, Part 6
・Part 1, Part 3, Part 10, Part 2
・Part 1, Part 2, Part 3
・Part 1, Part 15, Part 11, Part 3, Part 2, etc.

Let's focus on specific action (4).
Among the specific actions included in the correspondence information stored in the storage unit 100, for example, a specific action in which the following parts are associated in the following order may correspond to the specific action (4).
・Part 1, part 3, part 2 (L=3)
・Part 1, part 3, part 2, part 12 (L=3)
・Part 1, part 3, part 8 (L=2)
・Part 1, part 3, part 8, part 12 (L=2)
・Part 1, part 55, part 24, part 3 (L=2)
・Part 22, part 1, part 9, part 3 (L=2)
・Part 1, part 9, part 12 (L=1)
・Part 1, part 8, part 14, part 22 (L=1)
・Part 15, part 25, part 1 (L=1), etc.

In ST514, if the terminal device 20 (determining unit 150) determines that there is no "specific action associated with the M identified parts" as described above, the process returns to ST504 described above. return. In this case, in another embodiment, the terminal device 20 can increase the number of M parts to (M+α) parts and sequentially execute ST512 and ST514 again. Here, α is a natural number of 1 or more.

On the other hand, in ST514, the terminal device 20 (the determining unit 150 of the If it is determined that there is a specified specific action, the process moves to ST516. In ST516, the terminal device 20 (determining section 150 of "motion" can be determined as a specific motion (detected motion) performed by the performer. Note that the terminal device 20 (determining unit 150 of the terminal device 20) determines such "specific actions associated with the M identified parts" or "specific actions associated with the (M+α) identified parts." ” is included in the correspondence information, the optimal specific motion among the plurality of specific motions can be determined as the detection motion. The optimal specific motion may be, for example, a specific motion in which the M parts identified in ST512 are most closely matched among the plurality of specific motions.

Next, in ST518, the terminal device 20 (image generation unit 160) can generate a video including an animation of a virtual character that moves based on the detected motion determined in ST516. If the detection action determined in ST516 is, for example, the "angry face" illustrated in FIG. It is possible to generate a video 700 in which the facial expression of 702 is changed to an angry facial expression. Furthermore, if the detection operation determined in ST516 is, for example, "cross-out" as illustrated in FIG. It is possible to generate a moving image 700 in which the facial expression of a character 702 is changed to a so-called cross-eyed expression (an expression in which both eyes are represented by symbols and the mouth is represented by a rectangle). Furthermore, the terminal device 20 (the display unit 170) can display the video 700 generated in this way to present it to the performer.

Similarly, in ST518, (the communication unit 190 of) the terminal device 20 can transmit the video 700 generated in this way to the server device 30. Thereby, the terminal devices 20 (of the display units 170) of other viewing users connected to the server device 30 can also display this video 700 on their display units.

After this, in one embodiment, the process can return to ST504 described above.
In another embodiment, the process may return to ST502 described above as illustrated in FIG. 5. In ST502, the terminal device 20 can update the correspondence information stored in the storage section 100.

For example, when the video 700 generated based on the detection motion determined in ST516 is displayed on (the display unit 170 of) the terminal device 20 in ST518, the performer or the like who monitors this video 700 may You may feel that the specific action (determined) is not appropriate. In this case, the performer or the like can specify the specific action originally intended by the performer or the like via (the user interface section 180 of) the terminal device 20.

When such a designation is made by a performer or the like, the terminal device 20 (the image generation unit 160 and the display unit 170) can stop generating the moving image 700 based on the detection operation. Further, the terminal device 20 can generate and display the moving image 700 based on the data (measured data) regarding the performer's body, as in ST504 described above. In this way, the moving image 700 that reflects the intentions of the performer or the like can be transmitted to the server device 30 or the like.

Furthermore, when such a designation is made by a performer or the like, the terminal device 20 can update the correspondence information stored in the storage unit 100 based on information regarding the designation.
For example, consider a case where the performer or the like specifies "angry face" as shown in FIG. 6 as an appropriate specific action even though the detected action is "crossed out" as shown in FIG. In this case, the terminal device 20 selects "right end of right eye", "left end of left eye", "right end of right eye", "left end of left eye", and The correspondence information illustrated in FIG. 6 can be updated so that "right end of right eyebrow", "left end of left eyebrow", "left end of right eyebrow", and "right end of left eyebrow" are associated. Alternatively, the terminal device 20 may select the top M parts themselves identified in ST512 or at least one of the top M parts identified in ST512 for the specific action of "angry face". It is also possible to update the correspondence information illustrated in FIG. 6 so that the parts of the parts are associated with each other.

Note that constantly or frequently performing the operations shown in ST512 to ST514 may increase the power consumption of the terminal device 20, drain the battery of the terminal device 20, and/or reduce the processing speed of the terminal device 20. There is a possibility that it will be connected. Therefore, in order to at least partially suppress such a possibility, in the example shown in FIG. The terminal device 20 executes the operations shown in ST512 to ST514 based on the estimation that there is a possibility that the terminal device 20 has received a message. In other words, in the example shown in FIG. 5, if the reference value R based on the amount of change in multiple parts is below the threshold value, there is a possibility that the performer has not performed any specific movement. Based on the estimation, the terminal device 20 avoids the operations shown in ST512 to ST514.

Note that in another embodiment, the terminal device 20 may always or frequently perform the operations shown in ST512 to ST514. In this case, the operations shown in ST508 and ST510 may be omitted.

Furthermore, in the example shown in FIG. 5, a case has been described in which the amount of change per unit time of the plurality of parts used in ST512 is the amount of change per unit time of the plurality of parts obtained in ST506. However, the amount of change per unit time of the plurality of parts used in ST512 may be the sum of the amounts of change of at least one of the amounts of change illustrated below.
・Amount of change per unit time of multiple body parts acquired in ST506 (hereinafter, for convenience, this unit time will be referred to as "reference unit time")
・Amount of change in multiple parts obtained in at least one unit time that occurred after the reference unit time ・Amount of change in multiple parts obtained in at least one unit time that occurred before the reference unit time

6. Method for updating correspondence information Next, a specific example of a method for updating correspondence information stored in the terminal device 20 and the like will be described.
The correspondence information stored by the terminal device 20 and the like can be updated, for example, at the following three timings.
(1) At the time of initial use (2) At the time of first use every day (3) When an outlier occurs

First, the above (1) "initial use" will be explained. When using the service provided by the communication system 1, the performer A can also use correspondence information prepared in advance by the communication system 1 (for example, correspondence information updated by another performer). However, when the terminal device 20 of performer A uses correspondence information updated by another performer, there is a possibility that performer A's unintended motion (facial expression) may be determined as a detected motion. Therefore, it is important for the terminal device 20 to customize correspondence information for each performer (user). The reason why customization is performed in this way is that even if performers try to make the same facial expressions, each performer has a different ranking based on the state of the face, the parts of the face that change the most, and the magnitude of the changes in these parts. This is because they are different.

For example, when performer A makes a "right wink" facial expression, performer A's top multiple (for example, 3) parts with the largest amount of change are arranged in descending order of the amount of change, such as (i) right eyebrow, (ii) ) upper eyelid of the right eye; (iii) lower eyelid of the right eye. However, when performer B makes the same "right wink" facial expression, performer B's top multiple (for example, 3) parts with the largest amount of change are ranked in descending order of the amount of change, such as (i) above the right eye; It can be the eyelid, (ii) the right eyebrow, and (iii) the lower eyelid of the right eye. In this case, if performer A's terminal device 20 uses the correspondence information updated by performer B's terminal device 20 as is, there is a high possibility that performer A's right wink cannot be determined as a detection motion even if performer A winks to the right. .

Therefore, when using the service provided by the communication system 1 for the first time, each performer uses the terminal device 20 to register his or her facial expressions for each specific action (such as a special facial expression) reflected in the avatar object. (This becomes the teacher information for the target). Specifically, for example, the terminal device 20 displays instructions regarding each major specific motion (such as a special facial expression), and the performer can perform the specific motion according to the instructions. Thereby, the terminal device 20 can generate (update) correspondence information that associates each specific motion with at least one region having a large change amount corresponding to the specific motion. This correspondence information can be used as described above in ST514 in FIG. Furthermore, the terminal device 20 can generate a learning model corresponding to the performer by using the facial expressions (question information) and specific actions (answer information) made by the performer in accordance with the above instructions as teacher information. can. For example, such a learning model can operate so as to output a specific action when a performer's facial expression is input.

Next, the above (2) "daily first use" will be explained.
Even if the same performer intends to perform the same specific action, the parts of the body (face, etc.) that have a large amount of change detected by the terminal device 20 and the magnitude of the amount of change in these parts The ranking based on the ranking may vary from day to day. This can be caused by various factors including the performer's hairstyle, fatigue, shooting environment, and the positional relationship between the camera and the body (face, etc.).
Therefore, each performer can use the terminal device 20 to register his or her facial expressions for each specific action (special facial expression, etc.) reflected in the avatar object when using the device for the first time each day (for example, every morning). (This becomes the teacher information for the target as in the case (1) above).

In one embodiment, the terminal device 20 can perform the same learning as that performed in (1) above. Furthermore, in a more preferred embodiment, the terminal device 20 not only learns by simply associating the facial expressions made by the performer with specific movements to be output (special facial expressions, etc.), but also learns the following elements from teacher information ( (question information and answer information) can be used for learning.
(i) Facial expressions made by the performer (question information)
(ii) Performer's hairstyle (question information)
(iii) Performer's fatigue level (question information)
Note that this degree of fatigue may be determined, for example, from the amount of change in the parts of the face (the parts to be noted do not change, but the maximum and minimum range becomes narrower. In other words, the greater the degree of fatigue (While the difference between the maximum and minimum changes in a region may become smaller, the smaller the degree of fatigue, the larger the difference between the maximum and minimum changes in a region of interest.)
(iv) Shooting environment, such as the positional relationship between the camera and the body (face, etc.) (question information)
(v) Situation and/or Purpose Depending on the situation or purpose, the performer may decide that ``I will not use this face this time'' and exclude the specific action from being judged. This also reveals facial expressions that the performer does not want to use due to the design of the character. Note that the situation and purpose may mean, for example, what kind of virtual space (karaoke, stage, live venue, etc.) it is.
(vi) Specific movement (answer information)

Therefore, as the performer continues to use the service, the determination of the performer as a target converges. That is, as the performer continues to use the service, the specific action intended by the performer becomes more likely to be reflected in the avatar object.

Note that the above learning (2) may be performed by the performer's operation on the terminal device 20 when an unintended specific action (such as a special facial expression) is performed by the avatar object. For example, when the terminal device 20 causes the avatar object to perform a special facial expression A that the performer did not intend, the performer may give priority to the avatar object by operating the terminal device 20 so that the avatar object activates the special facial expression B that the performer originally wanted to activate. The ranking may be clearly set. In this case, the terminal device 20 can store learning information for determining special expression B preferentially, even though special expression A and special expression B are determined to be close. This process is similar to the character conversion of the keyboard, which is customized as it continues to be used. Therefore, a detailed explanation of the process will be omitted here.) Note that the generation (update) of the correspondence information is not limited to being executed at the time of the first use every day, but for example, when it is determined that the interval between the usage times of the video distribution application on the terminal 20 is longer than a certain period of time. may be executed.

Next, the above (3) "when an outlier occurs" will be explained.
When the top three parts of the face with the largest amount of change are (i) upper lip, (ii) lower lip, and (iii) eyebrows in descending order of the amount of change, the terminal device 20 displays the expression "a". (Whether the performer pronounces ``aaa'' or ``aah'', the same facial expression ``a'' is reflected on the avatar object.) Here, when the performer pronounces "Are~", the top three parts with the largest amount of change are (i) upper lip, (iii) eyebrow,
(ii) lower lip, and it is determined that there is no special expression to be assigned.

In this case, if the performer inputs the default (normal expression) in response to a question from the terminal device 20 via the user interface, the facial expression determination ends. stand by). On the other hand, the performer can register a new special facial expression in response to a question from the terminal device 20 via the user interface. In this case, when the top three parts of the face with the largest amounts of change are (i) upper lip, (iii) eyebrows, and (ii) lower lip in descending order of the amount of change, the terminal device 20 determines the new special feature. It becomes possible to activate facial expressions (that...).

This determination can be made using a distance function (sum of squares of each element) between the input value and each evaluation function as a determination as an "outlier". For example, there is an evaluation function fA(xi) for facial expression A and an evaluation function fB(xi) for facial expression B (there is also a default facial expression fDefault(xi), etc.), and from which function fA(), fB(), Facial expressions that are sufficiently far away and are not excluded from the determination as fDefault() can be proposed to the performer by the terminal device 20 as new facial expressions. On the other hand, the performer can set a threshold value that prioritizes judgment on stable targets by operating a slide or the like displayed on the user interface, for example, using a numerical value in the range of 0 to 1. .
For example, consider a situation in which smile fA, smile (small) fB, and smile (medium) fC are registered as targets in the initial state. For example, when the terminal device 20 detects a phenomenon in which the amount of change in any part deviates significantly during video distribution, the terminal device 20 instructs the performer to newly "register a new facial expression state" via the user interface. I can make suggestions. This allows the performer to register "loud laughter". Thereafter, the terminal device 20 can add a new target and evaluation function fD() to the determination loop.

As the performer continues to use the terminal device 20 that implements the above (1) to (3), the activation of specific actions (such as special facial expressions) by the terminal device 20 will be customized to each individual user and will converge. It is possible.

The updating of the algorithm (evaluation function and data table) basically converges as the terminal device 20 recursively stores the evaluation function and data table (correspondence information) based on the performer's operations. A new determination function can be registered by the performer using the service. This can be called "autonomous learning" because the developer does not need to write a new algorithm, but it is based on the teacher information provided by the performer. These evaluation functions and data sets can be called learning results for each performer, are small in size, and do not include personal information, so service operators can collect them and use them for averaging and optimizing algorithms.

7. At least one specific action predetermined in the modified communication system 1 includes a first action performed by the performer, a second action performed by the performer after a longer period of time after this first action, may include specific operations identified by. For example, a specific action of clapping can be identified by a first action in which the palms of both hands are placed at a distance from each other, and a second action in which the palms of both hands touch each other after this first action. can be done.

In this case, in the correspondence information stored in the storage unit 100, the terminal device 20 identifies at least one region with the highest variation in the first motion and the second region with respect to the specific motion of “clap”. At least one region with a large change in motion can be associated with the region. Furthermore, when the reference value R exceeds the threshold (ST510), the terminal device 20 first selects the top can be identified (ST512 applied mutatis mutandis). Next, the terminal device 20 can determine whether there is a first action corresponding to the top X1 parts identified in this way in the correspondence information (ST514 is applied mutatis mutandis). Here, X1 is an arbitrary natural number. Furthermore, the terminal device 20 can identify the top X2 parts with the largest amount of change for the second unit time that occurs after the first unit time and may include one or more unit times ( ST512 applied mutatis mutandis). Here, X2 is also an arbitrary natural number. Next, the terminal device 20 can determine whether there is a second action corresponding to the top X2 parts identified in this way in the correspondence information (ST514 is applied mutatis mutandis). If there is a first action corresponding to the top X1 parts in the correspondence information and a second action corresponding to the top X2 parts, the terminal device 20 A specific motion (such as applause) including the motion and the second motion can be determined as the detected motion (ST516 is applied mutatis mutandis).

Instead of being a data table as shown in FIG. 6, the correspondence information may be, for example, an algorithm including a plurality of evaluation functions prepared for each specific motion to be determined. This algorithm can be treated as information that associates each specific motion with at least one top region with a large amount of displacement.

Furthermore, in the various embodiments described with reference to FIGS. 4, 5, etc., the technology disclosed in the present application is applied, as an example, to generation of a moving image. However, the technology disclosed in the present application is applicable to various applications including email, messengers, word processors, etc., and to various services such as websites and SNS. In this case, in ST518, the terminal device 20 can display a pictogram and/or emoticon corresponding to the determined detection operation.
Furthermore, the technology disclosed in this application is also applicable to game applications and game services. In this case, in ST518, the terminal device 20 can control the motion of the game object based on the determined detection motion.

Furthermore, in the various embodiments described with reference to FIGS. 4, 5, etc., the terminal device 20 facing the performer (instead, the server device 30 or the studio unit 40 may be used. Hereinafter, these will be collectively referred to as A case has been described in which the terminal device 20 (hereinafter referred to as "terminal device 20, etc.") performs all the processing from acquiring data (measurement data) regarding the performer's body to transmitting the moving image. However, although the terminal device 20 or the like facing the performer needs to acquire data regarding the performer's body, subsequent operations may be performed by a device other than the terminal device 20 or the like.

In one example, after the terminal device 20, etc. acquires data regarding the performer's body, the terminal device 20, etc. executes at least one step among ST506, ST508, ST510, ST512, ST514, and ST516, and performs steps other than this step. The steps may be performed by one or more other devices connected to terminal device 20 or the like. For example, the steps up to the acquisition of data regarding the performer's body may be executed by the terminal device 20, etc., and the steps ST506 to ST518 (up to generation of the video) may be executed by one or more other devices alone or in a shared manner. . Alternatively, the steps up to ST506 may be executed by the terminal device 20 or the like, and the steps ST508 to ST518 (up to the generation of the moving image) may be executed by one or more other devices alone or in a shared manner. This makes it possible to reduce the calculation resources, power consumption, etc. required for the terminal device 20 and the like facing the performer.
In either case, the terminal device 20 or the like transfers data/information obtained by performing an intermediate step and required for performing a subsequent step to one or more other devices. Need to send.

Note that the above-mentioned "one or more other devices" can include the following devices in each of the first to third aspects.
- Server device 30 and/or viewing user's terminal device 20, etc. (in the case of the first aspect and the third aspect)
- Other server devices 30 and/or viewing user's terminal devices 20, etc. (in the case of the second aspect)
Here, the above-mentioned "one or more other devices" includes the viewing user's terminal device 20, and the method by which the viewing user's terminal device 20 executes ST518 (up to video generation) is the "client rendering" method. It is sometimes called.

Furthermore, in the embodiments described with reference to FIGS. 4, 5, etc., the correspondence information is associated with each of at least one specific movement and at least one of the parts of the performer's body having the largest amount of change. , but also the case where each of at least one specific motion is associated with an order based on the magnitude of the amount of change of at least one region having a large amount of change has been described. However, in another embodiment, the correspondence information associates each of the at least one specific motion with at least one region having the largest amount of change among the plurality of regions on the performer's body, while at least one of the at least one specific motion It is also possible not to associate each of them with the order based on the magnitude of the amount of change of at least one region with a large amount of change.

In this case, the terminal device 20 does not need to execute ST512. In ST514, the terminal device 20 determines that "specific actions associated with the M identified body parts" ( Alternatively, it can be determined whether a "specific motion associated with (M+α) identified parts" exists. Here, the "specific motion associated with the M identified parts" can include, for example, at least one of the above-described specific motion (1) and specific motion (2).

Further, the various embodiments described above can be used in combination with each other unless a contradiction occurs.

As described above, in various embodiments, after identifying at least one region with the largest amount of change per unit time based on data regarding the performer's (user's) body, It is possible to detect a specific motion that is associated in advance as a specific motion performed by the performer. As a result, it is possible to easily and quickly detect a specific action performed by a performer, thereby reducing computational resources, power consumption, and the like.

Furthermore, when a reference value based on the amount of change per unit time of a plurality of parts of the performer exceeds a threshold value, at least one part having the largest amount of change per unit time is identified; It is possible to detect a specific motion that is previously associated with one body part. Thereby, calculation resources, power consumption, etc. can be further reduced.

Therefore, according to various embodiments, images based on a performer's actions can be displayed in a new manner.

という数式を用いて算出され、ここで、ｘ_ｉは、前記演者の身体における複数の部位のうちの第ｉ番目の部位の単位時間当たりの変化量である」構成を採用することができる。
第５の態様に係るコンピュータプログラムにあっては、上記第１の態様から上記第４の態様のいずれかにおいて「前記少なくとも１つの特定動作の各々は、該特定動作に対応付けられた前記変化量が大きい上位少なくとも１つの部位を用いて表現される」構成を採用することができる。
第６の態様に係るコンピュータプログラムにあっては、上記第５の態様において「前記変化量が大きい上位少なくとも１つの部位の各々は、目、眉毛、鼻、口、耳、顎、頬、首、肩、手及び胸を含む群から選択される」構成を採用することができる。
第７の態様に係るコンピュータプログラムにあっては、上記第１の態様から上記第の態様のいずれかにおいて「決定された前記検出動作に基づいて画像を生成する、ように前記プロセッサを機能させる」構成を採用することができる。
第８の態様に係るコンピュータプログラムにあっては、上記第７の態様において「前記画像が、絵文字、アバターオブジェクト、及び／又は、ゲームオブジェクトを含む」構成を採用することができる。
第９の態様に係るコンピュータプログラムは、「少なくとも１つのプロセッサにより実行されることにより、少なくとも１つの特定動作の各々と、演者の身体における複数の部位のうち変化量が大きい上位少なくとも１つの部位と、を対応付けた情報を記憶し、演者の端末装置から受信した該演者の身体に関する測定データを用いて、単位時間当たりの該演者の身体における複数の部位の変化量に基づく参照値を取得し、前記参照値が閾値を上回る事象を検出した場合に、前記測定データを用いて前記演者の身体における単位時間当たりの変化量が大きい上位少なくとも１つの部位を識別し、前記情報を用いて、前記少なくとも１つの特定動作のうち、識別された前記単位時間当たりの変化量が大きい上位少なくとも１つの部位に対応付けられたいずれか１つの特定動作を、検出動作として決定する、ように前記プロセッサを機能させる」ことができる。
第１０の態様に係るコンピュータプログラムにあっては、上記第９の態様において「前記測定データを前記演者の端末装置から通信回線を介して受信する、ように前記プロセッサを機能させる」構成を採用することができる。
第１１の態様に係るコンピュータプログラムにあっては、上記第１の態様から上記第１０の態様のいずれかにおいて「前記少なくとも１つのプロセッサが、中央処理装置（ＣＰＵ）、マイクロプロセッサ、及び／又は、グラフィックスプロセッシングユニット（ＧＰＵ）を含む」構成を採用することができる。
第１２の態様に係るコンピュータプログラムにあっては、上記第１の態様から上記第１１の態様のいずれかにおいて「前記少なくとも１つのプロセッサが、スマートフォン、タブレット、携帯電話又はパーソナルコンピュータに搭載される」構成を採用することができる。
第１３の態様に係る方法は、「コンピュータにより読み取り可能な命令を実行する少なくとも１つのプロセッサにより実行される方法であって、該プロセッサが前記命令を実行することにより、少なくとも１つの特定動作の各々と、演者の身体における複数の部位のうち変化量が大きい上位少なくとも１つの部位と、を対応付けた情報を記憶し、演者の身体に関する測定データを用いて、単位時間当たりの該演者の身体における複数の部位の変化量に基づく参照値を取得し、前記参照値が閾値を上回る事象を検出した場合に、前記測定データを用いて前記演者の身体における単位時間当たりの変化量が大きい上位少なくとも１つの部位を識別し、前記情報を用いて、前記少なくとも１つの特定動作のうち、識別された前記単位時間当たりの変化量が大きい上位少なくとも１つの部位に対応付けられたいずれか１つの特定動作を、検出動作として決定する」ことができる。
第１４の態様に係る方法にあっては、上記第１３の態様において「前記少なくとも１つのプロセッサが、中央処理装置（ＣＰＵ）、マイクロプロセッサ、及び／又は、グラフィックスプロセッシングユニット（ＧＰＵ）を含む」構成を採用することができる。
第１５の態様に係る方法にあっては、上記第１３の態様又は上記第１４の態様において「前記少なくとも１つのプロセッサが、スマートフォン、タブレット、携帯電話、パーソナルコンピュータ、又は、サーバ装置に搭載される」構成を採用することができる。
第１６の態様に係るサーバ装置は、「少なくとも１つのプロセッサを具備し、該プロセッサが、コンピュータにより読み取り可能な命令を実行することにより、少なくとも１つの特定動作の各々と、演者の身体における複数の部位のうち変化量が大きい上位少なくとも１つの部位と、を対応付けた情報を記憶し、演者の身体に関する測定データを用いて、単位時間当たりの該演者の身体における複数の部位の変化量に基づく参照値を取得し、前記参照値が閾値を上回る事象を検出した場合に、前記測定データを用いて前記演者の身体における単位時間当たりの変化量が大きい上位少なくとも１つの部位を識別し、前記情報
を用いて、前記少なくとも１つの特定動作のうち、識別された前記単位時間当たりの変化量が大きい上位少なくとも１つの部位に対応付けられたいずれか１つの特定動作を、検出動作として決定する」ことができる。
第１７の態様に係るサーバ装置にあっては、上記第１６の態様において「前記測定データを前記演者の端末装置から通信回線を介して受信する」構成を採用することができる。
第１８の態様に係るサーバ装置にあっては、上記第１６の態様又は上記第１７の態様において「前記少なくとも１つのプロセッサが、中央処理装置（ＣＰＵ）、マイクロプロセッサ、及び／又は、グラフィックスプロセッシングユニット（ＧＰＵ）を含む」構成を採用することができる。 8. Supplementary Note The computer program according to the first aspect is ``executed by at least one processor to perform each of at least one specific motion and at least one area having the largest amount of change among a plurality of areas in the performer's body. and a reference value based on the amount of change in a plurality of parts of the performer's body per unit time is obtained using measured data regarding the performer's body, and the reference value exceeds the threshold value. If an event exceeding the above is detected, the measurement data is used to identify at least one region of the performer's body having a large amount of change per unit time, and the information is used to identify the top at least one region of the performer's body that exhibits a large amount of change per unit time. , the processor can be operated to determine, as the detected operation, any one specific operation associated with at least one of the identified regions having a large amount of change per unit time.
In the computer program according to the second aspect, in the first aspect, "the information includes each of the at least one specific motion and at least one of the top parts of the performer's body having the largest amount of change". and an order based on the magnitude of the amount of change of at least one region with the largest amount of change, and using the information, the identified part of the at least one specific motion is The processor is made to function so as to determine, as a detection operation, at least one region with a large amount of change per unit time, and any one specific operation that matches the order based on the magnitude of the amount of change. configuration can be adopted.
In the computer program according to the third aspect, in the first aspect or the second aspect, "the measurement data is used to identify at least one top region of the performer's body that has a large amount of change per unit time. and determining, as a first detected motion, any one of the at least one specific motion that is associated with the at least one region with the highest identified change amount per unit time. Then, among the at least one specific motion, any one of the identified motions associated with the at least one region with the largest change per unit time is determined as a second detected motion. , the processor is operated to determine a final detection operation based on the first detection operation and the second detection operation.
In the computer program according to the fourth aspect, in any one of the first aspect to the third aspect, "If the plurality of parts are N parts, the reference value is

where x _i is the amount of change per unit time of the i-th part of the plurality of parts in the performer's body.
In the computer program according to the fifth aspect, in any one of the first to fourth aspects, "Each of the at least one specific action is the amount of change associated with the specific action." It is possible to adopt a configuration in which the expression is expressed using at least one region with a large value.
In the computer program according to the sixth aspect, in the fifth aspect, "Each of the at least one region having a large amount of change is eyes, eyebrows, nose, mouth, ears, chin, cheeks, neck, "selected from the group including shoulders, hands, and chest" configurations may be employed.
In a computer program according to a seventh aspect, in any one of the first to third aspects, "the processor is caused to function to generate an image based on the determined detection operation". configuration can be adopted.
In the computer program according to the eighth aspect, it is possible to employ the configuration in the seventh aspect that "the image includes a pictogram, an avatar object, and/or a game object".
The computer program according to the ninth aspect is characterized in that the computer program is executed by at least one processor to perform each of at least one specific motion and at least one region having a large amount of change among the plurality of regions in the performer's body. , and obtains a reference value based on the amount of change in a plurality of parts of the performer's body per unit time using measurement data regarding the performer's body received from the performer's terminal device. , when an event in which the reference value exceeds a threshold is detected, using the measurement data to identify at least one top region of the performer's body with a large amount of change per unit time, and using the information to identify the The processor is configured to determine, as a detected motion, one of the at least one specific motions that is associated with at least one identified region having a large change per unit time. It is possible to “make someone do something”.
A computer program according to a tenth aspect employs a configuration in which the processor is made to function to receive the measurement data from the performer's terminal device via a communication line in the ninth aspect. be able to.
In the computer program according to the eleventh aspect, in any one of the first to tenth aspects, "the at least one processor is a central processing unit (CPU), a microprocessor, and/or A configuration including a graphics processing unit (GPU) can be adopted.
In the computer program according to the twelfth aspect, in any of the first to eleventh aspects, "the at least one processor is installed in a smartphone, a tablet, a mobile phone, or a personal computer." configuration can be adopted.
The method according to the thirteenth aspect is "a method performed by at least one processor that executes computer-readable instructions, wherein the processor executes each of the at least one specific operation. and at least one region with the largest change amount among the plurality of regions in the performer's body, and using measurement data regarding the performer's body, the performer's body per unit time is stored. When a reference value based on the amount of change in a plurality of body parts is obtained and an event in which the reference value exceeds a threshold is detected, at least one of the performers' body having the largest amount of change per unit time is determined using the measurement data. and using the information, select one of the at least one specific motion that is associated with at least one of the identified regions having the largest amount of change per unit time. , can be determined as a detection operation.
In the method according to the fourteenth aspect, in the thirteenth aspect, "the at least one processor includes a central processing unit (CPU), a microprocessor, and/or a graphics processing unit (GPU)." configuration can be adopted.
In the method according to the fifteenth aspect, in the thirteenth aspect or the fourteenth aspect, "the at least one processor is installed in a smartphone, a tablet, a mobile phone, a personal computer, or a server device. ” configuration can be adopted.
The server device according to the sixteenth aspect is provided with the following: “The server device includes at least one processor, and by executing computer-readable instructions, the server device performs each of the at least one specific operation and the plurality of operations in the performer's body. Among the parts, at least one part with the largest amount of change is stored, and using measurement data regarding the performer's body, it is based on the amount of change in the plurality of parts of the performer's body per unit time. When a reference value is obtained and an event in which the reference value exceeds a threshold is detected, the measurement data is used to identify at least one region of the performer's body having a large amount of change per unit time, and the information is of the at least one specific motion, which is associated with the at least one region with the largest identified change per unit time, as the detected motion." I can do it.
In the server device according to the seventeenth aspect, it is possible to adopt a configuration in which the measurement data is received from the performer's terminal device via a communication line in the sixteenth aspect.
In the server device according to the eighteenth aspect, in the sixteenth aspect or the seventeenth aspect, “the at least one processor is a central processing unit (CPU), a microprocessor, and/or a graphics processing unit (GPU)" can be adopted.

20 (20A to 20C) Terminal device 30 (30A to 30C) Server device 40 (40A and 40B) Studio unit 100 (200) Storage section 110 (210) Sensor section 120 (220) Change amount acquisition section 130 (230) Reference value Acquisition unit 140 (240) Identification unit 150 (250) Determination unit 160 (260) Image generation unit 170 (270) Display unit 180 (280) User interface unit 190 (290) Communication unit

Claims (1)

being executed by at least one processor;
retaining information associating each of at least one specific movement with at least one region having the largest amount of change among the plurality of regions in the performer's body;
Using measurement data regarding the performer's body, obtain a reference value based on the amount of change in a plurality of parts of the performer's body per unit time,
When detecting an event in which the reference value exceeds a threshold,
using the measurement data to identify at least one region of the performer's body with a large amount of change per unit time;
Using the information, any one of the at least one specific motion that is associated with the at least one identified region having the largest amount of change per unit time is determined as a detected motion. ,
A computer program for causing the processor to function as described above.

Images