TWI747258B - Method for generating action according to audio signal and electronic device - Google Patents

Abstract

The disclosure provides a method for generating an action according to an audio signal and an electronic device. The method includes: receiving an audio signal and extracting a high-level audio features therefrom; extracting a latent audio features from the high-level audio feature; in response to determining that the audio signal corresponds to a beat, obtaining a joint angle distribution matrix based on the latent audio features; in response to determining that the audio signal corresponds to music, obtaining a plurality of designated joint angles corresponding to the joint points based on the joint angle distribution matrix; and adjusting the joint angles of each joint point on the avatar according to the aforementioned designated joint angles.

Description

Method and electronic device for generating action based on audio signal

The present invention relates to a technology for controlling an avatar, and particularly relates to a method and an electronic device for generating actions based on audio signals.

In virtual reality (VR) and augmented reality (AR) experiences, avatars are a key part of these applications. If the avatar can have the same perception capabilities and feelings as the user, and can respond to the environment accordingly, this will greatly improve the user's sense of immersion.

In the prior art, there is a technology that allows virtual avatars to dance according to music. However, in order to achieve the above purpose, this technology needs to maintain a database storing a large number of preset dance steps for generating dance steps, which will consume more memory, so it is not easy to implement in edge devices (edge devices) ( Such as embedded systems or mobile devices).

Furthermore, when music appears in the VR/AR environment, the above technology will select one or more dance steps from the database based on certain predetermined hand-crafted features, and reorganize these dance steps into corresponding ones A series of dance steps in current music. Therefore, the above-mentioned technology cannot make the virtual avatar dance creatively.

In view of this, the present invention provides a method and an electronic device for generating actions based on audio signals, which can be used to solve the above technical problems.

The present invention provides a method for generating actions based on audio signals, including: receiving a first audio signal, and extracting a first high-level audio feature from the first audio signal; and extracting a first potential audio signal from the first high-level audio feature Features; in response to determining that the first potential audio feature indicates that the first audio signal corresponds to a first beat, a first joint angle distribution matrix is obtained according to the first potential audio feature, wherein the first joint angle distribution matrix includes a plurality of Gaussian distribution parameters , And the aforementioned Gaussian distribution parameter corresponds to a plurality of joint points on a virtual body; in response to determining that the first potential audio feature indicates that the first audio signal corresponds to a first piece of music, based on the first joint angle distribution matrix, it is obtained corresponding to the aforementioned joint Multiple designated joint angles of a point; adjust the joint angle of each joint point on the virtualized body according to the aforementioned designated joint angle.

The invention provides an electronic device, which includes a storage circuit and a processor. The storage circuit stores multiple modules. The processor is coupled to the storage circuit and accesses the aforementioned module to perform the following steps: receiving a first audio signal, and extracting a first high-level audio feature from the first audio signal; and extracting a first high-level audio feature from the first audio signal A first potential audio feature; in response to determining that the first potential audio feature indicates that the first audio signal corresponds to a first beat, a first joint angle distribution matrix is obtained according to the first potential audio feature, wherein the first joint angle distribution matrix includes multiple Gaussian distribution parameters, and the aforementioned Gaussian distribution parameters correspond to multiple joint points on a virtualized body; in response to determining that the first potential audio feature indicates that the first audio signal corresponds to a first music, it is obtained based on the first joint angle distribution matrix A plurality of designated joint angles corresponding to the aforementioned joint points; the joint angle of each joint point on the virtualized body is adjusted according to the aforementioned designated joint angle.

Based on the above, without maintaining the dance step database, the method of the present invention allows the virtual avatar to improvise corresponding actions (such as dance steps) with the current music, and is therefore suitable for application to electronic devices implemented as edge devices.

Please refer to FIG. 1, which is a schematic diagram of an electronic device according to an embodiment of the present invention. In different embodiments, the electronic device 100 is, for example, a computer device, an embedded system, a mobile device, etc., which can be used to provide AR/VR or other similar services, but it may not be limited thereto. As shown in FIG. 1, the electronic device 100 includes a storage circuit 102 and a processor 104.

The storage circuit 102 is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard disk Disk or other similar devices or a combination of these devices can be used to record multiple codes or modules.

The processor 104 is coupled to the storage circuit 102, and can be a general purpose processor, a special purpose processor, a traditional processor, a digital signal processor, multiple microprocessors, one or more combined digital signal processing The core microprocessor, controller, microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), any other type of integrated circuit , State machines, processors based on Advanced RISC Machine (ARM) and similar products.

In the embodiment of the present invention, the processor 104 can access the modules and program codes recorded in the storage circuit 102 to implement the method for generating actions based on audio signals proposed by the present invention. Details of the method are as follows.

Please refer to FIG. 2, which is a flowchart of a method for generating actions based on audio signals according to an embodiment of the present invention. The method of this embodiment can be executed by the electronic device 100 in FIG. 1. The details of each step in FIG. 2 will be described below with the components shown in FIG. 1. In addition, in order to make the content of this case easier to understand, the system architecture diagram shown in FIG. 3 will be supplemented below for description, but it is only used as an example and is not intended to limit the possible implementation of the present invention.

Generally speaking, the method of the present invention can determine the joint angles of each joint of the virtualized body in various dimensions when receiving a segment of audio signal (for example, an audio frame), so that the virtualized avatar can present appropriate actions as a whole. In different embodiments, the above-mentioned audio signal may come from any kind of sound, such as music, ambient sound, voice, etc., but it is not limited thereto.

In FIG. 3, the audio signal F1~FN is, for example, a plurality of consecutive audio frames, and for each audio signal F1~FN, the processor 104 can perform similar processing on it to generate an audio signal corresponding to the considered audio signal. The virtual avatar action of the signal. For ease of description, the audio signal F1 is temporarily used as an example for description below, but it is not intended to limit the possible implementations of the present invention.

First, in step S210, the processor 104 may receive the audio signal F1, and extract a high level audio feature H1 from the audio signal F1. In an embodiment, the audio signal F1 may include an audio frame, and it may be characterized as a vector (or array) with a specific dimension (for example, 2048×1), but it may not be limited thereto. In one embodiment, the processor 104 may input the aforementioned audio frame into a convolutional neural network (CNN) N1, so that the CNN N1 extracts high-level audio features H1 from the audio frame. In the embodiment of the present invention, the CNN N1 may include one or more convolutional layers for extracting corresponding high-level audio features from the received audio frame, but it is not limited to this. The technical details of the above-mentioned extraction of the high-level audio feature H1 by the CNN N1 can be referred to related prior art documents, which will not be described in detail here.

Afterwards, in step S220, the processor 104 may extract latent audio features L1 from the high-level audio features H1. In an embodiment, the processor 104 may input the high-level audio feature H1 into a first recurrent neural network (RNN) N2, so that the first RNN N2 extracts the potential audio feature L1 from the high-level audio feature H1. The above technical details of the potential audio feature L1 extracted by the first RNN N2 can be referred to related prior art documents, which will not be repeated here.

In addition, in this embodiment, the first RNN N2 can output a first internal state IS11 in addition to the potential audio feature L1 based on the high-level audio feature H1. For details, please refer to the RNN correlation Technical documents, not to repeat them here. In the embodiment of the present invention, the first RNN N2 may include a multi-stack structure for extracting corresponding potential audio features from the received high-level audio features, but it is not limited to this.

In addition, in one embodiment, the first internal state IS11 allows the first RNN N2 to further refer to the high-level audio feature H1 of the previous level to generate the corresponding potential audio when processing the high-level audio feature H2 corresponding to the next audio signal F2. Feature L2, and related details will be explained later.

In one embodiment, the processor 104 may determine whether the audio signal F1 corresponds to a beat (ie whether it is on beat) based on the latent audio feature L1, and determine whether the audio signal F1 corresponds to music based on the latent audio feature L1. In the embodiment of the present invention, the processor 104 can input the latent audio feature L1 into a specific neural network N3 (for example, it is composed of multiple fully-connected layers), so that the specific neural network N3 is based on The latent audio feature L1 determines whether the audio signal F1 corresponds to a beat and whether it corresponds to music, but it is not limited to this.

For ease of description, it is assumed below that the audio signal F1 corresponds to the beat and corresponds to the music (that is, it is not noise, human voice, or other non-musical sounds). Therefore, in step S230, in response to determining that the potential audio feature L1 indicates that the audio signal F1 corresponds to the beat, the processor 104 can obtain the joint angle distribution matrix M1 according to the potential audio feature L1, where the joint angle distribution matrix M1 may include multiple Gaussian distributions Parameters, and the aforementioned Gaussian distribution parameters can correspond to multiple joint points on a virtualized body. In an embodiment, the processor 104 may input the latent audio feature L1 into the second RNN N4, so that the second RNN N4 generates the joint angle distribution matrix M1 based on the latent audio feature L1. In addition, the second RNN N4 can also generate a second internal state IS12 based on the potential audio feature L1.

In one embodiment, the aforementioned virtual avatar is, for example, a character configured to dance according to music in an AR/VR environment. In addition, according to the relevant specifications of the biovision hierarchy (BVH), a virtual body can be defined with an absolute position of the hip joint point (which can be represented by x, y, and z) and 52 other joint points, and the 52 Each of the other joint points can be represented by a group of joint rotation angles in a three-degree space, for example (Rx, Ry, Rz). For example, for a first joint point on a virtualized body, the corresponding Rx, Ry, and Rz are respectively in the first dimension (such as the X axis), the second dimension (such as the Y axis), and the third dimension ( For example, the joint angle on the Z axis), but it is not limited to this.

To facilitate the description of the concept of the present invention, the joint points on the virtualized body considered in the following hypothesis may include the aforementioned hip joint points and 52 other joint points, but the present invention may not be limited to this. In addition, the following also assumes that the action of the considered avatar can be defined based on the relevant specifications of BVH, but the present invention may not be limited to this. In this case, the action of the avatar can be determined based on the BVH motion capture data file. In one embodiment, a BVH dynamic capture data file may include 159 values, which respectively correspond to the absolute positions of the aforementioned hip joint points (ie x, y, z) and the respective 52 other joint points (Rx, Ry). , Rz). Therefore, after obtaining the BVH motion capture data file, the actions of the virtual avatar can be determined accordingly, and the present invention can determine the 159 values in the BVH motion capture data file based on the generated joint angle distribution matrix M1, and then determine the virtual avatar Actions.

表示）及標準差（以

表示）。舉另一例而言，假設第一關節點在第二維度上還具有另一可動角度範圍（可理解為對應於第一關節點的Ry的可動角度範圍），而此另一可動角度範圍在本發明中可模型化為一另一高斯分布模型。在此情況下，關節角度分布矩陣M1中對應於第一關節點的Ry的列可包括2個元素，而此2元素可分別是所述另一高斯分布模型的期望值及標準差。 Specifically, in the first embodiment, the joint angle distribution matrix M1 can be implemented as a matrix with a dimension of 159x2, and the 159 columns in it correspond to the aforementioned x, y, z and the 52 other joints. Click the individual (Rx, Ry, Rz). For example, suppose that a certain joint point on the virtualized body (hereinafter referred to as the first joint point) has a first movable angle range in the first dimension (it can be understood as the movable angle range of Rx corresponding to the first joint point), The first movable angle range can be modeled as a first Gaussian distribution model in the present invention. In this case, the column of Rx corresponding to the first joint point in the joint angle distribution matrix M1 may include two elements, and these two elements may be the expected values of the first Gaussian distribution model (in terms of

Expressed) and standard deviation (in

Express). For another example, suppose that the first joint point has another movable angle range in the second dimension (which can be understood as the movable angle range corresponding to Ry of the first joint point), and this other movable angle range is in this The invention can be modeled as another Gaussian distribution model. In this case, the column of Ry corresponding to the first joint point in the joint angle distribution matrix M1 may include two elements, and the two elements may be the expected value and the standard deviation of the other Gaussian distribution model, respectively.

Based on the above teachings, those with ordinary knowledge in the field should be able to understand the meaning and content of the remaining columns in the joint angle distribution matrix M1, which will not be repeated here. In addition, in the first embodiment, the first row of the joint angle distribution matrix M1 may be composed of expected values in each column, for example, and the second row of the joint angle distribution matrix M1 may be composed of, for example, the standard deviations in each column, but it may not be limited to this.

After obtaining the joint angle distribution matrix M1, in step S240, in response to determining that the potential audio feature L1 indicates that the audio signal F1 corresponds to music, the processor 104 may obtain a plurality of designated joint angles corresponding to the joint points based on the joint angle distribution matrix M1 .

Taking the first joint point as an example again, assuming that the processor 104 wants to obtain the first specified joint angle of the first joint point in the first dimension, the processor 104 may be in the first movable angle range based on the first Gaussian distribution model. The first angle is internally sampled to be the first designated joint angle of the first joint point in the first dimension. For ease of understanding, the description will be supplemented with FIG. 4 below.

）的第一角度作為上述第一指定關節角度，但可不限於此。 Please refer to FIG. 4, which is a first Gaussian distribution model for modeling the first movable angle range according to the first embodiment of the present invention. In FIG. 4, it is assumed that the first joint point has the first movable angle range R1 in the first dimension, and the first Gaussian distribution model G1 can be used to model the first movable angle range R1, for example. In this case, the processor 104 may sample the first angle in the first movable angle range R1 based on the first Gaussian distribution model G1 as the first designated joint angle of the first joint point in the first dimension. In an embodiment, the processor 104 may randomly sample a first angle within the first movable angle range R1 based on the first Gaussian distribution model G1 as the first designated joint angle. In another embodiment, the processor 104 can also directly sample R1 within the first movable angle range corresponding to the expected value (ie,

The first angle of) is taken as the above-mentioned first specified joint angle, but it may not be limited to this.

In the same way, assuming that the processor 104 wants to obtain the specified joint angle of the first joint point in the second dimension, the processor 104 may sample a (randomly) within the other movable angle range based on the another Gaussian distribution model. The angle is taken as another specified joint angle of the first joint point in the second dimension. Based on the above teachings, those with ordinary knowledge in the art should be able to understand the way the processor 103 obtains the designated joint angles of each joint point in each dimension, which will not be repeated here.

After obtaining multiple designated joint angles corresponding to each joint point, in step S250, the processor 104 may adjust the joint angle of each joint point on the virtualized body according to the designated joint angle. In the first embodiment, the processor 104 may output the designated joint angle corresponding to each joint point in the form of a designated joint angle vector S1 (the dimension of which is, for example, 159×1). For example, assuming that the processor 104 samples the angle corresponding to the expected value for each joint point as the designated joint angle of each joint point, the processor 104 can directly use the first row of the joint angle distribution matrix M1 as the designated joint angle Vector S1, but the present invention may not be limited to this.

In this case, the processor 104 may, for example, generate a corresponding BVH motion capture data file based on the specified joint angle in the specified joint angle vector S1, and adjust the joint angle of each joint point of the virtualized body based on the BVH motion capture data file. For example, the processor 104 may adjust the joint angle of the first joint point in the first dimension to correspond to the aforementioned first specified joint angle (for example, the expected value of the first Gaussian distribution model G1). In addition, the processor 104 may also adjust the joint angle of the first joint point in the second dimension to correspond to the above-mentioned another specified joint angle (for example, the expected value of the above-mentioned another Gaussian distribution model). Based on this, the processor 104 can adjust the joint angles of each joint point of the virtual body in different dimensions according to the content of the BVH dynamic capture data file, so that the virtual avatar can present a specific movement (such as dance steps).

It can be seen from the above that, different from the conventional method of selecting existing dance steps from the database for reorganization, the method of the present invention can determine the joint angle of each joint point of the virtual body in each dimension according to the current audio signal, thereby allowing the virtual The avatar can improvise and dance to the beat based on the current music.

In other embodiments, a single joint point may have more than two movable angle ranges in a single dimension, and these movable angle ranges may be modeled as a multivariate mixed Gaussian model. The second embodiment will be used for further explanation below.

表示）、第一標準差（以

表示）、第二期望值（以

表示）及第二標準差（以

表示）。 In the second embodiment, it is assumed that a single joint point has two movable angle ranges in a single dimension, but it is not limited to this. In this case, the joint angle distribution matrix M1 can be realized as a matrix with a dimension of 159x4, and the 159 columns in it correspond to the aforementioned x, y, z and the 52 other joint points individually (Rx, Ry, Rz). Taking the first joint point as an example again, it is assumed that the first joint point has the first and second movable angle ranges in the first dimension (which can be understood as the movable angle range of Rx corresponding to the first joint point), and the first joint point 1. The second movable angle range can be modeled as a first multivariate mixture Gaussian distribution model in the present invention. In this case, the column of Rx corresponding to the first joint point in the joint angle distribution matrix M1 may include 4 elements, and these 4 elements may be the first expected values of the first multivariate mixed Gaussian distribution model (in terms of

Expressed), the first standard deviation (in

Expressed), the second expected value (in

Expressed) and the second standard deviation (in

Express).

Based on the above teachings, those with ordinary knowledge in the art should be able to understand the meaning and content of the remaining columns in the joint angle distribution matrix M1 in the second embodiment, which will not be repeated here. In addition, in the second embodiment, the first row of the joint angle distribution matrix M1 may be composed of, for example, the first expected value in each column, and the second row of the joint angle distribution matrix M1 may be composed of, for example, the first standard deviation in each column. The third row of the joint angle distribution matrix M1 may be composed of, for example, the second expected value in each column, and the fourth row of the joint angle distribution matrix M1 may be composed of, for example, the second standard deviation in each column, but it may not be limited thereto.

After obtaining the joint angle distribution matrix M1, in step S240, in response to determining that the potential audio feature L1 indicates that the audio signal F1 corresponds to music, the processor 104 may obtain a plurality of designated joint angles corresponding to the joint points based on the joint angle distribution matrix M1 .

Taking the first joint point as an example again, assuming that the processor 104 wants to obtain the first specified joint angle of the first joint point in the first dimension, the processor 104 may use the first multivariate mixture Gaussian distribution model in the first The first angle is sampled in the movable angle range or the second movable angle range as the first designated joint angle of the first joint point in the first dimension. For ease of understanding, the following will be supplemented with FIG. 5 for explanation.

及

）及第二可動角度範圍R2（其對應於

及

）。在此情況下，處理器104可基於第一多變量混合高斯分布模型G1’在第一可動角度範圍R11內或第二可動角度範圍R12內取樣第一角度以作為第一關節點在第一維度上的第一指定關節角度。在一實施例中，處理器104例如可基於第一高斯分布模型G1’在第一可動角度範圍內R11或第二可動角度範圍R12內隨機取樣第一角度作為上述第一指定關節角度。在另一實施例中，處理器104亦可直接在第一可動角度範圍內R11或第二可動角度範圍R12內中取樣對應於期望值（即，

或

）的第一角度作為上述第一指定關節角度，但可不限於此。 Please refer to FIG. 5, which is a first multivariate mixture Gaussian distribution model for modeling the first and second movable angle ranges according to the second embodiment of the present invention. In FIG. 5, it is assumed that the first joint point has a first movable angle range R11 and a second movable angle range R12 in the first dimension, and the first multivariate mixed Gaussian distribution model G1' can be used to model the first movable angle, for example. Range R11 (which corresponds to

and

) And the second movable angle range R2 (which corresponds to

and

). In this case, the processor 104 may sample the first angle in the first movable angle range R11 or the second movable angle range R12 based on the first multivariate mixture Gaussian distribution model G1' to serve as the first joint point in the first dimension. On the first specified joint angle. In an embodiment, the processor 104 may randomly sample a first angle in the first movable angle range R11 or the second movable angle range R12 based on the first Gaussian distribution model G1′ as the first specified joint angle. In another embodiment, the processor 104 may also directly sample the expected value in the first movable angle range R11 or the second movable angle range R12 (ie,

or

The first angle of) is taken as the above-mentioned first specified joint angle, but it may not be limited to this.

In other embodiments, assuming that there are two controllable avatars A and B in the AR/VR environment, and both avatars A and B have first joint points, the processor 104 may be based on the first multivariate The mixed Gaussian distribution model G1' samples an angle in the first movable angle range R11 to be the first designated joint angle of the first joint point on the virtual avatar A in the first dimension. In addition, the processor 104 may also sample an angle in the second movable angle range R12 based on the first multivariate mixture Gaussian distribution model G1′ to serve as the first designated joint angle of the first joint point on the virtual avatar B in the first dimension. , So that different virtual avatars show different dance steps in response to the music, but it is not limited to this. Based on the above teachings, a person with ordinary knowledge in the art should be able to understand the manner in which the processor 103 obtains the specified joint angles of each joint point in each dimension in the second embodiment, which will not be repeated here.

In addition, the first joint point may also have two movable angle ranges in the second dimension, and the two movable angle ranges may also be modeled as another multivariate Gaussian distribution model. In this case, the way for the processor 104 to determine the specified joint angle of the first joint point in the second dimension can refer to the above teaching, which will not be repeated here. In addition, the movable angle range of other joint points in each dimension can also be modeled into a corresponding multivariable Gaussian model based on the above teaching. The details can also refer to the above teaching, and will not be repeated here.

After obtaining multiple designated joint angles corresponding to each joint point, in step S250 of the second embodiment, the processor 104 may adjust the joint angle of each joint point of the virtualized body according to the designated joint angle. In the second embodiment, the processor 104 may output the designated joint angle corresponding to each joint point in the form of a designated joint angle vector S1 (the dimension of which is, for example, 159×1). For example, assuming that the processor 104 samples the angle corresponding to the first expected value for each joint point as the designated joint angle of each joint point, the processor 104 can directly use the first row of the joint angle distribution matrix M1 as the designated joint angle. The joint angle vector S1. For another example, assuming that the processor 104 samples the angle corresponding to the second expected value for each joint point as the designated joint angle of each joint point, the processor 104 can directly use the third row of the joint angle distribution matrix M1 As the designated joint angle vector S1, the present invention may not be limited to this.

In this case, the processor 104 may, for example, generate a corresponding BVH motion capture data file based on the specified joint angle in the specified joint angle vector S1, and adjust the joint angle of each joint point of the virtualized body based on the BVH motion capture data file. For example, the processor 104 may adjust the joint angle of the first joint point in the first dimension to correspond to the aforementioned first specified joint angle (for example, the first expected value or the second expected value of the first multivariate Gaussian distribution model G1'). Based on this, the processor 104 can adjust the joint angles of each joint point of the virtual body in different dimensions according to the content of the BVH dynamic capture data file, so that the virtual avatar can present a specific movement (such as dance steps).

Please refer to FIG. 6, which is a schematic diagram of a BVH dynamic capture data file and a corresponding virtual avatar according to an embodiment of the present invention. In this embodiment, after the processor 104 generates the BVH dynamic capture data file 610 according to the previous teaching, the processor 104 can adjust the joint angles of the joint points on the virtual avatar 620 in each dimension according to the content therein, so that the virtual avatar 620 presents specific movements, dance steps, postures, etc., but is not limited thereto.

It should be understood that the above embodiment assumes that the audio signal F1 corresponds to the beat and music. For other audio signals that do not correspond to the beat or music, the present invention can execute the method of the present invention based on different mechanisms. Take the third embodiment for further explanation.

For example, in the third embodiment, it is assumed that the audio signal F2 connected to the audio signal F1 corresponds to music but does not correspond to the beat (that is, not on the beat). In this case, the processor 104 can still perform step S210 to receive the audio signal F2, and extract the high-level audio feature H2 from the audio signal F2. In one embodiment, the processor 104 may input the audio signal F2 (for example, an audio frame) to the CNN N1, so that the CNN N1 extracts high-level audio features H2 from the audio signal F2.

After that, in step S220, the processor 104 may extract the potential audio feature L2 from the high-level audio feature H2. In an embodiment, the processor 104 may input the high-level audio feature H2 into the first RNN N2, so that the first RNN N2 extracts the latent audio feature L2 from the high-level audio feature H2 based on the first internal state IS11. In this embodiment, since the first internal state IS11 can be understood as an operation from the previous stage, the first internal state IS11 can be regarded as the historical internal state in the third embodiment. In addition, since the first internal state IS11 has information related to the high-level audio feature H1 of the previous level, the potential audio feature L2 extracted by the first RNN N2 can also consider the information of the previous level (or multiple levels). But it is not limited to this.

In addition, in this embodiment, the first RNN N2 can output not only the latent audio feature L2 based on the high-level audio feature H2, but also the first internal state IS21 for use in the next stage, but it is not limited to this.

In the third embodiment, the processor 104 can also input the latent audio feature L2 into the specific neural network N3, so that the specific neural network N3 determines whether the audio signal F2 corresponds to the beat and whether it corresponds to the music based on the latent audio feature L2. But it is not limited to this.

Since the audio signal F2 in the third embodiment has been assumed to correspond to music but not on the beat, the processor 104 can perform step S230 in a manner different from the first and second embodiments to generate the corresponding joint angle distribution matrix M2. Specifically, in the third embodiment, the processor 104 may obtain a historical joint angle distribution matrix, where the historical joint angle distribution matrix may include a plurality of historical Gaussian distribution parameters, and the aforementioned historical Gaussian distribution parameters may correspond to virtualization The joint points on the body. In the third embodiment, the aforementioned historical joint angle distribution matrix is, for example, the joint angle distribution matrix M1 generated in the previous operation, and the aforementioned historical Gaussian distribution parameter is the content of the joint angle distribution matrix M1, but it may not be limited to this. .

Afterwards, the processor 104 can convert this historical joint angle distribution matrix (ie, the joint angle distribution matrix M1) into a reference audio feature L2', and define this reference audio feature L2' as a (new) potential audio feature L2. After that, the processor 104 may, for example, input the reference audio feature L2' (ie, the new potential audio feature L2) into the second RNN N4 to obtain the joint angle distribution matrix M2 from the second RNN N4.

In short, since the audio signal F2 is not on the beat, the processor 104 can ignore the original potential audio feature L2, and instead use the reference audio feature L2' converted from the joint angle distribution matrix M1 as the (new) potential The audio feature L2 is input to the second RNN N4 to obtain the joint angle distribution matrix M2 from the second RNN N4.

In one embodiment, in order to convert the dimension of the joint angle distribution matrix M1 into a reference audio feature L2' suitable for inputting the second RNN N4, the processor 104 can simply use a fully connected layer neural network to perform the conversion. In addition, the processor 104 may also perform the aforementioned conversion based on a convolutional layer and a pooling layer, but it may not be limited thereto. Feed the (transformed) joint angle distribution matrix M1 into the second RNN N4 to obtain the joint angle distribution matrix M2 for the relevant principle, please refer to " Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis, cs.LG, 2017 ", in This will not be repeated here.

In addition, in the third embodiment, the second RNN N4 can further generate the joint angle distribution matrix M2 based on the reference audio feature L2' and the second internal state IS12, so as to generate more information while considering the previous one or more levels of information. A good joint angle distribution matrix M2, but not limited to this.

After generating the joint angle distribution matrix M2, the processor 104 may generate the corresponding designated joint angle vector S1 based on the mechanism taught in the first and second embodiments, and adjust the motion/dancing step/posture of the virtual avatar accordingly to Corresponds to the state of the audio signal F2.

In the fourth embodiment, it is assumed that the audio signal F3 corresponds to the beat and the music, so the processor 104 can adjust the action/dance/posture of the virtual avatar to correspond to the audio based on the mechanism taught in the first and second embodiments. The details of the state of the signal F3 will not be repeated here.

In addition, in the fifth embodiment, assuming that the specific neural network R3 determines that the potential audio feature (not shown separately) of the audio signal FN indicates that the audio signal FN does not correspond to either the beat or the music, the processor 104 may not adjust the virtual The joint angle of each joint point on the avatar, or adjust the virtual avatar to present an idle posture. In this way, the virtual avatar can be prevented from dancing on its own without music, but it is not limited to this.

Please refer to FIG. 7, which is a schematic diagram of a training phase according to an embodiment of the present invention. In FIG. 7, the training mechanism shown can be used to generate the CNN N1, the first RNN N2, the specific neural network N3, and the second RNN N4 mentioned in the previous embodiment. Specifically, in this embodiment, the processor 104 may first input the music training data to the aforementioned neural network to be trained (ie, CNN N1, first RNN N2, specific neural network N3, and second RNN N4) middle. In an embodiment, the relevant model parameters of each neural network can be initialized to random values, but it is not limited to this.

After that, the processor 104 may model the corresponding (univariate/multivariate) Gaussian model of the movable angle range of each joint point of the virtualized body in each dimension based on the dance step training data, and generate a predicted dance step accordingly. After that, the processor 104 may calculate a loss function based on the predicted dance steps and the corresponding dance step training data, and adjust the relevant model parameters of the aforementioned neural networks (for example, the weights of neurons) according to the results of the loss function. The above process can be executed repeatedly until the generated predicted dance step is sufficiently close to the corresponding dance step training data. For the technical details of the above training phase, please refer to related prior art documents, which will not be repeated here.

In summary, the method and electronic device proposed by the present invention can allow the virtual avatar in the AR/VR environment to improvisely dance on the beat according to the current music without maintaining the dance step database. In addition, the method of the present invention allows the electronic device to consume less memory, and allows the electronic device to perform related operations in real time. Therefore, even if the electronic device is an edge device with relatively limited resources, the method of the present invention can still allow the electronic device to smoothly control the virtual avatar to dance with music.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be subject to those defined by the attached patent scope.

100: electronic device 102: storage circuit 104: processor 610: BVH motion capture data file 620: Virtual Avatar G1: The first Gaussian distribution model G1’: The first multivariate Gaussian distribution model H1, H2: High-level audio features IS11, IS21: The first internal state IS12: Second internal state L1, L2: potential audio features L2’: Reference audio features M1, M2: Joint angle distribution matrix N1: CNN N2: First RNN N3: specific neural network N4: Second RNN R1, R11: the first movable angle range R12: The second movable angle range S1, S2: Specify the joint angle vector S210~S250: steps

FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. 2 is a flowchart of a method for generating actions based on audio signals according to an embodiment of the present invention. FIG. 3 is a system architecture diagram drawn according to an embodiment of the present invention. FIG. 4 is a first Gaussian distribution model for modeling the first movable angle range according to the first embodiment of the present invention. FIG. 5 is a first multivariable mixture Gaussian distribution model for modeling the first and second movable angle ranges according to the second embodiment of the present invention. FIG. 6 is a schematic diagram of a BVH dynamic capture data file and a corresponding virtual avatar according to an embodiment of the present invention. Fig. 7 is a schematic diagram of a training phase according to an embodiment of the present invention.

S210~S250: steps

Claims (23)

A method for generating an action based on an audio signal includes: receiving a first audio signal, and extracting a first high-level audio feature from the first audio signal; extracting a first potential audio feature from the first high-level audio feature ; In response to determining that the first potential audio feature indicates that the first audio signal corresponds to a first beat, a first joint angle distribution matrix is obtained according to the first potential audio feature, wherein the first joint angle distribution matrix includes a plurality of Gaussian distribution parameters, and the Gaussian distribution parameters correspond to multiple joint points on a virtual body; in response to determining that the first potential audio feature indicates that the first audio signal corresponds to a first music, based on the first joint angle The distribution matrix obtains a plurality of designated joint angles corresponding to the joint points; and adjusts the joint angle of each joint point of the virtual body according to the designated joint angles. The method according to claim 1, wherein the first audio signal includes a first audio frame, and the step of extracting the first high-level audio feature from the first audio signal includes: inputting the first audio frame into a volume A neural network is used to extract the first high-level audio feature from the first audio frame by the convolutional neural network. The method according to claim 1, wherein the step of extracting the first potential audio feature from the first high-level audio feature includes: The first high-level audio feature is input into a first recurrent neural network, so that the first recurrent neural network extracts the first potential audio feature from the first high-level audio feature. The method according to claim 1, further comprising: inputting the first potential audio feature into a specific neural network, so that the specific neural network determines whether the first audio signal corresponds to the first potential audio feature based on the first potential audio feature And determining whether the first audio signal corresponds to the first music based on the first potential audio feature. The method according to claim 1, wherein in response to determining that the first potential audio feature indicates that the first audio signal does not correspond to any beat, the method further includes: obtaining a historical joint angle distribution matrix, and the historical joint angle The distribution matrix is converted into a reference audio feature, and the reference audio feature is defined as the first potential audio feature, wherein the historical joint angle distribution matrix includes a plurality of historical Gaussian distribution parameters, and the historical Gaussian distribution parameters correspond to the virtual The joint points on the avatar are obtained; the first joint angle distribution matrix is obtained according to the first potential audio feature. The method according to claim 1, wherein the step of obtaining the first joint angle distribution matrix according to the first potential audio feature includes: inputting the first potential audio feature into a second recurrent neural network, so that the second The recurrent neural network generates the first joint angle distribution matrix based on the first potential audio feature. The method according to claim 1, wherein the step of extracting the first potential audio feature from the first high-level audio feature includes: Obtain a first historical internal state; input the first high-level audio feature into a first recurrent neural network so that the first recurrent neural network extracts from the first high-level audio feature based on the first historical internal state The first potential audio feature. The method according to claim 7, wherein the step of obtaining the first historical internal state comprises: receiving a first historical audio signal prior to the first audio signal, and extracting a first historical audio signal from the first historical audio signal Historical high-level audio features; input the first historical high-level audio features into the first recurrent neural network, so that the first recurrent neural network generates the first historical internal state based on the first historical high-level audio features and corresponds to the A first historical potential audio feature of the first historical high-level audio feature. The method according to claim 8, wherein the step of obtaining the first joint angle distribution matrix according to the first potential audio feature includes: obtaining a second historical internal state; and inputting the first potential audio feature to a second recurrent nerve A network to generate the first joint angle distribution matrix based on the second historical internal state and the first potential audio feature by the second recurrent neural network. The method according to claim 9, wherein the step of obtaining the second historical internal state includes: The first historical potential audio feature is input to the second recurrent neural network, so that the second recurrent neural network generates the second historical internal state based on the first historical potential audio feature. The method according to claim 1, wherein in response to determining that the first potential audio feature indicates that the first audio signal does not correspond to any music, the method further includes: not adjusting the joints of the joint points on the virtualization body Angle, or adjust the avatar to present an idle posture. The method according to claim 1, wherein the joint points include a first joint point, the first joint point has a first movable angle range in a first dimension, and the Gaussian distribution parameters include a first joint point An expected value and a first standard deviation, wherein the first expected value and the first standard deviation correspond to a first Gaussian distribution model for modeling the first movable angle range. The method according to claim 12, wherein the designated joint angles include a first designated joint angle corresponding to the first joint point in the first dimension, and the first joint angle distribution matrix is based on the first designated joint angle corresponding to the The step of specifying the joint angles of the joint points includes: sampling a first angle within the first movable angle range based on the first Gaussian distribution model as the first angle of the first joint point in the first dimension Specify the joint angle. The method according to claim 13, wherein the first angle is an angle corresponding to the first desired value in the first movable angle range. The method according to claim 13, wherein the step of adjusting the joint angle of each joint point of the virtual body according to the specified joint angles includes: adjusting the joint angle of the first joint point in the first dimension to Corresponds to the first designated joint angle. The method according to claim 1, wherein the joint points include a first joint point, the first joint point has a first movable angle range and a second movable angle range in a first dimension, and the Gaussian The distribution parameters include a first expected value, a first standard deviation, a second expected value, and a second standard deviation. The first expected value, the first standard deviation, the second expected value, and the second standard deviation correspond to the A first multivariate mixture Gaussian distribution model for modeling the first movable angle range and the second movable angle range, wherein the first expected value and the first standard deviation correspond to the first movable angle range, and the second The expected value and the second standard deviation correspond to the second movable angle range. The method according to claim 16, wherein the designated joint angles include a first designated joint angle corresponding to the first joint point in the first dimension, and based on the first joint angle distribution matrix, the first designated joint angle corresponding to the The step of specifying the joint angles of the joint points includes: sampling a first angle in the first movable angle range or the second movable angle range based on the first multivariate mixture Gaussian distribution model as the first angle at the first joint The first designated joint angle in the first dimension. The method according to claim 17, wherein the first angle is an angle corresponding to the first desired value in the first movable angle range, or an angle corresponding to the second desired value in the second movable angle range. The method according to claim 12, wherein the first joint point has another movable angle range in a second dimension, and the Gaussian distribution parameters include another expected value and another standard deviation, wherein the The other expected value and the other standard deviation correspond to another Gaussian distribution model used to model the other movable angle range, and the specified joint angles further include a value corresponding to the first joint point in the second dimension. A second designated joint angle, and the method further includes: sampling a second angle in the other movable angle range based on the another Gaussian distribution model as the first joint point in the second dimension 2. Specify the joint angle. The method according to claim 19, wherein the second angle corresponds to the other desired value. The method according to claim 19, further comprising: adjusting the joint angle of the first joint point in the second dimension to correspond to the second designated joint angle. The method according to claim 1, wherein the step of adjusting the joint angle of each joint point of the virtualized body according to the specified joint angles includes: generating a biological visual hierarchical dynamic capture data file based on the specified joint angles, and The biological visual hierarchy dynamic capture data file adjusts the joint angle of each joint point of the virtual body. An electronic device includes: a storage circuit which stores a plurality of modules; and a processor which is coupled to the storage circuit and accesses the modules to perform the following steps: receiving a first audio signal and receiving The first audio signal extracts a first high-level audio feature; extracts a first potential audio feature from the first high-level audio feature; responding to determining that the first potential audio feature indicates that the first audio signal corresponds to a first For one beat, a first joint angle distribution matrix is obtained according to the first potential audio feature, wherein the first joint angle distribution matrix includes a plurality of Gaussian distribution parameters, and the Gaussian distribution parameters correspond to a plurality of joints on a virtual body Points; in response to determining that the first potential audio feature indicates that the first audio signal corresponds to a first music, based on the first joint angle distribution matrix to obtain a plurality of designated joint angles corresponding to the joint points; according to the designated The joint angle adjusts the joint angle of each joint point on the virtual body.

Images