JP7156138B2 - Information processing device, light action generation method, and light action generation program - Google Patents

Description

The present invention relates to an information processing device, a light action generation method, and a light action generation program.

In recent years, communication devices that are used for purposes such as communication and entertainment have been developed. Communication devices include, for example, robots that operate by moving their joints, smart speakers and AI (Artificial Intelligence) speakers that communicate with viewers by voice.

The communication device operates by, for example, emitting light from a light source or moving joints according to an action definition created by a creator. For example, the creator can make the communication device dance to the music by defining the light emission waveform of the light source and the movement of the joints in accordance with the music that is played when the communication device operates.

In addition, for example, the communication device outputs voice according to speech data specifying the contents of speech to have a conversation. A technique called lip-sync is also known, which generates a light emission waveform of a light source based on a voice that is spoken. With lip-syncing, by illuminating a light source that mimics the movement of a human mouth in conjunction with spoken words, it is possible to make it appear as if the person is speaking. Lip-sync makes it easier for the viewer to understand what the communication device is saying. Note that hereinafter, regardless of whether the creator-defined light source light emission or the light source light emission generated by lip-syncing, an action caused by light emission of the light source of the communication device may be referred to as a light action.

In this regard, techniques related to communication devices are known (for example, Patent Literature 1 and Patent Literature 2).

JP 2017-226051 A Japanese Patent Publication No. 2009-509673

However, the waveform of the luminescence intensity of the light source in the light action created by the creator and the luminescence intensity waveform of the light source of the light action generated by the lip sync are similar, making it difficult for the viewer to distinguish them. be. As a result, it may be indistinguishable whether the light source is e.g. emitting light with a creator-defined light action or representing the communication device speaking.

In one aspect, the present invention aims to generate a lip-sync light action that is distinguishable from a creator-defined light action.

An information processing apparatus according to one aspect of the present invention emits light in a defined action of content including a defined action of emitting a light source with a defined emission waveform and a lip-sync action of causing the light source to emit light according to an uttered voice. An identification unit that analyzes the frequency components of the waveform and identifies representative frequency components that represent the light emission waveform, and an adjustment that adjusts the frequency used in the light emission waveform of the light source in the lip-sync action to a frequency that can be distinguished from the representative frequency components. including the part and

Creator-defined light actions and identifiable lip-sync light actions can be generated.

FIG. 4 is a diagram illustrating a time-series operation of content executed by a communication device; FIG. 11 illustrates the generation of light source emission waveforms in an exemplary lip-sync; FIG. 10 is a diagram illustrating defined actions and lip-syncing performed by a communication device; FIG. 4 is a diagram illustrating a block configuration of a generation device that generates light actions during lip-sync according to an embodiment; FIG. 5 is a diagram illustrating the flow of generating a light action by lip-syncing according to the embodiment; FIG. 10 is a diagram illustrating adjustment of emission waveforms of light actions during lip-sync according to embodiments; It is a figure which illustrates the content information which concerns on embodiment. FIG. 7 is a diagram illustrating an operation flow of light action generation processing in lip sync according to the embodiment; FIG. 10 is a diagram showing an example of determining bias and determining frequency components to be suppressed based on light actions of a creator for a predetermined period according to the embodiment; FIG. 5 is a diagram illustrating suppression control of suppression target frequency components in lip sync according to the embodiment; FIG. 11 is a diagram illustrating a block configuration of a communication device according to another embodiment; FIG. FIG. 11 is a diagram illustrating an operational flow of light action generation processing in lip-sync executed by a control unit of a communication device according to another embodiment; It is a figure which illustrates the hardware constitutions of the information processing apparatus for implement|achieving the production|generation apparatus which concerns on embodiment. FIG. 10 is a diagram illustrating a hardware configuration for realizing a communication device according to another embodiment; FIG.

Several embodiments of the present invention will be described in detail below with reference to the drawings. In addition, the same code|symbol is attached|subjected to the element which corresponds in several drawings.

FIG. 1 is a diagram illustrating, in chronological order, content operations executed by the communication device 100. As shown in FIG. The communication device 100 includes, for example, a robot that operates by moving joints, a device that communicates with a viewer using words, such as a smart speaker and an AI speaker that communicates with the viewer by voice. In the example of FIG. 1, a robot is illustrated as the communication device 100 .

Also, the content defines, for example, a series of operations to be executed by the communication device 100 . The content may include, for example, periods of lip-syncing and periods of defining actions. Note that the content may include a plurality of lip-sync periods and definition action periods.

The communication device 100 includes a light source 101 such as an LED (light emitting diode). Then, during the lip-sync period, communication device 100 outputs voice according to speech data indicating the content of speech. Further, the communication device 100 generates a light emission waveform of the light source 101 in lip-sync based on the output sound, and causes the light source 101 to emit light with the generated light emission waveform. Note that an action caused by light emission from the light source 101 during the lip sync period is sometimes called a lip sync action.

FIG. 2 is a diagram illustrating generation of an emission waveform of light source 101 in an exemplary lip-sync. For example, as shown in FIG. 2(a), assume that there is a character string "please see the tenjibutsu" that is uttered during the lip-sync period.

In this case, as shown in FIG. 2B, speech synthesis technology can be used to synthesize speech from a character string to generate speech waveform data. The voice waveform data is reproduced, for example, when the communication device 100 is made to speak.

Also, based on the audio waveform data, the light emission waveform of the light source 101 can be generated by lip-sync. For example, the frequencies of human voice are distributed in the range of approximately 100 Hz to 2000 Hz. On the other hand, it is said that the range of 0.2 Hz to 50 Hz is the limit for humans to perceive changes in light intensity. Therefore, it is difficult to directly use the sound waveform data as the light emission waveform of the light source 101 . In lip-syncing, the light source 101 emits light as if the communication device 100 were speaking to the viewer. many. Therefore, processing is performed to generate a waveform in a frequency range suitable for lip-sync, such as a frequency range of 0.3 to 6 Hz, from a human voice waveform in a frequency range of 100 Hz to 2000 Hz. In one example, a waveform suitable for lip-sync can be generated by determining the envelope of audio waveform data.

FIG. 2(c) is a diagram illustrating an envelope waveform obtained from speech waveform data. As illustrated in FIG. 2(c), a waveform in a frequency range of 0.3-6 Hz can be generated based on audio data including a human voice waveform in a frequency range of 100-2000 Hz. In lip-sync, for example, when reproducing the voice waveform data, the communication device 100 can speak by changing the intensity of the light source 101 with a waveform having a correlation with the waveform of the voice waveform data thus obtained. It is possible to make the viewer recognize as if The lip-sync makes it easier for the viewer to understand the contents of the speech of the communication device 100 .

Further, during the defined action period, the communication device 100 operates according to the action defined by the creator. For example, the creator may define the angle of each joint and the light emission intensity of the light source 101 in association with the elapsed time from the start time of the defined action according to the use of the defined action to be generated. For example, when defining an action of dancing to music, the creator may define a dancing posture of the communication device 100 for each time period and cause the communication device 100 to dance. Also, the creator may define the light emission waveform of the light source 101 in the defined action, for example. For example, the creator may define the light emission waveform of the light source 101 according to the music that is played when the communication device 100 operates, the movement of the communication device 100, the content of the speech, and the like.

However, as described above, the light emission waveform of the light source 101 in the definition action created by the creator and the light emission waveform of the light source 101 due to lip-sync are similar, and it may be difficult for the viewer to distinguish between them. . As a result, it may be indistinguishable whether the light source is e.g. emitting light with a creator-defined light action or representing the communication device speaking.

FIG. 3 is a diagram illustrating defined actions and lip-syncs that are successively executed by the communication device 100. As shown in FIG. FIG. 3(a) illustrates the operation of the communication device 100 in chronological order. Also, FIG. 3B illustrates changes in the joint angle of the left arm of the communication device 100 . The communication device 100 swings the left arm up and down after rotating the angle of the left arm joint to be horizontal in the period of definition action 1 . After that, the communication device 100 maintains the posture during the lip-sync period, and swings the left arm up and down again during the definition action 2 period.

Further, as shown in the utterance of FIG. 3(c), during the lip sync period, the communication device 100 utters, "Please look at the results of Migite."

FIGS. 3(d) and 3(e) illustrate the light emission waveform of the light source 101 in light action. The light emission waveform of the light source 101 during the defined action 1 period and the light emission waveform of the light source 101 during the defined action 2 period are set by, for example, a creator who creates the action of the communication device 100 . Also, the light emission waveform of the light source 101 during the lip-sync period can be generated based on the waveform of the speech waveform data, as illustrated with reference to FIG. 2, for example.

Here, for example, the light emission period of the light source 101 during the defined action 1 or the defined action 2 period and the light emission period of the light source 101 during the lip-sync period can be distinguished as shown in FIG. Suppose they differ in degree. In this case, the viewer of the communication device 100 can recognize the switching between the defined action period and the lip-sync period. As a result, the viewer can more easily understand the utterance content from the light emission of the light source 101 during the lip-sync period.

However, for example, it is assumed that the light emission period of the light source 101 during the definition action 1 or the definition action 2 is close to the light emission period of the light source 101 during the lip-sync period, as shown in FIG. 3(e). In this case, it is difficult for the viewer of communication device 100 to recognize switching between the defined action period and the lip-sync period. As a result, the viewer may not be able to distinguish whether the light emission of the light source 101 is the light emission in the definition action or the light emission due to the lip-sync indicating that the communication device 100 is speaking. In this case, there is a risk that the effect of lip-syncing to promote the viewer's understanding of the content cannot be obtained, or that the viewer's understanding may be hindered in some cases. Therefore, it is desired to provide a technique for controlling the light emission of the light source 101 in the definition action generated by the creator and the light emission of the light source 101 by lip-sync so as to be distinguishable. A first embodiment will be described below.

(First embodiment)
FIG. 4 is a diagram illustrating a block configuration of a generation device 400 that generates an action of the light source 101 during lip sync according to the embodiment. The generation device 400 may be, for example, an information processing device such as a personal computer (PC), a mobile PC, or a tablet terminal used by creators to generate content and definition actions. Generation device 400 includes, for example, control unit 401 and storage unit 402 . The control unit 401 may operate, for example, as the identification unit 411 and the adjustment unit 412 . The storage unit 402 of the generation device 400 stores information such as content information 700 described later, for example. Details of these units and details of information stored in the storage unit 402 will be described later.

Further, in the embodiment described below, a light emission waveform of the light source 101 is generated by lip-sync that can be distinguished from the light emission of the light source 101 in the definition action defined by the creator.

FIG. 5 is a diagram illustrating the flow of generation of the light emission waveform of the light source 101 by lip sync according to the embodiment.

In step 501 (hereinafter, the step is written as "S", for example, S501), the control unit 401 of the generation device 400 generates a light action based on speech data during the lip-sync period. For example, the control unit 401 of the generation device 400 may generate a lip-sync light emission waveform based on speech waveform data of an utterance, as described with reference to FIG.

In S502, the control unit 401 of the generation device 400 analyzes the frequency of the light emission waveform of the light source 101 defined by the definition action included in the period other than the lip-sync period of the same content. For example, the control unit 401 of the generation device 400 may analyze the frequency components included in the emission waveform of the defined action adjacent to the lip-sync period.

In S503, the control unit 401 of the generation device 400 controls the light emission waveform of the light source 101 during the lip-sync period, for example, based on the analysis result of the definition action. For example, the control unit 401 adjusts the lip-sync emission waveform so that the frequencies are distinguishable from the main frequency components of the emission waveform of the light source 101 in the definition action adjacent to the lip-sync period. you can As a result, it is possible to prevent the viewer of the communication device 100 from being confused by not being able to recognize the switching between the definition action created by the creator and the lip-sync period.

In lip-sync, if a change in light intensity roughly corresponding to the strength of the voice is visible, the viewer tends to recognize it as an utterance. Therefore, it is possible to generate the emission waveform of the light source 101 during the lip-sync period without using some of the frequency components.

FIG. 6 is a diagram illustrating adjustment of the emission waveform of the light source 101 during the lip-sync period according to the embodiment. FIG. 6(a) is, for example, the light emission waveform of the light source 101 generated from the sound waveform data in FIG. 2(c). Contains frequency components.

FIG. 6(b) shows a waveform obtained by suppressing high frequency components among the frequency components included in the light emission waveform of FIG. 6(a). For example, the emission waveform of FIG. 6(b) can be obtained by passing the emission waveform of FIG. 6(a) through a low-pass filter. Here, it is assumed that a low-pass filter extracts a component that changes with a period of about 0.5 to 2 Hz, for example. Also in this case, for example, the components that change at about 0.5 to 2 Hz in the voice waveform data tend to include components corresponding to the strength of the voice that occurs during speech. Therefore, even if the light emission of the light source 101 is controlled by the light emission waveform obtained by suppressing the high frequency components with a low-pass filter, the viewer will perceive that the light emission intensity of the light source 101 changes according to the change in the strength of the voice. It looks like there is As a result, the light emitted from the light source 101 can give the viewer the illusion that the communication device 100 is speaking.

FIG. 6(c) is a waveform obtained by suppressing low frequency components among the frequency components included in the emission waveform of FIG. 6(a). For example, the emission waveform of FIG. 6(c) can be obtained by passing the emission waveform of FIG. 6(a) through a high-pass filter. Here, it is assumed that a high-pass filter extracts a component that changes with a period of about 2 to 6 Hz, for example. In this case as well, for example, the components that change at a cycle of about 2 to 6 Hz in the voice waveform data tend to include components that change when each character of voice is uttered. Therefore, even if the light emission of the light source 101 is controlled by the light emission waveform obtained by suppressing the low-frequency components with a high-pass filter, the viewer will feel that the light emission intensity of the light source 101 is changing in accordance with the utterance of the characters. looks like As a result, the light emitted from the light source 101 can give the viewer the illusion that the communication device 100 is speaking.

In this way, with lip-sync, if a waveform that is correlated with the voice waveform is extracted, it is possible to make the viewer recognize that the speaker is speaking even if some frequency components are removed. Therefore, it is possible to generate the light emission waveform of the light source 101 during the lip-sync period by avoiding the frequency components of the light emission waveform of the light source 101 used in the definition action created by the creator.

Also, as described above, it is said that the range of 0.2 Hz to 50 Hz is the limit for human beings to perceive changes in the intensity of light. Among them, the range in which humans can easily recognize changes in light intensity is limited to some extent, such as 0.3 to 6 Hz. Therefore, both the frequency used for the light emission of the light source 101 in the defined action by the creator and the frequency used for the light emission of the light source 101 in the lip sync tend to use the same frequency in the range of 0.3 to 6 Hz.

However, when a creator defines an operation of the communication device 100 and creates a defined action, the creator tends to cause the light source 101 to emit light in a cycle matching the purpose of the defined action. For example, when using the communication device 100 for guidance, the creator may refer to the blinking period of the turn signals of the vehicle, and may generate a light action so that the intensity changes at a period close to that period. Also, for example, when a creator creates a definition action that causes the communication device 100 to dance to music, there is a tendency to select the period of light emission intensity change in the light action in accordance with the tempo of the music.

That is, for example, even if creators are allowed to freely create definition actions included in content to be streamed for a certain purpose, the frequency components included in the light emission waveform of the light source 101 used in the definition actions are often biased. . Therefore, it is possible to generate the light emission waveform of the light source 101 during the lip-sync period by avoiding the frequency components of the light emission waveform of the light source 101 used in the definition action created by the creator.

Then, by adjusting the light emission cycle of the light action during the lip sync period to a cycle that can be identified from the light action light emission cycle defined by the definition action included in the content, the definition action and the lip sync by the creator are adjusted. makes it easier to identify switching.

As an example, assume that the creator has created a light action that causes the light source 101 to emit light at a slow cycle of about 1 Hz in the definition action. In this case, for example, in the light emission waveform of the light source 101 during the lip-sync period, frequencies of less than 2 Hz are filtered, and the light emission intensity of the light source 101 is controlled in a period as fast as 2 Hz or more. As a result, for example, there is a difference in the rhythm of light emission at the time of switching between the defined action and the lip-sync period, so the viewer can distinguish between the defined action period defined by the creator and the lip-sync period. . In another example, it is assumed that the creator has created a light action that emits the light source 101 at a fast cycle of about 3 Hz in the definition action. In this case, for example, frequencies of 2 Hz or more are filtered in the light emission waveform of the light source 101 during the lip-sync period, and the light emission intensity of the light source 101 is controlled at a slow cycle of less than 2 Hz. As a result, for example, there is a difference in the rhythm of light emission at the time of switching between the defined action and the lip-sync period, so the viewer can distinguish between the defined action period defined by the creator and the lip-sync period. .

Therefore, it is possible to prevent the viewer from confusing the lip-sync period with the defined action period created by the creator.

The generation of light actions during lip-sync according to embodiments is described in more detail below.

FIG. 7 is a diagram illustrating content information 700 according to the embodiment. The content information 700 defines the operation of the communication device 100 in content. In the content information 700, for example, entries are registered in which time, utterance, light action, joint angle 1, and joint angle 2 are associated with each other. The time is, for example, information indicating a period during which the operation of the entry is executed in the content. For the utterance, for example, a character string to be uttered by the communication device 100 is registered. In the example of FIG. 7, when the communication device 100 does not speak during the operation period corresponding to the entry, "none" is registered as the speech. Information specifying light emission of the light source 101 in a period corresponding to the entry is registered in the light action. Note that in the example of FIG. 7, when the communication device 100 does not cause the light source 101 to emit light during the operation period corresponding to the entry, "none" is registered as the light action. Also, when the light emission of the light source 101 of the communication device 100 is controlled by lip sync during the operation period corresponding to the entry, "lip sync" is registered in the utterance of the entry of the content information 700 .

The joint angle 1 and the joint angle 2 are, for example, information defining angles of respective joints provided in the communication device 100 during an operation period corresponding to the entry. Note that the joints included in the communication device 100 are not limited to the joint angles 1 and 2, and may include more joints. In another embodiment, the communication device 100 may include no joints. good too. In the embodiments described below, for example, the joint angle 1 is the angle of the right arm joint as seen from the viewer when facing the communication device 100, and the joint angle 2 is the angle of the left arm joint. A case will be described as an example.

By defining actions of the communication device 100 in the content information 700, the creator can cause the communication device 100 to perform various actions. For example, in the example of the content information 700 in FIG. 7, the time from 0 seconds to 3 seconds is the defined action period defined by the creator. The creator can define the operation of the communication device 100 by setting the content of speech, light action, angles of each joint, etc. of the communication device 100 in a period of 0 to 3 seconds. Also, in the example of FIG. 7, the lip-sync period is from 3 seconds to 8 seconds, and the creator sets the lip-sync to the light action to match the voice waveform corresponding to the character string of the entry's utterance. light emission of the light source 101 can be controlled.

Next, FIG. 8 is a diagram illustrating an operation flow of light action generation processing by lip sync according to the embodiment. For example, the control unit 401 of the communication device 100 may start the operation flow of FIG. 8 when an instruction to execute the content operation is input.

In S801, the control unit 401 reads the content information 700 that defines the operation of the content for which the execution instruction has been input.

In S802, the control unit 401 refers to the speech information included in the read content information 700, and acquires voice waveform data during the lip-sync period. For example, the control unit 401 may obtain voice waveform data by synthesizing the voice of the character string uttered from the character string registered in the utterance of the content information 700 . Alternatively, in another embodiment, for example, voice waveform data may be registered in the content information 700 instead of or in addition to the character string of speech. In this case, the control unit 401 may read the audio waveform data during the lip-sync period from the content information 700 in the process of S802.

In S803, the control unit 401 generates a light emission waveform of the light source 101 during the lip-sync period from the audio waveform data generated in S802. For example, the control unit 401 may generate a waveform of the emission intensity of the light source 101 according to the amplitude of the audio waveform data. In one example, the control unit 401 may generate the light emission waveform of the light source 101 by obtaining the envelope of the sound waveform data. In addition, embodiment is not limited to this. In another embodiment, the control unit 401 processes the voice waveform data with a band-pass filter that passes components of a predetermined frequency band (for example, 0.3 to 6 Hz) used in lip sync, and obtains a waveform signal An emission waveform may be generated having the emission intensity of the light source 101 proportional to the value. Still other embodiments may use an exponential function instead of proportionality, or use a transform function, such as a function that limits the luminous intensity to a predetermined value if the luminous intensity exceeds a predetermined value. 101 emission waveforms may be generated.

In S<b>804 , the control unit 401 analyzes the frequency component of the light emission waveform of the light source 101 during the period other than the lip sync included in the content information 700 . As an example, the control unit 401 may analyze the frequency components of the light emission waveform of the light source 101 during periods other than the lip sync registered in the content information 700 using FFT (fast Fourier transform). Note that the analysis of frequency components is not limited to this, and other techniques may be used.

In S805, the control unit 401 determines whether or not the frequency components of the light emission waveform of the light source 101 in the period other than the lip-sync period have a bias that satisfies a predetermined condition, based on the analysis result. In one example, the control unit 401 identifies a band formed by frequency components included in the light emission waveform of the light source 101 during a period other than lip sync. Then, the control unit 401 determines whether the band falls within a range that can be represented by ±20% for any frequency (for example, 1.0 Hz ±0.2 Hz, 2.0 Hz ±0.4 Hz, etc.). ) may be used to determine whether or not the frequency components are biased. In addition, when specifying a band formed by frequency components included in the light emission waveform of the light source 101 during a period other than the lip sync, the control unit 401 determines that the frequency component has a predetermined intensity (for example, an intensity of 30% of the maximum peak) or more. A band may be specified after extracting a frequency component.

Then, when the band formed by the frequency components included in the light emission waveform of the light source 101 in the period other than the lip-sync period has a width exceeding the range that can be expressed by ±20% with respect to an arbitrary frequency, the control unit 401 may determine that there is no bias in the band. In this case, the frequency components included in the light emission waveform of the light source 101 during the period other than the lip-sync period are widely distributed in a predetermined frequency range such as 0.3 to 6 Hz used for light action. there is In this case, it may be difficult to set the emission frequency for lip-sync so as to avoid confusion with the frequency for periods other than lip-sync. Therefore, the control unit 401 may determine NO in S805, and in S806, outputs warning information prompting correction so as to bias the frequency used in the content, and the operation flow ends.

On the other hand, for example, the band formed by the frequency components included in the light emission waveform of the light source 101 in the period other than the lip sync is distributed within a range that can be represented by ±20% with respect to any frequency. and In this case, the frequency components included in the light emission waveform of the light source 101 during the period other than the lip-sync period are biased, and the frequency of the light action during the lip-sync period is the frequency included in the light emission waveform of the light source 101 during the period other than the lip-sync period. It is possible to set avoiding components. Therefore, the control unit 401 may determine YES in S805, and the flow proceeds to S807. Note that the bias determination in S805 is not limited to this, and may be performed by other methods. For example, the width of the frequency range used to determine the bias is not limited to ±20%, and may be set to other ranges such as ±5% to ±50%. Also, the width of frequencies used for bias determination need not be expressed in % (percentage), and bias determination may be performed in a predetermined width such as 0.5 to 2.0 Hz, for example.

In S807, the control unit 401 identifies a representative frequency component that represents the light emission waveform of the light source 101 during a period other than lip sync. In one example, the control unit 401 specifies the maximum peak frequency among the frequency components of the light emission waveform of the light source 101 during the period other than the lip sync period as the representative frequency component representing the light emission waveform of the light source 101 during the period other than the lip sync period. you can

Note that the identification of the representative frequency component is not limited to this, and may be determined by other methods. For example, in another embodiment, the control unit 401 sorts the frequency components of the light emission waveform of the light source 101 in periods other than the lip-sync period in descending order of energy. Then, the control section 401 may specify the average value of the frequencies of the frequency components whose energy values are in the top 30% as the representative frequency component. Also, when calculating the average value, a weighted average weighted by the energy value may be used. In yet another embodiment, the control unit 401 sorts the frequency components of the light emission waveform of the light source 101 in the period other than the lip-sync period in order of the frequency components with the highest energy, and the frequencies formed by the components whose energy values are in the top 30%. A band may be identified as a representative frequency component.

In step S<b>808 , the control unit 401 determines suppression target frequency components to be suppressed in the light emission waveform of the light source 101 during the lip sync period based on the representative frequency components, and suppresses the determined suppression target frequency components. For example, when one frequency is identified as the representative frequency component in S807, the control unit 401 sets a predetermined width (for example, ±0.2 Hz, representative frequency×0.1 Hz width) in the plus and minus directions of the representative frequency component. etc.) is determined as the frequency component to be suppressed. Then, the control unit 401 attenuates the suppression target frequency component in the light emission waveform of the light source 101 during the lip-sync period. Also, the strength of attenuation may be set to a pass gain of 10-50% in the attenuation band. In one example, the strength of attenuation may be set to 30% pass gain in the attenuation band formed by the frequency components to be suppressed and 20% pass gain at the center frequency of the attenuation band.

Further, for example, when the frequency band is specified in S807 as the representative frequency component, the control unit 401 sets the representative frequency component as the frequency band to be suppressed, and sets the frequency band to be suppressed to a pass gain of 50 to 10%. can be attenuated by In one example, the control unit 401 attenuates the suppression target frequency band with a pass gain of 30%.

Also, the bandpass filters used for attenuation can be implemented in various ways. In one example, the bandpass filter is implemented by performing an FFT on the emission waveform of the light source 101 to attenuate the frequency component of interest, and then obtaining the attenuated emission waveform data by an IFFT (inverse fast Fourier transform). good. Alternatively, the bandpass filter may be approximately implemented using a real-space filter such as an FIR (Finite Impulse Response) filter.

In S<b>809 , the control unit 401 stores the acquired light emission waveform data after attenuation in the storage unit 402 as light action data in the lip-sync period, and the operation flow ends. In one example, the control unit 401 may store the post-attenuation light emission waveform data in the storage unit 402 in association with the entry in which the utterance of the content information 700 is set to lip sync.

As described above, according to the operation flow of FIG. 8, the control unit 401 generates the light action during the lip-sync period so as to be distinguishable from the light emission of the light source 101 during the defined action period defined by the creator. be able to.

(Modification 1)
Next, a modified example of the first embodiment will be described. In the above-described embodiment, for example, in the processing of S804 to S807, the frequency of the light action in the period other than the lip-sync period included in the content is used to determine the bias and determine the frequency component to be suppressed. However, embodiments are not so limited. For example, different frequencies before and after a switch may be sufficient to prevent confusion between creator-defined light actions and lip-sync light actions. Therefore, in the modified example, the control unit 401 uses the frequency components of the light actions defined by the creator within a predetermined period from the time of switching to determine bias and determine frequency components to be suppressed. make a decision.

FIG. 9 is a diagram illustrating an example of determination of bias and determination of frequency components to be suppressed based on light actions for a predetermined period according to the embodiment. For example, as shown in FIG. 9A, the control unit 401 acquires a waveform for a predetermined period immediately before switching to lip-sync in the adjacent creator-defined light action before lip-sync. Then, the control unit 401 may determine the bias and determine the frequency component to be suppressed based on the acquired frequency component of the waveform for the predetermined period.

In addition, as shown in FIG. 9B, the control unit 401 acquires a waveform for a predetermined period immediately after switching from lip-sync in an adjacent creator-defined light action after lip-sync. Then, the control unit 401 may determine the bias and determine the frequency component to be suppressed based on the acquired frequency component of the waveform for the predetermined period.

Furthermore, for example, as shown in FIG. 9C, the control unit 401 controls the light action defined by the creators adjacent before and after the lip-sync and the creator-defined Get the waveform of the light action. Then, the control unit 401 may determine the bias and determine the frequency component to be suppressed based on the acquired frequency component of the waveform for the predetermined period.

As described above, the control unit 401 may acquire the light emission waveform of the light source 101 for a predetermined period and analyze the frequency in S804 described above. Then, in S805, the control unit 401 may determine whether or not there is a bias in the extracted frequency components of the light emission waveform of the light source 101 during the predetermined period. In S807 and S808, the control unit 401 may specify a representative frequency component based on the extracted frequency component of the light emission waveform of the light source 101 during the predetermined period, and determine the frequency component to be suppressed from the specified representative frequency component. . In one example, the control unit 401 may specify, as the representative frequency component, the maximum peak frequency among the extracted frequency components of the light emission waveform of the light source 101 during the predetermined period.

For example, as described above, by using the light emission waveform of the light action defined by the creator in a predetermined period after switching, confusion between the light action defined by the creator and the light action by lip sync can be efficiently suppressed. can. In addition, the creator can freely set the light emission waveform of the light source 101 during a period other than the predetermined period to generate a light action.

(Modification 2)
Next, another modified example of the first embodiment will be described. As noted above, it may be sufficient to have different frequencies before and after the switch to prevent confusion between, for example, creator-defined light actions and lip-sync light actions. Therefore, in the following modified example, the control unit 401 increases the attenuation strength of the frequency component to be suppressed at the time of switching between the light action defined by the creator in the process of S808 and the light action by lip sync, and is controlled to be weak during the period of

FIG. 10 is a diagram illustrating suppression control of suppression target frequency components according to the modification. FIG. 10 is a graph in which the vertical axis represents the control strength of the filter and the horizontal axis represents the elapsed time in lip sync. In the example shown in FIG. 10, the filter control strength is 100% at the start and end of lip-sync.

Further, in the example of FIG. 10, the filter strength is lowered according to the elapsed time from the start of the lip-sync, and is set to 0% after the elapse of a predetermined time. In the example of FIG. 10, the filter strength is set to 0% a predetermined time before the end of the lip-sync, and the filter strength is gradually increased during the period from then until the end time.

Controlling the strength of such filters can be performed, for example, by: For example, the elapsed time from switching to lip sync is assumed to be T1. Also, the time required for speech from the start to the end of lip-sync can be estimated from, for example, the character string of the speech. The remaining time of lip-sync can be obtained by subtracting the elapsed time: T1 from the time required for speech, and this remaining time is T2. In this case, the filter control strength should be increased when either T1 or T2 is small. For example, if the smaller one of T1 and T2 is T, and the predetermined length of time for controlling the strength of the filter is 10 seconds, the filter strength can be set by Equation 1 below.
(10−T)/10×100 [%] Formula 1

In this case, when T is 10 seconds or more, the filter control strength is 0%, and when T is a value of 10 or less, the filter control strength can be increased as T becomes smaller. Note that when the filter control strength is 100%, for example, if the frequency band to be suppressed is set to a pass gain of 30% in S808, the control unit 401 sets the filter control strength to 100% for 30%. , and the filter may be operated with a pass gain of 30%. Further, when the control strength of the filter is 50%, the control section 401 may set the control strength of the filter to 50% for 30% and operate the filter with a pass gain of 15%.

For example, as described above, by controlling the period and intensity of filtering, the light source 101 is caused to emit light in a wide range of expressions according to the speech during the period when the intensity of the filter is weakened even during the lip-sync period. be able to.

Further, for example, as illustrated in FIG. 9C, when controlling the light emission waveform of the light source 101 during the period of lip sync with the light actions of the creator before and after the lip sync, the light actions of the creator before and after the lip sync Different frequencies may be used. In this case, the frequency to be attenuated at the start of lip-sync is determined from the creator's light action before lip-sync, and the frequency to be attenuated at the end of lip-sync is determined from the creator's light action after lip-sync. You can decide on an individual basis. Further, even if the frequencies to be attenuated at the start and end of lip-sync are separately determined in this way, as described with reference to FIG. , even if the frequency to be attenuated is changed, the viewer's discomfort can be suppressed.

(Second embodiment)
Next, a second embodiment will be described. In the above-described embodiment, when a creator generates content, an information processing device such as a computer used for content generation is used as the generation device 400, and light action generation processing in lip sync is executed as an example. there is However, embodiments are not limited to this, and the process of generating light actions in lip-syncing according to embodiments may be performed at other timings and with other devices. In one example, the process of generating a light action in lip-syncing according to the embodiment may be performed in the communication device 100 .

FIG. 11 is a diagram illustrating the block configuration of the communication device 100 according to the embodiment. Communication device 100 includes, for example, control unit 1101 , storage unit 1102 , light source control unit 1103 , and light source 101 . The control unit 1101 includes, for example, a specifying unit 1111 and an adjusting unit 1112. The storage unit 1102 of the communication device 100 stores information such as the content information 700 described above, for example. The light source control unit 1103 controls the light emission intensity of the light source 101 according to instructions from the control unit 1101, for example.

FIG. 12 is a diagram illustrating an operational flow of light action generation processing in lip-sync performed by the control unit 1101 of the communication device 100 according to the embodiment. For example, the control unit 1101 of the communication device 100 may start the operation flow of FIG. 12 when an instruction to execute a content operation is input.

Note that the processing from S1201 to S1207 may correspond to the processing from S801 to S804, S807, and S808 in FIG. 8, respectively. For example, in S1201 to S1207, the control unit 1101 may perform the same processing as the processing in S801 to S804, S807, and S808. Note that the control unit 1101 may read the content information 700 from the storage unit 1102 in S1201.

In S1208, the control unit 1101 executes the content operation using the light emission waveform of the light source 101 during the lip-sync period determined in the processing from S1201 to S1207, and the operation flow ends.

As described above, the light emission control of the light source 101 during the lip sync period according to the embodiment can also be executed in the communication device 100 .

Note that the action that the communication device 100 performs in the content may be dynamically generated. For example, the communication device 100 may dynamically select the details of the action to be executed in the content according to the viewer's age and gender detected by a sensor. As an example, if the viewer is a child such as an elementary school student, up-tempo and lively content such as "A lot of elementary school students came today. Let's dance with me!" When the user is an elderly person, control may be performed such that relatively calm content is played. In this way, for example, even when content is dynamically generated, the control unit 1101 performs the lip-sync light action generation process according to the embodiment on the generated content to perform lip-sync. Light emission of the light source 101 during the period may be controlled.

Although the embodiment has been exemplified above, the embodiment is not limited to this. For example, the operational flow described above is an example, and embodiments are not limited thereto. If possible, the operation flow may be executed by changing the order of the processes, may include additional processes, or may omit some of the processes. For example, the processing of S805 and S806 in FIG. 8 may be omitted if the creator is not prompted to make corrections.

Further, in the above-described embodiment, the case where the communication device 100 is a robot is described as an example, but the embodiment is not limited to this. For example, in another embodiment, embodiments may be applied in a communication device 100 that includes speakers and light sources to communicate with an audience. In other words, for example, the communication device 100 may not include moving joints such as arms, and in that case, the content information 700 may not include joint angle information. Further, in the above-described embodiments, for example, pulse width modulation may be used to control the emission intensity of the light source 101 .

Note that in the above-described embodiment, the control unit 401 of the generation device 400 operates as the identification unit 411, for example, in the processes from S801 to S807 in FIG. Also, in the process of S808, the control unit 401 of the generation device 400 operates as the adjustment unit 412, for example. In the processing from S1201 to S1206 in FIG. 12, the control unit 1101 of the communication device 100 operates as the identification unit 1111, for example. Also, in the process of S1207, the control unit 1101 of the communication device 100 operates as the adjustment unit 1112, for example.

FIG. 13 is a diagram illustrating a hardware configuration of an information processing device 1300 such as a computer for realizing the generation device 400 according to the embodiment. A hardware configuration for realizing the generating device 400 of FIG. 13 includes, for example, a processor 1301, a memory 1302, a storage device 1303, a reading device 1304, a communication interface 1306, and an input/output interface 1307. Note that the processor 1301, memory 1302, storage device 1303, reader 1304, communication interface 1306, and input/output interface 1307 are connected to each other via a bus 1308, for example.

The processor 1301 may be, for example, a single processor, multiple processors, or multiple cores. The processor 1301 provides some or all of the functions of the control unit 401 described above by executing a program describing the procedure of the operation flow of FIG. 8 described above, for example, using the memory 1302 . For example, the processor 1301 of the generation device 400 may operate as the identification unit 411 and adjustment unit 412 described above by reading and executing a program stored in the storage device 1303 .

The memory 1302 is, for example, a semiconductor memory and may include a RAM area and a ROM area. The storage device 1303 is, for example, a hard disk, a semiconductor memory such as a flash memory, or an external storage device. Note that RAM is an abbreviation for Random Access Memory. Also, ROM is an abbreviation for Read Only Memory.

Reader 1304 accesses removable storage medium 1305 according to instructions from processor 1301 . The removable storage medium 1305 is, for example, a semiconductor device (USB memory, etc.), a medium for inputting/outputting information by magnetic action (magnetic disk, etc.), a medium for inputting/outputting information by optical action (CD-ROM, DVD, etc.). Note that USB is an abbreviation for Universal Serial Bus. CD is an abbreviation for Compact Disc. DVD is an abbreviation for Digital Versatile Disk. The storage unit 402 described above includes, for example, a memory 1302 , a storage device 1303 , and a removable storage medium 1305 . For example, the content information 700 may be stored in the storage device 1303 of the generation device 400 .

Communication interface 1306 transmits and receives data to and from a network and other devices according to instructions from processor 1301 . Input/output interface 1307 may be, for example, an interface between an input device and an output device. The input device is, for example, a device such as a keyboard or mouse that receives instructions from the user. The output device is, for example, a display device such as a display and an audio device such as a speaker.

Also, FIG. 14 is a diagram illustrating a hardware configuration for realizing the communication device 100 according to the embodiment. A hardware configuration for realizing communication device 100 in FIG. The information processing device 1400 includes, for example, a processor 1401 , a memory 1402 , a communication interface 1406 and an input/output interface 1407 . Note that the processor 1401, memory 1402, communication interface 1406, and input/output interface 1407 are connected to each other via a bus 1408, for example.

The processor 1401 may be, for example, a single processor, multiple processors, or multiple cores. The processor 1401 provides some or all of the functions of the control unit 1101 described above by executing a program describing the procedure of the operation flow of FIG. 12 described above, for example, using the memory 1402 . For example, the processor 1401 of the communication device 100 may operate as the identifying unit 1111 and the adjusting unit 1112 described above by reading and executing programs stored in the memory 1402 .

The memory 1402 is, for example, a semiconductor memory and may include a RAM area and a ROM area. The memory 1402 is, for example, an example of the storage unit 1102 described above. For example, the memory 1402 of the communication device 100 may store the content information 700 .

Communication interface 1406 transmits and receives data to and from a network and other devices as directed by processor 1401 . Input/output interface 1407 may be, for example, an interface between an input device and an output device. A sound output device 1409 such as a speaker is connected to the input/output interface 1407, for example. The sound output device 1409 may, for example, according to the instruction of the processor 1401, output the sound of uttering the character string registered in the utterance of the content information 700. FIG. Also, the input device may be a device such as a button or a touch panel that receives an instruction from the user, for example.

Also, the light source control circuit 1411 controls light emission of the light source 101 such as an LED according to instructions from the processor 1401 . The light source control circuit 1411 is an example of the light source control section 1103 described above. Drive control circuit 1421 controls drive circuit 1422 that drives joints of communication device 100 such as motors, according to instructions from processor 1401 . Note that if the hardware configuration of FIG. 14 is that of a smart speaker instead of a robot, for example, the drive control circuit 1421 and the drive circuit 1422 may be omitted.

Each program according to the embodiment is provided to the generation device 400 and the communication device 100 in the form described below, for example.
(1) It is pre-installed in the storage device 1303 and memory 1402 .
(2) provided by removable storage medium 1305;
(3) provided by a server such as a program server;

Note that the hardware configuration described with reference to FIGS. 13 and 14 is an example, and the embodiment is not limited to this. For example, some or all of the functions of the control unit 401 and the control unit 1101 described above may be implemented as hardware such as FPGA and SoC. Note that FPGA is an abbreviation for Field Programmable Gate Array. SoC is an abbreviation for System-on-a-chip.

Several embodiments are described above. However, it should be understood that the embodiments are not limited to the embodiments described above, but encompass various variations and alternatives of the embodiments described above. For example, it will be appreciated that various embodiments may be embodied with varying elements without departing from the spirit and scope thereof. Also, it will be understood that various embodiments can be implemented by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments. Furthermore, various embodiments can be implemented by deleting or replacing some components from all the components shown in the embodiments, or by adding some components to the components shown in the embodiments. It will be understood by those skilled in the art that

100: communication device 101: light source 400: generation device 401: control unit 402: storage unit 411: identification unit 412: adjustment unit 1101: control unit 1102: storage unit 1103: light source control unit 1111: identification unit 1112: adjustment unit 1300: Information processing device 1301 : Processor 1302 : Memory 1303 : Storage device 1304 : Reading device 1305 : Removable storage medium 1306 : Communication interface 1307 : Input/output interface 1308 : Bus 1400 : Information processing device 1401 : Processor 1402 : Memory 1406 : Communication interface 1407: input/output interface 1408: bus 1409: sound output device 1411: light source control circuit 1421: drive control circuit 1422: drive circuit

Claims (11)

Analyzing the frequency components of the light emission waveform in the definition action of content including a defined action of emitting light with a defined light emission waveform and a lip-sync action of causing the light source to emit light in response to an uttered voice, an identifying unit that identifies a representative frequency component that represents an emission waveform;
an adjustment unit that adjusts the frequency used in the light emission waveform of the light source in the lip sync action to a frequency distinguishable from the representative frequency component;
An information processing device, including The adjustment unit
Based on the representative frequency component, a frequency component to be suppressed including the representative frequency component is determined;
generating a light emission waveform of the light source in the lip sync action by attenuating the frequency component to be suppressed among the frequency components included in the light emission waveform of the light source generated in response to the uttered voice;
2. The information processing apparatus according to claim 1, characterized by: 3. The specifying unit outputs warning information prompting correction of the defined action when frequency components of the emission waveform in the defined action are not biased to satisfy a predetermined condition. The information processing device according to . The specifying unit emits light of the first defined action within a predetermined period from timing of switching between a first defined action executed immediately before the lip sync action and the lip sync action among the defined actions. identifying the representative frequency component based on the waveform;
4. The information processing apparatus according to claim 2, wherein: The specifying unit emits light of the first defined action within a predetermined period from timing of switching between a first defined action executed immediately before the lip sync action and the lip sync action among the defined actions. Based on the waveform, determine whether there is a bias that satisfies the predetermined condition;
4. The information processing apparatus according to claim 3, characterized by: The adjustment unit sets a first intensity to attenuate the frequency component to be suppressed at a timing of switching between the first definition action and the lip sync action, and sets the first intensity to a first intensity. setting the strength to a second strength that attenuates the frequency component to be suppressed with a strength weaker than the first strength at a second timing after a predetermined time has elapsed from the timing of switching to the lip-sync action. 6. The information processing apparatus according to claim 4 or 5, characterized by: The specifying unit emits light of the second defined action within a predetermined period from timing of switching between the second defined action executed immediately after the lip sync action and the lip sync action among the defined actions. identifying the representative frequency component based on the waveform;
5. The information processing apparatus according to any one of claims 2 to 4, characterized by: The specifying unit emits light of the second defined action within a predetermined period from timing of switching between the second defined action executed immediately after the lip sync action and the lip sync action among the defined actions. Based on the waveform, determine whether there is a bias that satisfies the predetermined condition;
4. The information processing apparatus according to claim 3, characterized by: The adjustment unit sets a first strength for attenuating the frequency component to be suppressed at the timing of switching between the second definition action and the lip-sync action, and performs the second definition action and the lip-sync action. setting the strength to a second strength that attenuates the frequency component to be suppressed with a strength weaker than the first strength at a second timing that is a predetermined time before the timing of switching to the lip-sync action. 9. The information processing apparatus according to claim 7 or 8, characterized by: Analyzing the frequency components of the light emission waveform in the definition action of content including a defined action of emitting light with a defined light emission waveform and a lip-sync action of causing the light source to emit light in response to an uttered voice, Identify a representative frequency component that represents the light emission waveform,
Adjusting the frequency used in the light emission waveform of the light source in the lip sync action to a frequency distinguishable from the representative frequency component;
A light action generation method executed by an information processing device, comprising: Analyzing the frequency components of the light emission waveform in the definition action of content including a defined action of emitting light with a defined light emission waveform and a lip-sync action of causing the light source to emit light in response to an uttered voice, Identify a representative frequency component that represents the light emission waveform,
Adjusting the frequency used in the light emission waveform of the light source in the lip sync action to a frequency distinguishable from the representative frequency component;
A light action generation program that causes an information processing device to execute processing.

Images