JP2016080908A - Signal processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To visually process a sound by preventing the synchronization of a video and the sound from collapsing.SOLUTION: A video operation input part 104 is operated by the user of a karaoke device 100, and generates operation content data corresponding to the operation, and applies the operation content data to a voice processing parameter generation part 105 and a video processing parameter generation part 106. The voice processing parameter generation part 105 generates a voice processing parameter from the operation content data applied by the video operation input part 104, and applies the voice processing parameter to a voice processing part 102 for contributing to control of voice processing. Also, the video processing parameter generation part 106 generates a video processing parameter from the operation content data applied by the video operation input part 104, and applies the video processing parameter to a video processing part 108 for contributing to control of video processing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for processing a signal representing each of video and sound that are reproduced in synchronization with each other.

  With the development of digital technology, it is common for general users to create and publish content consisting of video and sound that are played back synchronously, and to further process one of the video and sound that make up such content It has become. An example of the former is a video posted on a video posting site, and an example of the latter is a karaoke device. Among them, in particular, with respect to karaoke apparatuses, in recent years, those capable of realizing a variety of sound processing and video processing have become popular.

  As an example of sound processing in a karaoke apparatus, frequency conversion with a key conversion as a specific example is given to a sound signal representing the sound waveform of an accompaniment sound. For example, a user of a karaoke device having a key conversion function adjusts a key of an accompaniment sound by moving up and down a virtual slider displayed on a slider of a remote controller of the karaoke device or a display device attached to the karaoke device, You can sing karaoke songs with accompaniment sounds according to your own keys.

  On the other hand, as a specific example of the video processing in the karaoke device, various video contents displayed on the display device as the karaoke song progresses (for example, a background video in accordance with the image of the karaoke song or an overlay on this background video) (For example, subtitle video of lyrics displayed). Specific examples of the karaoke apparatus having this type of video processing function include karaoke apparatuses disclosed in Patent Documents 1, 2, and 3. Video processing in the karaoke apparatus disclosed in each of these documents is as follows.

  Patent Document 1 discloses video processing for synthesizing a singer's video in real time with a background video and lyrics subtitle video displayed on a display device. Patent Document 2 discloses video processing for synthesizing a video from a digital camera with a lyrics subtitle video as a background video.

  Further, Patent Document 3 discloses video processing for performing processing such as superimposing an ornament image set by a singer on an avatar indicating a singer and combining it with a background video. An avatar is a character that imitates a human being.

JP 2001-045373 A JP 2009-020234 A Japanese Patent No. 5018797

  In the karaoke apparatuses disclosed in Patent Documents 1, 2, and 3, although image processing is considered, sound processing corresponding to the image processing is not considered. Originally, the video reproduced by the karaoke apparatus is reproduced in synchronization with the sound, and it is not preferable that the synchronization with the sound is lost by processing only the video.

  Similarly, it is not preferable that the synchronization of both of the video and the audio to be reproduced in synchronization is changed, and the synchronization between the two is lost. However, in the conventional sound processing, such consideration has not been paid. In addition, in conventional sound processing, it was possible to perform rough processing such as increasing or decreasing the volume of the sound signal by direct operation, but performing detailed processing such as changing the frequency distribution of the sound signal I couldn't. The former can be performed without visually grasping the characteristics (for example, frequency distribution) of the sound signal, but the latter is difficult unless the characteristics of the sound signal are visually grasped. Further, the conventional sound processing has a problem that it is impossible to visually grasp the fine change of sound caused by the processing.

  The present invention has been made in view of the circumstances as described above. The purpose of the present invention is to prevent at least one of the synchronized video and sound from being disrupted, and further to allow the user to process the sound. It is to provide a technique that enables visual recognition.

  In order to solve the above-described problems, the present invention provides a parameter generation unit that generates a video processing parameter that represents processing content for a video displayed on a display device according to an input sound signal, and a sound that is expressed by the input sound signal. And a video processing unit that performs processing based on the video processing parameters for video data representing video to be displayed on the display device, and supplies the processed video data to the display device. Providing the device. According to the present invention, the video processing parameters for processing the video to be displayed on the display device in synchronization with the sound reproduced according to the input sound signal are generated according to the input sound signal. Therefore, the synchronization of video and sound is not lost.

  In a more preferred aspect, the signal processing device includes a video generation unit that generates video data to be processed by the video processing unit based on the input sound signal, and an operation for the video displayed on the display device. An input video operation input unit; and a sound processing unit that processes the input sound signal based on a sound processing parameter and outputs the processed sound, and the parameter generation unit performs the processing according to the input sound signal. In place of the process of generating the video processing parameter, the process of generating the video processing parameter according to the operation input to the video operation input unit, or the operation input to the input sound signal and the video operation input unit In response to this, processing for generating the video processing parameter is executed, and the sound processing parameter is generated in accordance with an operation input to the video operation input unit. The features.

  For example, if the video processing parameter is generated by the parameter generation unit in response to an operation on the video operation input unit, when an operation for processing the video is performed on the video displayed on the display device, A video processing parameter representing the processing content is generated according to the operation, and a sound processing parameter representing the processing content for the sound output in synchronization with the video is generated according to the operation. The video data representing the video displayed on the display device is processed based on the video processing parameter, and the input sound signal representing the output sound is processed based on the sound processing parameter. . For this reason, if the level of the video processing parameters generated according to the operation performed on the video operation input unit and the level of the sound processing parameters are aligned (for example, the video processing parameters for enlarging the video according to the pinch-out operation). And a sound processing parameter that reduces the volume in response to a pinch-in operation and a sound processing parameter that lowers the volume. Visual and audio processing can be performed without synchronizing video and sound.

  As a preferred aspect, the input sound signal includes a sound signal representing a user's singing voice, and the video data is data representing each part of an avatar that imitates the user, and different sound for each part of the avatar. An effect is assigned, and the parameter generation unit generates a video processing parameter indicating processing content for the part for each part of the avatar operated by the video operation input unit, and a part for which the operation is performed A sound processing parameter that adjusts the acoustic effect corresponding to is generated. According to this aspect, the sound includes the user's singing voice. Furthermore, the video displayed on the display device is an avatar that imitates the user, and when an operation is performed for each part of the avatar, different acoustic effects assigned to each part of the avatar are applied to the sound. Therefore, even if the video of the part of the avatar is processed, the synchronization of the video and the sound is not lost, and the voice processing for applying the acoustic effect corresponding to the part of the avatar can be visually performed.

  Further, as another preferred aspect, a parameter generation process for generating a video processing parameter indicating processing content for a video to be displayed on a display device on a computer according to an input sound signal, and a sound expressed by the input sound signal are synchronized. And a video processing process for processing the video data representing the video to be displayed on the display device based on the video processing parameters and executing the processed video data to the display device. Also according to this aspect, the video processing parameters for processing the video to be displayed on the display device in synchronization with the sound reproduced according to the input sound signal are generated according to the input sound signal. Therefore, the synchronization of video and sound is not lost.

It is a block diagram which shows the structure of the karaoke apparatus 100 which is 1st Embodiment of the signal processing apparatus of this invention. 3 is a conceptual diagram of a voice processing table stored in the karaoke apparatus 100. FIG. 3 is a conceptual diagram of a video processing table stored in the karaoke apparatus 100. FIG. 3 is a conceptual diagram illustrating a reverb change table stored in the karaoke apparatus 100. FIG. 3 is a conceptual diagram illustrating a background video change table stored in the karaoke apparatus 100. FIG. It is a flowchart which shows the flow of the process in which the audio processing parameter production | generation part 105 produces | generates an audio processing parameter. It is a flowchart which shows the flow of the process in which the video process parameter production | generation part 106 produces | generates a video process parameter. 3 is a diagram illustrating an example of an image displayed on an image operation input unit 104 of the karaoke apparatus 100. FIG. It is the figure which illustrated the image | video displayed after performing pinch out operation to the video operation input part. 6 is a diagram illustrating an image displayed after performing a touch operation on the image operation input unit 104. FIG. It is a flowchart which shows the flow of the process in which the audio processing parameter production | generation part 105 produces | generates an audio processing parameter. It is a flowchart which shows the flow of the process in which the video process parameter production | generation part 106 produces | generates a video process parameter. It is a conceptual diagram of the avatar part process table memorize | stored in the karaoke apparatus which is 2nd Embodiment of this invention. It is a conceptual diagram of the compressor processing table memorize | stored in the karaoke apparatus. It is a block diagram which shows the structure of the karaoke apparatus 200 which is 3rd Embodiment of this invention. It is a flowchart which shows the flow of the process in which the feature-value analysis part 204 produces | generates an image processing parameter.

Embodiments of the present invention will be described below with reference to the drawings.
<First Embodiment>
(A: Configuration)
FIG. 1 is a block diagram showing a configuration of a karaoke apparatus 100 which is a first embodiment of a signal processing apparatus of the present invention. The karaoke apparatus 100 is a device that picks up a user's singing voice and outputs it along with an accompaniment sound to a sound emitting device such as a speaker, and also displays an avatar, which is a character imitating the user, with a background image and displays it on a display device. is there. As shown in FIG. 1, a karaoke apparatus 100 includes an audio input unit 101, an audio processing unit 102, an audio output unit 103, a video operation input unit 104, an audio processing parameter generation unit 105, an image processing parameter generation unit 106, and an image generation unit. 107, a video processing unit 108, a background video acquisition unit 109, and a video output unit 110. Among the units constituting the karaoke device 100, the audio processing unit 102, the audio processing parameter generation unit 105, the video processing parameter generation unit 106, the video generation unit 107, and the video processing unit 108 are a CPU (Central Producing Unit: 1 is a software module realized according to a control program (not shown in FIG. 1) stored in advance in the karaoke apparatus 100.

  The background video acquisition unit 109 is connected to a host computer via a network. The host computer distributes the accompaniment sound data representing the accompaniment sound of the karaoke song selected by the user and the background image data to the background image acquisition unit 109. The background video data is data representing video to be displayed on the display device in synchronization with the accompaniment sound. The background video data of this embodiment is composed of a plurality of video data representing different videos. For example, video data representing a music hall and video data displaying a bathroom are included in the background video data. The background video acquisition unit 109 receives background video data and accompaniment sound data distributed from the host computer, gives the background video data to the video processing unit 108, and gives the accompaniment sound data to the audio processing unit 102. In FIG. 1, illustration of signal lines connected to the background video acquisition unit 109 and the audio processing unit 102 is omitted.

  The voice input unit 101 is a microphone, for example. As shown in FIG. 1, the karaoke apparatus 100 of this embodiment has a plurality of audio signal input units 101. Each of the plurality of voice input units 101 collects voices of a plurality of singers who sing with each of them, and gives a voice signal representing a waveform of the collected voices to the voice processing unit 102. In this embodiment, when the karaoke apparatus 100 is used by a plurality of singers, each singer sings with a different voice input unit 101. Therefore, the number of voice input units 101 must be the same as the number of singers or more than the number of singers. The voice input unit 101 not used by the singer does not collect voice and does not generate a voice signal.

  The audio processing unit 102 processes the audio signals given from the audio input unit 101 and the background video acquisition unit 109, distributes the processed audio signals to each of the plurality of audio output units 103, and outputs them. The audio output unit 103 is, for example, a speaker. The audio output unit 103 emits the sound represented by the audio signal given from the audio processing unit 102. In the present embodiment, the numbers of the voice input unit 101 and the voice output unit 103 do not match (N and T shown in FIG. 1 are not equal), but the number of both may match. Further, in the present embodiment, a plurality of audio input units 101 and a plurality of audio output units 103 are provided, but one each may be provided.

  As an example of the audio signal processing performed by the audio processing unit 102, an increase or decrease in the volume of the audio signal given from the audio input unit 101, an acoustic effect such as reverb is added to the audio signal, or a plurality of audio output units 103 The sound image can be localized by adjusting the output balance. These processes are performed according to the voice processing parameters generated by the voice processing parameter generation unit 105.

  The audio signal generated by the audio input unit 101 is also given to the video generation unit 107. The video generation unit 107 reads avatar data corresponding to each given audio signal from a plurality of avatar data stored in a storage device (not shown in FIG. 1), and outputs the avatar data to the video processing unit 108. To do. More specifically, each of the plurality of avatar data is stored in the storage device in association with a different audio signal identifier. The audio signal identifiers are numbers 1 to N assigned to the audio input units 101 shown in FIG. That is, the number of voice input units 101 provided in the karaoke apparatus 100 matches the number of avatar data stored in the storage device. When the audio signal is given from the audio signal input unit 101, the video generation unit 107 reads out the avatar data corresponding to the audio signal identifier indicating the audio signal input unit 101 from the storage device, and sends the avatar data to the video processing unit 108. give.

  Note that the audio signal identifier stored in the storage device in association with the avatar data is not limited to the number assigned to the audio input unit 101, and is waveform data of the audio signal and feature amount data representing the characteristics of the audio signal. It may be. When waveform data or feature value data is used as the audio signal identifier, the video generation unit 107 determines the waveform or feature of the audio signal when the audio signal is received from the audio input unit 101, and the waveform Or the process which reads the avatar data corresponding to the audio | voice identifier which shows the characteristic from a memory | storage device is performed. In this aspect, even if the singer changes the microphone to another microphone during the song, the same avatar as before the change is displayed on the display device even after the change.

  The user who uses the voice input unit 101 indicated by the voice signal identifier may be selected according to his / her preference from avatar data prepared in advance as avatar data associated with the voice signal identifier. Even if the user selects from the data representing each part of the avatar prepared in advance (for example, the avatar's face, torso, limbs, etc.) according to the user's preference, good. The selection and generation of the avatar data may be performed in the karaoke apparatus 100, or the selection or generation is performed by a device other than the karaoke apparatus 100 such as a personal computer, and the selection or generation result is stored in the karaoke apparatus 100. It may be a

  The video processing unit 108 processes both the avatar data given from the video generation unit 107 and one video data selected from a plurality of video data included in the background video data given from the background video acquisition unit 109. Then, both are combined and provided to the video output unit 110 as output video data. The video output unit 110 is a display device such as a monitor, and displays a video represented by given output video data. Selection, processing, and synthesis in the video processing unit 108 are performed according to the video processing parameters generated by the video processing parameter generation unit 106. Examples of processing performed in accordance with the video processing parameters include avatar enlargement or reduction, avatar display position change, background video change, and the like. In a situation where the singer selects a song to be sung in the karaoke apparatus 100 and the accompaniment sound starts to flow and the singer has not yet started singing, the video output unit 110 does not display the avatar but displays only the background video. This is because if the singer has not started singing, an audio signal is not given from the audio input unit 101 to the video generation unit 107, and the video generation unit 107 does not output the avatar data to the video processing unit 108. Thereafter, when the singer starts singing, the image processing unit 108 displays the avatar data and the background image so that the avatar is displayed in a predetermined size in the center of the background image until the image processing parameter is given. Synthesize the data.

  The video operation input unit 104 is a touch panel provided so as to cover the entire display surface of the video output unit 110. The video operation input unit 104 is operated by the user of the karaoke apparatus 100, generates operation content data corresponding to the operation, and supplies the operation content data to the audio processing parameter generation unit 105 and the video processing parameter generation unit 106. In the present embodiment, the entire display surface of the video output unit 110 is the video operation input 104, but only a part of the display surface may be the video operation input unit 104. In addition, the video operation input unit 104 may be a part of a device different from the video output unit 110 (for example, a remote controller of the karaoke device 100). However, when the video operation input unit 104 is a part of the remote control of the karaoke apparatus 100, the video processing unit 108 outputs the video not only to the video output unit 110 but also to the remote control. In this case, the images displayed on the image output unit 110 and the image operation input unit 104 are the same.

  Specific examples of operations performed on the video operation input unit 104 include a pinch-in operation and a pinch-out operation (hereinafter referred to as a pinch-in / out operation) and a touch operation. The pinch-in operation is an operation of touching the video operation input unit 104 with two fingers and moving each finger so as to narrow the interval between the two fingers. The pinch-out operation is an operation in which the user touches and moves to widen the interval between two fingers. When a pinch-in / out operation is performed, the video operation input unit 104 generates operation content data indicating the pinch-in / out operation, and supplies the operation content data to the audio processing parameter generation unit 105 and the video processing parameter generation unit 106. In this operation content data, from the coordinates at which the touch of each of the two fingers involved in the operation is started (for example, the coordinates with the upper left corner of the display area of the video operation input unit 104 as the origin, the same applies hereinafter). Data representing the trajectory up to the coordinates where the finger is released is included.

  The touch operation is an operation in which the user touches the video operation input unit 104 with one finger, traces the screen of the video operation input unit 104 while touching, and releases the finger from the screen at an arbitrary position. By this touch operation, the video operation input unit 104 generates operation content data indicating the touch operation (that is, data representing a locus from the coordinate where the touch is started to the coordinate where the finger is released), and the operation content data is voiced. The processing parameter generation unit 105 and the video processing parameter generation unit 106 are provided. In this embodiment, the case where the touch operation is performed on the avatar and the case where the touch operation is performed on the background image are distinguished, and the latter is particularly referred to as “flick operation” or “swipe operation”. Call it.

  The audio processing parameter generation unit 105 generates an audio processing parameter from the operation content data given from the video operation input unit 104 and gives it to the audio processing unit 102. The audio processing parameter generation unit 105 stores in advance an audio processing table shown in FIG. 2 and a reverb change table shown in FIG. As shown in FIG. 2, the audio processing table is associated with each of various operations performed on the video operation input unit 104, and represents data for processing to be performed on the audio when the operation is performed. Is stored. On the other hand, as shown in FIG. 4, the reverb change table stores data indicating the contents of the reverb in association with an identifier (number in FIG. 4) indicating the type of reverb applied to the voice. Note that the voice processing parameter generation unit 105 stores the identifier of the currently applied reverb. The voice processing parameter generation unit 105 generates a voice processing parameter representing the processing content according to the operation content from the operation content indicated by the operation content data given from the video operation input unit 104 and the stored content of the audio processing table. To do. If the operation content is a flick operation or a swipe operation, the audio processing parameter generation unit 105 further generates an audio processing parameter by referring to the reverb change table.

  The video processing parameter generation unit 106 generates a video processing parameter from the operation content data given from the video operation input unit 104 and gives it to the video processing unit 108. In the video processing parameter generation unit 106, a video processing table shown in FIG. 3 and a background video change table shown in FIG. 5 are stored in advance. As shown in FIG. 3, the video processing table is associated with each of various operations performed on the video operation input unit 104, and represents data to be processed on the video when the operation is performed. Is stored. On the other hand, as shown in FIG. 5, the background video change table stores data indicating the content of the background video in association with an identifier (number in FIG. 5) indicating the type of background video applied to the video. . Note that the video processing parameter generation unit 106 stores the identifier of the currently displayed background video. Also, the reverb and the background image corresponding to the same number in FIGS. 4 and 5 correspond to each other. The video processing parameter generation unit 106 generates video processing parameters representing the processing content corresponding to the operation content from the operation content indicated by the operation content data given from the video operation input unit 104 and the stored content of the video processing table. To do. If the operation content is a flick operation or a swipe operation, the video processing parameter generation unit 106 further generates a video processing parameter with reference to the background video change table.

  FIG. 6 is a flowchart showing a flow of processing in which the audio processing parameter generation unit 105 generates audio processing parameters, and FIG. 7 is a flowchart showing a flow of processing in which the video processing parameter generation unit 106 generates video processing parameters. is there. As apparent from comparing FIG. 6 and FIG. 7, step SA103, step SA105, and step SA106 correspond to step SB103, step SB105, and step SB106, and the contents are the same except for these steps. The same step number is used in FIGS. The audio processing parameter generation unit 105 and the video processing parameter 106 each start generation of the audio processing parameter and the video processing parameter when the operation content data is received from the video operation input unit 104. Each of the audio processing parameter generation unit 105 and the video processing parameter generation unit 106 determines whether or not the operation indicated by the operation content data is a pinch-in / out operation (step SA101). Specifically, each of the audio processing parameter generation unit 105 and the video processing parameter generation unit 106 performs an operation indicating a pinch-in / out operation when the operation content data includes data representing the trajectory of two fingers. The content data is determined. When the determination result in step SA101 is “Yes”, the audio processing parameter generation unit 105 and the video processing parameter generation unit 106 indicate that at least one of the coordinates of the two fingers that started the touch indicated by the operation content data is It is determined whether or not it is within the area corresponding to the avatar (step SA102). If the determination result in step SA102 is “Yes”, the audio processing parameter generation unit 105 refers to the audio processing table and adjusts the volume of the audio signal according to the interval between two fingers after the pinch-in / out operation. A voice processing parameter to be increased or decreased is generated (step SA103). Similarly, when the determination result in step 102 is “Yes”, the video processing parameter generation unit 106 refers to the video processing table and sets the size according to the interval between two fingers after the pinch-in / out operation. The avatar is reduced / enlarged, and image processing parameters for synthesizing the avatar and the background image before the pinch-in / out operation are generated (step SB103). On the other hand, if the determination result in step SA102 is “No”, the audio processing parameter generation unit 105 and the video processing parameter 106 do not generate the audio processing parameter and the video processing parameter, and step SA103 and The generation process ends without executing step SB103.

  Next, a case where the audio processing parameter generation unit 105 and the video processing parameter generation unit 106 receive operation content data indicating a touch operation will be described. 6 and 7 are also used for the description in this case. Each of the audio processing parameter generation unit 105 and the video processing parameter generation unit 106 determines whether or not the operation represented by the operation content data is a touch operation on the avatar when the determination result in step SA101 is “No”. (Step SA104). Specifically, each of the audio processing parameter generation unit 105 and the video processing parameter generation unit 106, when the coordinates of the touch start position indicated by the received operation content data are coordinates in the area corresponding to the avatar, It is determined that the touch operation is performed on the avatar. If the coordinates are outside the area, it is determined that the operation is a flick operation or a swipe operation.

  When the determination result in step SA104 is “Yes”, the video processing parameter generation unit 106 refers to the stored content of the video processing table, and moves the avatar display position to instruct to combine with the background video. A machining parameter is generated (step SB105). Specifically, the video processing parameter generation unit 106 generates a video processing parameter that instructs to move the avatar to the touch end position indicated by the received operation content data and combine it with the background video. On the other hand, when the determination result in step SA104 is “Yes”, the voice processing parameter generation unit 105 refers to the stored content of the voice processing table and refers to the sound image corresponding to the avatar (the voice signal associated with the avatar. A sound processing parameter is generated that instructs to move the localization position of the sound image corresponding to the sound signal indicated by the identifier, that is, the sound image corresponding to the singer's singing sound corresponding to the avatar (step SA105). Specifically, the voice processing parameter generation unit 105 sets a voice processing parameter that instructs to adjust the output balance of each voice output unit 103 so that the sound image is localized at the touch end position indicated by the received operation content data. Generate.

  On the other hand, if the determination result in step SA104 is “No”, the video processing parameter generation unit 106 changes the background video and determines that the received operation content data is a flick operation or a swipe operation, Video processing parameters for instructing synthesis are generated (step SB106). Specifically, the video processing parameter generation unit 106 first determines that an instruction to change the background video is given with reference to the video processing table, and an identifier indicating the background video at that time (ie, the video processing parameter generation unit 106). The identifier of the background video to be changed is identified from the stored identifier in the video change table. Then, the video processing parameter generation unit 106 updates the identifier stored in the video processing parameter to the identifier specified in the above-described manner, and sets the video processing parameter for instructing to combine the background video indicated by the identifier with the avatar. Generate. For example, when the background video is a video arbitrarily set by the producer or user of the karaoke apparatus 100 (that is, the initial background video corresponding to number 1 in FIG. 5), the video processing parameter generation unit 106 performs a flick operation or a swipe operation. Thus, a video processing parameter for instructing to select the background video as the music hall (that is, the background video corresponding to the number 2 in FIG. 5) is generated. Thereafter, when the flick operation or the swipe operation is performed again, the video processing parameter generation unit 106 instructs to select the background video as the bathroom (that is, the background video corresponding to number 3 in FIG. 5). Is generated. When the flick operation or swipe operation is performed one after another, the image processing parameter generation unit 106 generates image processing parameters for selecting the background image one after another, but the background image change table shown in FIG. 5 before the flick operation or swipe operation is performed. In the case of the background video corresponding to the last number, video processing parameters for selecting the initial background video corresponding to the number 1 are generated by flicking or swiping. In this way, the video processing parameter generation unit 106 generates video processing parameters for selecting background images one after another when the flick operation or swipe operation is repeated. Note that there is no change in the avatar before and after the flick operation or swipe operation, and there is no change in the position where the background image is synthesized.

Similarly, if the determination result in step SA104 is “No”, the voice processing parameter generation unit 105 determines that the received operation content data is a flick operation or a swipe operation, and applies the reverb to be added to the voice corresponding to the avatar. A voice processing parameter for instructing the change is generated (step SA106). Specifically, the speech processing parameter generation unit 105 first determines that reverb change has been instructed with reference to the speech processing table, and stores an identifier indicating the reverb at that time (ie, stored in the speech processing parameter generation unit 105). Identifier) and the stored contents of the reverb change table specify the reverb identifier of the change destination. Then, the voice processing parameter generation unit 105 updates the identifier stored in the voice identifier to the identifier specified in the above manner, and voice processing that instructs to add the reverb indicated by the identifier to the voice corresponding to the avatar. Generate parameters. For example, when the reverb is not applied (that is, when the reverb corresponding to the number 1 is not applied in FIG. 4), the sound processing parameter generation unit 105 determines whether the sound is being listened to in the music hall by the flick operation or the swipe operation. A speech processing parameter is generated that indicates that reverb is applied (that is, reverb corresponding to number 2 is applied in FIG. 4). Thereafter, when a flick operation or a swipe operation is performed again, the audio processing parameter generation unit 105 applies reverberation as if listening to the audio in the bathroom (that is, applies reverberation corresponding to number 3 in FIG. 4). A voice processing parameter for instructing this is generated. The voice processing parameter generation unit 105 generates voice processing parameters for selecting reverb one after another when flick operation or swipe operation is performed one after another. Before the flick operation or swipe operation, the voice processing parameter generation unit 105 generates an end of the reverb change table shown in FIG. In the case of the reverb corresponding to the number 1, a voice processing parameter that does not perform the reverb corresponding to the number 1 is generated by a flick operation or a swipe operation. In this way, the speech processing parameter generation unit 105 generates speech processing parameters for selecting reverb one after another regardless of how many times the flick operation or swipe operation is repeated.
The above is the configuration of the karaoke apparatus 100 of the present embodiment.

(B: Operation)
Hereinafter, the operation of the karaoke apparatus 100 will be described by taking as an example a case where the user of the karaoke apparatus 100 is one person. When the user turns on the power of the karaoke apparatus 100, the video output unit 110 displays a menu screen that prompts the user to select a karaoke song. A user who visually recognizes the menu screen can select a karaoke song to be sung and input an instruction to start performance by operating a remote controller or the like. When a karaoke song is selected, the background image acquisition unit 109 acquires background image data corresponding to the karaoke song from the host computer and acquires accompaniment sound data of the karaoke song from the host computer. When an instruction to start performance is input, the background video acquisition unit 109 starts outputting the background video data acquired from the host computer to the video processing unit 108, and the audio of the accompaniment sound data acquired from the host computer. Output (not shown in FIG. 1) to the processing unit 102 is started. The volume of the accompaniment sound is adjusted, for example, by moving a slider on the remote controller of the karaoke apparatus 100 up and down.

  When the output of the background video data from the background video acquisition unit 109 to the background processing unit 108 is started in this way, the video output unit 110 starts displaying the video corresponding to the karaoke song to be sung, and the audio output unit In 103, sound emission of the accompaniment sound of the karaoke song is started. When the singer grasps that the singing start timing has been reached from the video and the accompaniment sound, the singer starts the singing with the voice input unit 101. As described above, since the user does not start singing until the singing start point is reached, from the start of outputting the background video data to the background processing unit 108 from the background video acquisition unit 109 to the singing start timing, The corresponding avatar is not displayed on the video output unit 110. When the user starts to sing, an audio signal representing the singing voice is given to the audio processing unit 102 and to the video generation unit 107. The video generation unit 107 starts outputting the avatar data representing the avatar corresponding to the user to the video processing unit 108. The video processing unit 108 synthesizes the given avatar data so as to be positioned at the center coordinates of the background video data, and provides the resultant data to the video output unit 110 as video output data. Therefore, the video output unit 110 displays a video in which an avatar is arranged at the center of the background video as shown in FIG. Further, the output balance is adjusted so that the user's singing voice output from each audio output unit 103 can be heard from the position of the avatar displayed on the video output unit 110.

  When the display of the avatar is started as described above, the user of the karaoke apparatus 100 can perform an operation on the avatar or an operation on the background image on the video operation input unit 104. In addition, as operation with respect to an avatar, the touch operation which moves the position of the pinch in / out operation mentioned above and an avatar is mentioned. Hereinafter, an operation performed by the karaoke apparatus 100 when these operations are performed will be described.

(B-1: Operation when an avatar is operated)
As described above, operations for an avatar include pinch-in / out operations and touch operations.

(B-1-1: Operation when a pinch-in / out operation is performed on an avatar)
First, an operation when a pinch-in / out operation for an avatar is performed will be described.
When a pinch-in / out operation is performed on the avatar displayed on the video output unit 110, as described above, the video operation input unit 104 generates operation content data indicating the pinch-in / out operation, and performs voice processing. This is given to the parameter generation unit 105 and the video processing parameter generation unit 106. As described above, the audio processing parameter generation unit 105 generates the audio processing parameter that instructs to decrease / increase the volume of the audio corresponding to the avatar according to the operation amount of the pinch-in / out operation, and generates the video processing parameter The unit 106 generates a video processing parameter that instructs to reduce / enlarge the size of the avatar.

  The voice processing parameter is given to the voice processing unit 102, and the voice processing unit 102 reduces the volume of the voice signal given from the voice input unit 101 corresponding to the avatar for which the pinch-in / out operation is performed according to the voice processing parameter. / The processing to increase is performed, and the processed audio signal is given to the audio output unit 103. The audio output unit 103 emits a given audio signal. As for the relationship between the avatar pinch-in / out operation and the decrease / increase of the volume of the audio signal, the volume of the audio signal is increased by the pinch-in operation, and the volume of the audio signal is decreased by the pinch-out operation. Also good. However, such an association between the operation content and the processing content is not desirable because it does not match the user's intuition. It is desirable to associate the operation content with the processing content such that the volume of the audio signal is reduced by the pinch-in operation and the volume of the audio signal is increased by the pinch-out operation.

  The video processing unit 108 performs reduction / enlargement processing on the avatar data of the avatar that has been pinched in / out according to the video processing parameters, and synthesizes the background video data and the processed avatar data into output video data. . Then, the output video data is given to the video output unit 110. The video output unit 110 displays the given output video data to the user. Therefore, the entire avatar displayed on the video output unit 110 and the video operation input unit 104 is reduced / enlarged. Note that the entire avatar may be enlarged by a pinch-in operation on the entire avatar, and the entire avatar may be reduced by a pinch-out operation. However, the correspondence between the operation content and the processing content matches the user's intuition. It is not desirable because it is not.

  FIG. 9 is a diagram illustrating an image displayed after a pinch-out operation is performed on the video operation input unit 104. As apparent from a comparison between FIG. 8 and FIG. 9, when a pinch-out operation is performed on the entire avatar, the entire avatar is displayed in an enlarged manner. Furthermore, the sound corresponding to the enlarged avatar is emitted from the sound output unit 103 with an increased volume. Thus, according to the present embodiment, the synchronization between video and audio is not lost.

(B-1-2: Operation when a touch operation is performed on an avatar)
Next, an operation when a touch operation for moving the position of the avatar is performed will be described.
When a touch operation for moving the position of the avatar is performed, the video processing parameter generation unit 106 generates a video processing parameter for moving the avatar to the touch operation end position. The video processing parameter generation unit 106 gives the video processing parameters to the video processing unit 108, and the video processing unit 108 synthesizes the avatar data and the background video data based on the video processing parameters and outputs them to the video output unit 110 as output video data. give. The video output unit 110 displays a video represented by the given output video data. Therefore, the avatar displayed on the video output unit 110 moves to the touch operation end position for the avatar (see FIG. 10). When the avatar displayed on the video operation input unit 104 is moved outside the screen, the video processing parameter generation unit 106 generates a video processing parameter that does not display the avatar, and the audio processing parameter generation unit 105 A voice processing parameter that instructs to mute the voice corresponding to the avatar is generated. Therefore, the avatar is not displayed on the video output unit 110 and the video operation input unit 104, and the user cannot hear the voice corresponding to the avatar.

  When a touch operation for moving the position of the avatar is performed, as described above, the sound processing parameter generation unit 105 displays a sound image corresponding to the avatar (that is, a sound image of the user's singing sound) at the touch operation end position. A voice processing parameter for instructing adjustment of the output balance of the voice output unit 103 is generated so as to be localized. The voice processing parameter generation unit 105 gives the voice processing parameter to the voice processing unit 102, and the voice processing unit 102 performs the above processing on the voice signal given from the voice input unit 101 corresponding to the avatar on which the touch operation is performed. Applied to the audio output unit 103. The audio output unit 103 emits a sound represented by a given audio signal. Therefore, the user obtains an audible feeling as if the attacker is singing from the touch operation end position in the touch operation on the avatar. In this case, the synchronization between the video and audio is not lost.

(B-2: Operation when an operation is performed on the background image)
Finally, the operation when an operation is performed on the background video will be described.
When a flick operation or swipe operation is performed on the background video displayed on the video operation input unit 104, the video processing parameter generation unit 106, as described above, causes the background video to have a different background from that before the flick operation or swipe operation. Video processing parameters for instructing video selection are generated. The video processing parameter generation unit 106 gives the video processing parameter to the video processing unit 108, and the video processing unit 108 uses the same avatar data as before the operation and background video data different from that before the operation based on the video processing parameter. Combining them and giving them to the video output unit 110 as output video data. In the output video data, the position of the avatar data relative to the background video data is the same before and after the operation. The video output unit 110 displays a video represented by the given output video data. Therefore, in the video output unit 110, a background video different from that before the flick operation or swipe operation is displayed, but the same avatar as before the operation is displayed at the same position as before the operation with respect to the background video.

On the other hand, the audio processing parameter generation unit 105 generates an audio processing parameter that instructs to apply reverberation corresponding to the background video after the flick operation or swipe operation to the audio signal. The voice processing parameter generation unit 105 gives the voice processing parameter to the voice processing unit 102, reverberates the voice signal based on the voice processing parameter, and gives the voice signal to the voice output unit 103. The audio output unit 103 emits a sound represented by a given audio signal. For this reason, the audio output unit 103 emits a sound that has undergone reverberation corresponding to the background image after the flick operation or swipe operation. In this case, the synchronization between the video and audio is not lost.
The above is the operation of the karaoke apparatus 100 of the present embodiment.

  As described above, according to the karaoke apparatus 100 of the present embodiment, by operating the video operation input unit 104, both audio processing parameters and video processing parameters corresponding to the operation are generated. There is no loss of audio synchronization. Furthermore, the user of the karaoke apparatus 100 can visually understand the characteristics of the audio and can perform the audio processing visually by operating the video displayed on the video operation input unit 104.

(C: Modification)
Various modifications can be considered for the karaoke apparatus 100 of the present embodiment. The modification is shown below.

(1) The pinch-in / out operation or touch operation on the avatar, the flick operation or swipe operation on the background image may be an operation on an operator (for example, a slider) provided on the remote controller of the karaoke apparatus 100. The video operation input unit 104 may be replaced with a pointing device such as a mouse. Further, by operating a pointing device simultaneously with pressing a specific button attached to the keyboard or remote controller of the karaoke apparatus 100, the same operation as a pinch-in / out operation or the like may be performed for the first time. The pinch-in / out operation for the avatar is not limited to the operation performed in the area corresponding to the avatar of the video operation input unit 104, and may be performed in the vicinity of the area corresponding to the avatar. That is, even if the pinch-in / out operation is performed in the vicinity of the area corresponding to the avatar, not in the area corresponding to the avatar of the video operation input unit 104, it is the same as the pinch-in / out operation performed in the area corresponding to the avatar. In this manner, the operation content data may be generated. Further, a motion capture device or a voice identification device may be used as the video operation input unit 104. In addition, the video operation input unit 104 may generate operation content data by performing an operation of pinching and extending an avatar in the pseudo 3D space.

(2) Each of a larger number of users than the number of voice input units 101 provided in the karaoke apparatus 100 may sing one by one using the voice input units 101 alternately. In this case, if the number of the voice input unit 101 is used as the voice signal identifier, the avatar displayed on the video output unit 110 is not switched even if the user who uses the voice input unit 101 is changed. That is, in this case, as many video avatars as the audio input units 101 provided in the karaoke apparatus 100 are displayed on the video output unit 110. On the other hand, if waveform data or feature data is used as an audio signal identifier, a different avatar can be displayed for each user. In addition, waveform data or feature value data is used as an audio signal identifier even when different avatars are displayed when one user inputs audio to one audio input unit 101 and when multiple users input audio simultaneously. What is necessary is just to use it, and when displaying the same avatar, the number of the audio | voice input part 101 should just be used as an audio | voice signal identifier.

(3) Lyric subtitle display data and background video data are delivered from the host computer to the background video acquisition unit 109 via the network, and the background video acquisition unit 109 supplies the lyrics subtitle display data and the background video data to the video processing unit 108. The video processing unit 108 may process the avatar data under the control of the video processing parameters, synthesize the processed avatar data, the background video data, and the lyrics subtitle display data, and give the combined image to the video processing unit 108. Specifically, relative coordinates (coordinates with the upper left corner of the image representing the avatar as the origin) indicating the position of the mouth position of the avatar at the arrangement position are determined in advance for each avatar. The video processing unit 108 calculates the coordinates of the mouth of the avatar for synthesizing the lyrics subtitle display data (coordinates having the upper left corner of the display surface as the origin) from the coordinates of the arrangement position of the avatar with respect to the background video. Then, the video processing unit 108 positions the lyrics subtitle display data in the vicinity of the avatar's mouth, and synthesizes the video output data so that the lyrics are ejected and displayed at the avatar's mouth. In this case, the video processing unit 108 synthesizes the avatar data, the background video data, and the lyrics subtitle display data so that the lyrics are displayed in front of the avatar so that the lyrics can be seen.

(4) When a shadow or an afterimage is given to the avatar, an echo effect may be applied to the audio signal corresponding to the avatar according to an operation on the shadow or the afterimage. Specifically, the echo effect is applied in association with the data representing the operation content of touching the avatar displayed on the video operation input unit 104 and releasing the touched finger from the video operation input unit 104 without moving it. Is stored in the voice processing table. Further, data representing that a shadow is displayed around the avatar is stored in the video processing table in association with data representing the same operation. In such an aspect, when the avatar is touched and moved away from the video operation input unit 104 without moving the touched finger, a shadow is displayed around the avatar and an echo effect is applied to the sound. In addition, data indicating that the echo effect is weakly or strongly applied is associated with data indicating the pinch-in / out operation performed on the area corresponding to the shadow around the avatar displayed on the video operation input unit 104. Store in the voice processing table. Further, data representing that the shadow range around the avatar is reduced / enlarged is stored in the video processing table in association with the data representing the same operation. When a pinch-in / out operation is performed on the shadow around the avatar displayed on the video operation input unit 104, the echo effect applied to the sound corresponding to the avatar becomes weak or strong, and the shadow range around the avatar is reduced / Expanding. In addition, the type of echo to be applied changes according to the data representing the operation content of touching the shadow around the avatar and releasing the touched finger from the video operation input unit 104 without moving it. The data to be represented may be stored in the voice processing table, and the data representing the change of the avatar's clothes may be stored in the video processing table corresponding to the data representing the same operation.

(5) Touching the avatar displayed on the video operation input unit 104 and associating it with data representing the operation content of performing the operation of moving away from the video operation input unit 104 twice without moving the touched finger. Data indicating that the surroundings of the avatar are blurred may be stored in the video processing table, and data indicating that the overlapping singing effect is applied may be stored in the audio processing table in association with the data indicating the same operation. In this aspect, when the avatar displayed on the video operation input unit 104 is touched and the touched finger is moved away from the video operation input unit 104 without moving, the surroundings of the avatar are blurred and displayed. And the overlapping singing effect is applied to the voice. The operation performed twice in succession may be a pinch-in / out operation or a touch operation. Further, data indicating that the overlapping singing effect is weakly or strongly applied corresponding to the data indicating the pinch-in / out operation performed on the area corresponding to the blur around the avatar displayed on the video operation input unit 104. May be stored in the audio processing table, and data indicating that the blur range around the avatar is reduced / enlarged may be stored in the video processing table corresponding to the data indicating the same operation. When a pinch-in / out operation is performed on the blur around the avatar displayed on the video operation input unit 104, the overlapping singing effect applied to the voice corresponding to the avatar becomes weak or strong, and the range of the blur around the avatar is reduced. /Expanding. Further, instead of the data indicating that the periphery of the avatar bleeds, data indicating that the same avatar is displayed with a plurality of positions shifted may be stored in the video processing table. Furthermore, corresponding to the data representing the pinch-in / out operation performed on the area corresponding to the overlap around the avatar displayed on the video operation input unit 104, the data representing the weak or strong overlapping singing effect is sounded. It may be stored in the processing table, and data indicating that the number of overlaps around the avatar is decreased / increased may be stored in the video processing table corresponding to the data indicating the same operation.

(6) Although the avatar has a shape imitating a human, the avatar may have a shape other than the shape imitating a human. For example, the shape may be a building, a car, a tree, a mountain, or a musical instrument. The shape may be completely unrelated to the input voice. The avatar data may be video data showing the singer himself photographed with a camera (not shown). The video displayed on the video output unit 110 is not limited to a composite video of a background video and an avatar, and may be a video consisting of only an avatar.

(7) The karaoke apparatus 100 may have a function of scoring the input voice. Furthermore, the video processing unit 108 may take an aspect in which the entire avatar is changed according to the scoring result. More specifically, three types of avatar data corresponding to the same audio signal identifier are stored in a storage device (not shown). The avatar data represents the same character's joy, normality, and joyfulness, and is associated with the score of the scoring result of the karaoke apparatus 100. For example, when the scoring is 100 out of 100, the avatar data indicating the state of joy corresponds between the scoring results of 70 and 100, and the normal state is between 31 and 69. The represented avatar data is supported, and the avatar data representing the state of crawl is supported between 0 and 30 scoring results. The video generation unit 107 reads avatar data corresponding to the scoring result from the storage device, and gives the avatar data to the video processing unit 108. The video processing unit 108 synthesizes the avatar data and the background video, and provides the video output unit 110 as video output data. The video output unit 110 displays the video output data to the user. Therefore, an avatar corresponding to the scoring result is displayed on the video output unit 110. However, if the karaoke apparatus 100 is not scoring or is scoring, the video generation unit 107 gives the avatar data representing the normal state to the video processing unit 108.

(8) The face of the avatar may be a morphing display with the face of the original artist of the song sung by the singer. As described above, when the karaoke apparatus 100 has a scoring function, the face of the avatar may be closer to the face of the original artist as the scoring score is higher. More specifically, the image data of the face of the original artist of the song sung by the singer is stored in a storage device (not shown). After scoring, the video generation unit 107 reads the image data of the original artist's face from the storage device, performs the following calculation according to the image data of the avatar's face and the scoring result, and outputs it to the video processing unit 108. . That is, if the scoring score is C out of 100 points, pixel A in the original artist's face image data and pixel B at the same position as pixel A in the original avatar face image data are used. The sum of the pixel A multiplied by C / 100 and the pixel B multiplied by 1-C / 100 is set as the pixel value corresponding to the pixel A after scoring.

Second Embodiment
In the first embodiment, a pinch-in / out operation and a touch operation are performed on the entire avatar displayed on the video operation input unit 104. On the other hand, in this embodiment, each operation such as a pinch-in / out operation is performed by designating a part of the avatar instead of the entire avatar. This is a point where the present embodiment and the first embodiment are significantly different. Since the hardware configuration of the karaoke apparatus of the present embodiment is the same as that of the karaoke apparatus 100 of the first embodiment, the detailed description will be omitted with reference to FIG.

  The operation for the avatar part is designated in this way with the operation for designating the part to be manipulated (in this embodiment, the operation of touching the same position in the region corresponding to the part to be manipulated three times in succession). Operation (pinch-in / out and touch operation) for the selected part. For this reason, the operation content data regarding the operation on the avatar part includes data representing an operation for designating the operation target part, which is different from the operation content data for the operation on the entire avatar. FIG. 11 is a flowchart showing a flow of processing in which the audio processing parameter generation unit 105 generates audio processing parameters, and FIG. 12 is a flowchart showing a flow of processing in which the video processing parameter generation unit 106 generates video processing parameters. is there. As apparent from a comparison between FIG. 11 and FIG. 12, Step SA203 and Step SA204 correspond to Step SB203 and Step SB204, respectively, and the contents are the same except for these steps. Use the same step number. The audio processing parameter generation unit 105 and the video processing parameter 106 each start generation of the audio processing parameter and the video processing parameter when the operation content data is received from the video operation input unit 104. The audio processing parameter generation unit 105 and the video processing parameter generation unit 106 determine whether or not the operation content data received from the video operation input unit 104 includes data representing an operation for designating a region to be operated (step). SA201). If the determination result in step SA201 is “No”, the audio processing parameter generation unit 105 and the video processing parameter generation unit 106 end the generation processing without generating the audio processing parameter and the video processing parameter. On the other hand, if the determination result in step SA201 is “Yes”, the audio processing parameter generation unit 105 and the video processing parameter generation unit 106 determine whether or not the operation content data includes data representing a pinch-in / out operation. Is determined (step SA202). When the determination result in step SA202 is “Yes”, the audio processing parameter generation unit 105 and the video processing parameter generation unit 106 generate the audio processing parameter and the video processing parameter, respectively (step SA203 and step SB203). Here, various modes can be considered as to what kind of operation plays the role of designating the region to be operated. For example, an aspect in which the above-mentioned role is played in the operation of touching the region in the region corresponding to the avatar part three times in succession, or a long press in the region corresponding to the part of the avatar (that is, the same position for a predetermined time or more) A mode in which the above customer discount is assigned to the operation of keeping touching with a) is conceivable. In the present embodiment, the former aspect is adopted.

  The video processing parameter generation unit 106 stores a video processing table that stores data representing processing content to be applied to the avatar when the operation is performed in association with data representing each operation content of the operation on each part of the avatar. The background video change table (see FIG. 5) and the avatar part processing table shown in FIG. 13 are stored in advance. The video processing table stores data indicating that the part is processed in association with the data representing the operation content for the part of the avatar, and the avatar part processing table is associated with the avatar part. Data representing specific processing contents to be applied to the part is stored. For example, the video processing table stores data indicating that the avatar part is reduced / enlarged in association with data indicating a pinch-in / out operation for the avatar part. The video processing table stores data indicating that a part other than the body is moved in association with data indicating a touch operation on a part other than the body of the avatar, and corresponds to data indicating a touch operation on the body. In addition, data representing the change of the avatar's costume is stored. The video processing parameter generation unit 106 stores an identifier representing the avatar costume being displayed on the video output unit 110.

  When the determination result in step SA202 is “Yes”, the video processing parameter generation unit 106 instructs to reduce / enlarge the part according to the data representing the operation content included in the received operation content data. A machining parameter is generated (step SB203). On the other hand, if the determination result in step SA202 is “No”, the video processing parameter generation unit 106 determines that the received operation content data is data indicating a touch operation, and the video processing parameter according to the touch operation. Is generated (step SB204). More specifically, if the touch position indicated by the operation content data is other than the body of the avatar, the video processing parameter generation unit 106 assumes that the touch operation is for a body other than the body of the avatar, and the display position of the part corresponding to the touch position Video processing parameters for instructing movement of the image are generated. If the touch position indicated by the operation content data is the body of the avatar, the video specifying the update destination costume from the stored content of the video processing table and the identifier of the costume of the avatar at that time, and an instruction to update to the costume Generate machining parameters.

  The voice processing parameter generation unit 105 stores data indicating the processing content to be applied to the voice corresponding to the avatar when the operation is performed in association with the data indicating the operation content of the operation on each part of the avatar. An audio processing table, a reverb change table (see FIG. 4), and a compressor processing table shown in FIG. 14 are stored in advance. In the present embodiment, data representing the decrease / increase of the parameter that defines the processing content by the compressor in association with the pinch-in / out operation for the avatar part is stored in the voice processing table.

  Processing with a compressor means that if the volume of the sound represented by the audio signal to be processed exceeds a preset threshold (threshold value), the excess volume is compressed at a set ratio (ratio) and the set release time By releasing, it means reducing the maximum value of the volume that fluctuates with the progress of karaoke songs. As parameters that define the content of processing by the compressor, there are three parameters of gain, attack time, and knee in addition to the threshold value, ratio, and release time. The gain is a value indicating increase / decrease in the volume of the sound, and the attack time is the above ratio after the volume of the sound represented by the audio signal to be processed exceeds the threshold value (that is, after the compression of the volume starts). It is the time to reach. Knee is a value indicating the degree of compression until the ratio is reached in the vicinity of the threshold value.

  The compressor processing table of the present embodiment stores data indicating which items of the compressor are to be adjusted in association with the avatar part. For example, in the compressor processing table shown in FIG. 14, a gain is associated with the avatar's face, a knee is associated with the avatar's body, and an attack time is associated with the avatar's right hand. It has been. Furthermore, in the compressor processing table shown in FIG. 14, the release time is associated with the avatar's left hand, the threshold value is associated with the avatar's right foot, and the ratio is associated with the avatar's left foot. It is attached.

  The voice processing table of the present embodiment stores data indicating that processing other than the compressor is performed on the voice for the avatar in association with data indicating a touch operation on the avatar part. For example, data representing a touch operation on the avatar's right hand is associated with data representing raising the bass range of the avatar, and data representing a touch operation on the avatar's left hand includes the treble range of the avatar. Data representing lifting is associated, and data representing a touch operation on the body of the avatar is associated with data representing changing the acoustic effect according to the updated costume by the touch operation. And so on. For example, if the costume after the change is Japanese clothes, the change represents an enka-like sound effect.

  When the determination result in step SA202 is “Yes”, the voice processing parameter generation unit 105 gives a voice that instructs to apply a compressor effect corresponding to the operation content to the voice processing table and the voice signal corresponding to the avatar. Processing parameters are generated (step SA203). For example, if the operation is a pinch-in / out operation on the face of the avatar, the audio processing parameter generation unit 105 generates an audio processing parameter that instructs to decrease / increase the gain of the audio signal. If the operation is a pinch-in / out operation on the body of the avatar, the audio processing parameter generation unit 105 generates an audio processing parameter that instructs to decrease / increase the knee. If the operation is a pinch-in / out operation on the right hand of the avatar, the voice processing parameter generation unit 105 generates a voice processing parameter that instructs to shorten / extend the attack time. If the operation is a pinch-in / out operation on the left hand of the avatar, the audio processing parameter generation unit 105 generates an audio processing parameter that instructs to shorten / extend the release time. When the avatar's right foot is a pinch-in / out operation, the voice processing parameter generation unit 105 generates a voice processing parameter for instructing a decrease / increase of the threshold value. If the operation is a pinch-in / out operation on the left foot of the avatar, the voice processing parameter generation unit 105 generates a voice processing parameter that instructs to decrease / increase the ratio. On the other hand, when the determination result in step SA202 is “No”, the voice processing parameter generation unit 105 refers to the stored contents of the voice processing table and instructs the voice processing to adjust the acoustic effect corresponding to the part. A parameter is generated (step SA204). For example, when the touch operation is performed on the body of the avatar and the costume updated by the touch operation is a kimono, the sound processing parameter generation unit 105 performs the sound processing instructing to give an enka sound effect. For example, a parameter is generated.

Note that the avatar part size / length reduction / enlargement and the compressor parameter reduction / increase / shortening / extension may be reversed for pinch-in / out operations on the avatar part. Furthermore, the part of the avatar that has performed the pinch-in / out operation and the part of the avatar that is reduced / enlarged by the video operation input unit 104 may not match. For example, when a pinch-in operation is performed on the body of the avatar, the length of the right hand of the avatar may be increased and the gain of the audio signal may be increased. However, such association between the operation content and the processing content is not necessarily desirable because it does not match the user's intuition. In addition, the relationship between the operation for the avatar part and the parameters of the compressor does not necessarily have the relationship described above. For example, when a pinch-in / out operation is performed on the avatar body, the avatar body may be reduced / expanded, and the attack time may be shortened / extended. These are realized by rewriting a part corresponding to the avatar operation part of the avatar part processing table of FIG. 13 and the compressor processing table of FIG.
The above is the configuration of the karaoke apparatus 100 of the present embodiment.

  Due to such a configuration, even in the karaoke apparatus of the present embodiment, even if some processing is performed on the video (avatar), the video and audio are not synchronized. Although the second embodiment of the present invention has been described above, this embodiment may be modified as follows.

(1) As a modification of the present embodiment, the aspects (1) to (8) of the modification of the first embodiment may be taken. As a modification of the present embodiment, a mode in which sound and video are processed by an operation on the entire avatar as in the first embodiment may be employed.

(2) When waveform data or feature value data is used as an audio signal identifier, avatar part data associated with the waveform or feature of the audio signal is prepared, and the karaoke apparatus 100 responds to the input audio signal. Alternatively, each part may be selected from the part data of the avatar and combined into one avatar.

(3) When a touch operation is performed on the avatar's face, the shape of the avatar's face may be changed so that the compressor effect is strongly applied to the audio signal. In this aspect, the face of the avatar may change from a circle, which is a shape imitating the shape of a human face, to a square or a hexagon. Moreover, the whole avatar may change by performing a touch operation on the face of the avatar. The change of the entire avatar is a change of the avatar's facial expression, hairstyle, or physique, so that the entire avatar may be changed to, for example, kelp other than humans.

<Third Embodiment>
FIG. 15 is a block diagram showing a configuration of a karaoke apparatus 200 according to the third embodiment of the present invention. 15, the same components as those in FIG. 1 are denoted by the same reference numerals as those in FIG. As apparent from a comparison between FIG. 1 and FIG. 15, the configuration of the karaoke apparatus 200 includes a feature amount analysis unit 204 instead of the video operation input unit 104, the audio processing parameter generation unit 105, and the video processing parameter generation unit 106. This is different from the configuration of the karaoke apparatus 100. The feature amount analysis unit 204 is also a software module that is realized by the CPU of the karaoke apparatus 200 according to a control program (not shown in FIG. 15) stored in the karaoke apparatus 200. Below, it demonstrates centering on the feature-value analysis part 204 which shows the characteristic of this embodiment notably.

  FIG. 16 is a flowchart illustrating a flow of processing in which the feature amount analysis unit 204 generates a video processing parameter. The feature amount analysis unit 204 divides the audio signal received from the audio input unit 101 by a predetermined time (step SA301), and performs audio analysis on each of the audio signals for the predetermined time obtained in this way. (Step SA302), and a video processing parameter reflecting the result of the voice analysis is generated (step SA303). The voice analysis performed by the feature amount analysis unit 204 has three modes: a mode for analyzing only the volume of the voice signal, a mode for analyzing the volume and dynamic range of the voice signal, and a mode for analyzing only the spectral envelope. . The dynamics range is the ratio between the maximum and minimum values of the audio signal volume, and the spectral envelope is, for example, a Fourier transform of the audio signal for a certain time, taking the logarithm of the base 10 and multiplying by 20 Is.

  When the feature amount analysis unit 204 is configured to analyze only the sound volume of the audio signal, the feature amount analysis unit 204 measures the average sound volume for each predetermined time described above. The feature amount analysis unit 204 stores in advance an average volume table (not shown) that associates the average volume with the overall size of the avatar. The average volume table of the present embodiment stores data indicating that the avatar is increased as the average volume increases, but conversely, data indicating that the avatar is decreased as the average volume increases. An aspect may be sufficient. The feature amount analysis unit 204 generates a video processing parameter that determines the size of the entire avatar from the stored content of the average volume table and the measured average volume every time the predetermined time elapses, and the video processing parameter is processed into the video processing parameter. Part 108 is given.

  The video processing unit 108 generates video output data from the background video given from the background video acquisition unit 109, the avatar given from the video generation unit 107, and the video processing parameters given from the feature amount analysis unit 204. This is given to the video output unit 110. The video output unit 110 displays a video represented by the video output data. The video processing unit 108 is provided with a plurality of background video data from the background video acquisition unit 109. The background video data arbitrarily selected by the producer or user of the karaoke apparatus 100 (or randomly selected by the video processing unit 108). Background video data) is used for synthesis with the processed avatar data. Further, the video processing unit 108 performs synthesis so that the avatar data is positioned at the center of the background video data. The voice input unit 101 may collect ambient noise as voice even when the singer is not singing. For this reason, in this aspect, when the volume of the audio signal given from the audio input unit 101 is equal to or lower than a preset threshold value, a process for generating a video processing parameter for hiding the avatar and giving it to the video processing unit 108 is performed. You may make the feature-value analysis part 204 perform. Further, when the volume of the audio signal given from the audio input unit 101 is equal to or lower than a preset threshold value, the same effect can be obtained even if the video generation unit 107 does not output the avatar data corresponding to the audio signal. .

  Next, when the feature amount analysis unit 204 is configured to analyze the volume and dynamics range of the audio signal, the feature amount analysis unit 204 sets the average volume per predetermined time for the audio signal given from the audio input unit 101. And the dynamics range is measured, and the video processing parameter that changes the avatar's face color to warm color is generated. More specifically, the feature amount analysis unit 204 determines that the singer's enthusiasm is high when the average volume of the audio signal exceeds the threshold and the average dynamics range of the audio signal is smaller than the threshold. Therefore, in order to make the avatar also sing, the feature amount analysis unit 204 generates a video processing parameter for changing the avatar's face color to a warm color, and gives the video processing parameter to the video processing unit 108. Thereafter, the feature amount analysis unit 204 is the same as the aspect in which only the average sound volume is analyzed. The feature amount analysis unit 204 stores the average volume table, and changes the avatar's face color to a warm color and changes the avatar's size according to the average volume of the audio signal to the feature amount analysis unit. 204 may be generated.

  Finally, in a case where the feature amount analysis unit 204 analyzes only the spectrum envelope, the feature amount analysis unit 204 measures a spectrum envelope for each predetermined time set in advance for the speech signal given from the speech input unit 101. And the feature-value analysis part 204 reads the spectrum envelope and voice avatar of the original artist of the song which the singer sings, and is stored in the memory | storage device which is not illustrated, The spectrum envelope of the voice of the singer who measured and measured For example, the comparison is performed using a correlation function. The feature amount analysis unit 204 morphs the avatar corresponding to the input audio signal and the original artist's avatar according to the result of comparison and collation between the measured spectral envelope and the original spectral envelope (that is, the value of the correlation function). Such video processing parameters are generated and given to the video processing unit 108. Thereafter, the feature amount analysis unit 204 is the same as the aspect in which only the average sound volume is analyzed. In the aspect in which the feature amount analysis unit 204 analyzes only the spectrum envelope, when the measured spectrum envelope and the original spectrum envelope completely match, the avatar displayed on the video output unit 110 becomes the avatar of the original artist.

  Also in this aspect, the above-mentioned average volume table is stored in the feature amount analysis unit 204, and in addition to the morphing according to the similarity with the original spectrum envelope, the image processing for changing the size of the avatar according to the average volume The parameter may be generated by the feature amount analysis unit 204, and further, an image processing parameter for changing the avatar's face color to a warm color according to the enthusiasm may be generated. In addition, the feature quantity analysis unit 204 is connected to the host computer via a network, and the spectral envelope and avatar of the original artist's voice of the song sung by the singer are not from a storage device (not shown) but from the host computer. A mode of being directly distributed to the feature amount analysis unit 204 may be adopted.

The audio processing unit 102 distributes the audio signal given from the audio input unit 101 to the audio output unit 103 and outputs it. This point is the same as in the first embodiment, but the voice processing unit 102 of the present embodiment is different from the first embodiment in that no voice processing parameters are given and the voice signal is not processed. Different.
The above is the configuration of the karaoke apparatus 200 of the present embodiment.

  Because of such a configuration, in the karaoke apparatus 200 of the present embodiment, video processing parameters corresponding to the input audio signal are generated, so that the synchronization between video and audio is not lost. Furthermore, since the avatar changes according to the audio signal, the user of the karaoke apparatus 200 can visually understand the change in audio from the video displayed on the video output unit 110. Although the karaoke apparatus 200 according to the third embodiment of the present invention has been described above, this embodiment may be modified as follows.

(1) As a modification of the present embodiment, the modes (2), (3), (6), (7) and (8) of the modification of the first embodiment may be taken.

(2) When generating video processing parameters that change the avatar's face color to warm, measure the average body temperature of the singer at a preset time using thermography without judging the degree of enthusiasm based on volume and dynamics range And the average body temperature may be used for judgment of the enthusiasm. If the singer is singing, the body temperature should go up. Therefore, if the average body temperature of the singer measured by the thermography exceeds a preset threshold, the feature value generation unit 204 generates a video processing parameter for changing the avatar's face color to a warm color, and provides the video processing unit 108 with the video processing parameter. . In addition, you may take the aspect which combined the aspect which measures average body temperature with the aspect which measures the average sound volume and average dynamics range of said audio | voice signal. In this aspect, when the average volume of the audio signal exceeds the threshold value, the average dynamics range of the audio signal is smaller than the threshold value, and the average body temperature of the singer exceeds the threshold value, the feature amount analysis unit 204 displays the avatar face color. An image processing parameter for generating a warm color is generated. Of course, the feature analysis unit 204 may store the average volume table and generate the video processing parameter for changing the size of the avatar as described above according to the average volume of the audio signal.

(3) The present embodiment may be combined with the first embodiment. In this case, the video processing parameter generation unit 106 and the feature amount analysis unit 204 may be present together, or may not be present. In a mode in which both do not coexist, the feature amount analysis unit 204 does not exist, and the video processing parameter generation unit 106 receives an audio signal from the audio input unit 101 and generates a video processing parameter. Furthermore, the video processing parameter generation unit 106 has the function of the feature amount analysis unit 204.

<Other embodiments>
In the first to third embodiments, the application example of the present invention to the karaoke apparatus has been described. However, the present invention may be applied to a signal processing apparatus other than a karaoke apparatus, and specifically as follows.

(1) The sound collected by the voice input unit 101 is not limited to a human voice, and may be a sound of a musical instrument, or may be an accompaniment sound in which various sounds such as a voice and a musical instrument are mixed. . Even for sounds other than these human sounds, the video generation unit 107 can generate avatars corresponding to each sound, visually understand the characteristics of each sound, and visually process each sound. .

(2) In the first to third embodiments, the case where the audio signal representing the sound collected by the audio input unit 101 is input in real time has been described. However, it is also possible to apply a sound processing corresponding to the latter to the former for the sound and the background video that are recorded in advance and recorded in synchronization with each other. An embodiment in which processing is applied to the latter may be used. In this aspect, the audio input unit 101 and the background video acquisition unit 109 can be omitted. Furthermore, since the audio output unit 103 and the video output unit 110 can be replaced by, for example, an output device of a personal computer, this aspect can be realized only by software. That is, this aspect is suitable for moving image editing performed on a personal computer.

(3) The present invention may be applied to a portable device such as a cellular phone. For example, when the present invention is applied to a mobile phone, the voice input unit 101 corresponds to a microphone of the mobile phone, and the voice output unit 103 corresponds to a speaker of the mobile phone. The background video data acquired by the background video acquisition unit 109 corresponds to, for example, video data stored in the mobile phone, and the video output unit 110 and the video operation input unit 104 correspond to the screen of the mobile phone. The sound collected by the microphone can be processed according to the stored video, or the stored video can be processed according to the sound collected by the microphone. Further, the present invention may be applied to stationary and portable game machines. In this case, the controller or touch panel of the game machine corresponds to the video device input unit 104. The touch panel is operated with a touch pen or stylus. Further, when the game machine is compatible with a motion capture device, the motion sensor corresponds to the video operation input unit 104.

(4) The present invention may be applied to a server device for an application service provider (ASP). Specifically, the server device having the audio processing unit 102, the audio processing parameter generation unit 105, the video processing parameter generation unit 106, the video generation unit 107, and the video processing unit 108 of the first embodiment or the second embodiment, or the first A server apparatus having the audio processing unit 102, the video generation unit 107, the video processing unit 108, and the feature amount analysis unit 204 of the third embodiment is connected to an electric communication line such as the Internet. The former server device receives an audio signal, background video data, and operation content data via an electric communication line, performs audio processing and video processing, and transmits the processed audio signal and output video data via the electric communication line. Send. The latter server device receives an audio signal and background video data via an electric communication line, performs audio processing and video processing, and transmits the processed audio signal and output video data via the electric communication line. In this aspect, the user transmits each signal (in the case of the former server device, operation content data) to the server device via the electric communication line, which represents audio and video that are synchronously reproduced from the personal computer. By receiving and playing back the audio signal and video data processed by the server device via the network, the same video processing and audio processing as in the above embodiments can be performed.

  Also, in this aspect, the user causes the server device to process the video data and audio signal transmitted by the user and the video data and audio signal transmitted from the other user to the server device, and the processed video data and audio signal are processed. You may receive it. For example, by synthesizing one's avatar and another user's avatar to generate video data as if both avatars are duet, an audio signal representing one's singing voice and an audio signal representing the other user's singing voice And generating a sound signal representing sound as if duet.

(5) In the first to third embodiments, the sound output unit 103 is a sound emitting device that emits sound represented by the sound signal given from the sound processing unit 102. However, for example, an amplifier or a mixer is used. An external device other than the sound emitting device may be used. When the karaoke device 100 is used for recording, the external device may be a recording device. Furthermore, the aspect which uses the said external device and a sound emission device together may be sufficient.

DESCRIPTION OF SYMBOLS 100,200 ... Karaoke apparatus, 101 ... Audio | voice input part, 102 ... Audio | voice processing part, 103 ... Audio | voice output part, 104 ... Image | video operation input part, 105 ... Audio | voice processing parameter generation part, 106 ... Video | video processing parameter generation part, 107 ... Image generation unit 108... Image processing unit 109. Background image acquisition unit 110. Image output unit 204.

Claims (3)

A parameter generation unit that generates a video processing parameter representing processing content for a video to be displayed on the display device according to an input sound signal;
A video processing unit that performs processing based on the video processing parameters on video data representing video to be displayed on the display device in synchronization with the sound represented by the input sound signal, and supplies the processed video data to the display device When,
A signal processing apparatus comprising: A video generation unit that generates video data to be processed by the video processing unit based on the input sound signal;
A video operation input unit for inputting an operation on a video displayed on the display device;
A sound processing unit that performs processing based on a sound processing parameter and outputs the input sound signal;
With
The parameter generator is
In place of the process of generating the video processing parameter according to the input sound signal, the process of generating the video processing parameter according to the operation input to the video operation input unit, or the input sound signal and the video operation input The processing for generating the video processing parameter according to an operation input to the unit is executed, and the sound processing parameter is generated according to the operation input to the video operation input unit. The signal processing apparatus according to 1. The input sound signal includes a sound signal representing a user's singing voice,
The video data is data representing each part of an avatar that imitates the user,
Different sound effects are assigned to each part of the avatar,
The parameter generation unit generates a video processing parameter representing processing content for the part for each part of the avatar operated by the video operation input unit, and adjusts an acoustic effect corresponding to the operated part. The signal processing apparatus according to claim 2, wherein a sound processing parameter to be generated is generated.

Images