JP2016102982A - Karaoke system, program, karaoke voice reproduction method, and voice input processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a karaoke device that can prevent the satisfaction level of a user from deteriorating due to the output delay of singing voice caused by a variety of processing in a communication terminal device.SOLUTION: A communication terminal device constituting a karaoke system is provided with a novel audio library 706 on a platform 70. The new library 706 outputs voice data from a microphone to an HDMI (R) port 402 in a path-through manner, generates duplicate voice data to supply it to a karaoke application 80.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a karaoke system and a voice input processing device that executes voice input processing.

  Conventionally, communication terminal devices such as game terminal devices, set-top boxes (hereinafter referred to as “STB”), PCs (personal computers), USB (Universal Serial Bus) and HDMI (registered trademark (High-Definition Multimedia Interface)) A home karaoke network system that allows a user to perform karaoke by connecting a microphone (hereinafter referred to as a “microphone”) and a television receiver to the interface is realized.

  Specifically, in this type of home karaoke network system, a communication terminal device such as an STB transmits accompaniment sound data including a guide melody for karaoke (hereinafter, “accompaniment sound”) from a server device connected to the network. Karaoke music data (hereinafter referred to as “music data”) including data and lyrics data indicating the lyrics of the music.

  Then, the communication terminal device outputs the accompaniment sound from the television receiver connected to the interface, displays the lyrics on the television receiver while synchronizing with the accompaniment sound, and the user follows the accompaniment sound and the like on the microphone. The user's voice collected in this way (hereinafter referred to as “singing voice”) is also output from the television receiver.

  In particular, the communication terminal device converts the collected singing sound into sound data of a predetermined format (hereinafter referred to as “dry sound data”) with a microphone, and performs various audio effect processing (for the dry sound data). For example, reverberation processing, effect processing such as key change and voice change) is performed, and effect sound such as reverberation sound is output from the television receiver together with singing sound and accompaniment sound.

  On the other hand, in this kind of karaoke system, it is required to synchronize the display timing of lyrics and the output timing of accompaniment sounds, singing voices, etc. in order to let the user sing without a sense of incongruity. Therefore, recently, the delay time amount of the lyrics display in the television receiver is acquired in advance, and the output timing of the singing sound, the accompaniment sound, and the effect sound corresponding to the dry sound data output from the microphone for the delay time. There has been proposed a system and method for executing delay control and synchronizing the display timing of lyrics with the output timing of singing voice, accompaniment sound, effect voice, and the like (for example, Patent Document 1).

JP 2010-276949 A

  However, in the method described in Patent Document 1, although the display timing and the audio output timing are synchronized, the output delay of the singing voice generated due to the processing in the communication terminal device is eliminated. In other words, it is not sufficient as a method for improving user satisfaction when performing karaoke at home.

  The present invention has been made in order to solve the above-described problems, and its purpose is to prevent a decrease in user satisfaction due to output delay of singing voice based on various processes in the communication terminal device. It is to provide a karaoke system that can be used.

  In order to solve the above-described problem, the present invention provides a receiving unit that receives an instruction input from a user who selects at least one piece of music data from a plurality of pieces of music data, an acquisition unit that acquires the selected music data, and an external unit. An interface that is connected to a device and outputs predetermined data that has been input to the external device in accordance with a pre-defined procedure, and when a voice uttered by the user is input, the sound is converted into audio data in a predetermined format. While receiving the input of the voice data as input voice data from the voice input means that outputs, the input voice data that has been input is duplicated to generate duplicate voice data, and any of the input voice data and the duplicate voice data Input / output management means for outputting one of the audio data to the interface, and input / output from the input / output management means. Audio data different from the audio data output to the interface is acquired, predetermined processing is executed based on the acquired audio data and the acquired music data, and the processed predetermined data is transmitted to the interface. And a data processing means for outputting the data.

  With this configuration, the present invention, for example, outputs audio data output from audio input means such as a microphone (that is, dry sound data (hereinafter referred to as “dry sound data”)) to the interface, or In addition, since it is possible to directly output the duplicate voice data obtained by duplicating the dry sound to the interface, it is possible to prevent the user's satisfaction from being lowered due to the output delay of the singing voice based on various processes in the communication terminal device. .

  In general, a karaoke system used as a consumer device such as a home television receiver uses voice (ie, dry sound) obtained by A / D (Analog to Digital) conversion of a user's singing voice collected by a microphone. ) Data (hereinafter referred to as “dry sound data”), audio effect processing based on the dry sound data is executed, and the generated effect sound is combined with the singing sound and accompaniment sound and connected to the interface For example, a configuration is employed in which various types of data synthesized are output to a display device with audio output such as a television receiver.

  On the other hand, the communication terminal device performs effect processing on the digital data of the dry sound data, so if a platform configuration having a plurality of layers is used to drive a karaoke application, each layer The delay associated with buffering at the time increases.

  Therefore, the present invention outputs singing sound based on various processes even in a platform having a plurality of layers for driving a karaoke application by directly outputting dry sound data or its duplicate sound data to an interface. A delay can be prevented and a decrease in user satisfaction can be prevented.

  The voice input processing device and the like according to the present invention can prevent a decrease in user satisfaction due to a delay in the output of singing voice due to processing in the communication terminal device.

It is a system configuration figure showing the composition of one embodiment in the karaoke network system concerning the present invention. It is a block diagram which shows the structure of the communication terminal device of one Embodiment. It is a figure which shows an example of the platform structure of the communication terminal device of one Embodiment. It is a figure which shows an example of the karaoke background image containing the evaluation data displayed in the karaoke system of one Embodiment. It is a flowchart performed when performing a karaoke in the communication terminal device of one Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiment described below has a karaoke system for home use, has a karaoke system for home use, provides music data for karaoke via a network, and allows a user to perform karaoke. It is an embodiment in the case of applying a device, a program, a karaoke voice reproduction method, and a voice input processing device.

[1] Karaoke Network System First, a schematic configuration of the karaoke network system 1 of the present embodiment will be described with reference to FIG. FIG. 1 is a system configuration diagram showing the configuration of the karaoke network system 1 in the present embodiment. In order to prevent the figure from becoming complicated, only a part of the karaoke systems 10 is shown in FIG. That is, in the actual karaoke network system 1, there are more karaoke systems 10 than displayed in FIG.

  The karaoke network system 1 according to the present embodiment is an apparatus (for example, a device installed in a facility such as each home of the user via the network 20 for karaoke music data corresponding to the music that the user desires to sing). This is a system for providing karaoke to the user by displaying the lyrics of the music while reproducing the accompaniment sound data included in the music data by the apparatus.

  Specifically, the karaoke network system 1 of the present embodiment is installed in each home or facility, and manages a plurality of karaoke systems 10, a network 20, and music data that are used by a user. And a music providing server device 30 that provides corresponding music data in response to the request.

  The network 20 is configured, for example, by connecting a public telephone network including a mobile telephone network and an IP (Internet Protocol) network. However, the configuration of the network 20 is not limited to this.

  The karaoke system 10 of the present embodiment includes an interface in a predetermined standard such as USB or HDMI (registered trademark), a communication terminal device 40 that performs processing for causing a user to perform karaoke, and the communication terminal device 40. A television receiver 50 connected to an output port (hereinafter also referred to as a “digital output port”) for outputting audio and images such as an HDMI (registered trademark) port, and outputting the audio and images, and the communication terminal device 40. The microphone 60 is connected to an input / output port for data transmission, such as a USB port, and collects the singing voice uttered by the user.

  The communication terminal device 40 receives broadcast signals such as cable television broadcast, satellite broadcast, terrestrial television broadcast (including digital broadcast and analog broadcast), IP broadcast (for example, broadband VOD), and can be viewed on the television receiver 50. A terminal device such as a set-top box, a personal computer, a game device (including a portable game device) or a portable information terminal device (for example, a smartphone or a tablet-type terminal device) that converts it into a simple signal, and is connected to the network 20 In addition, it is configured to execute processing for allowing the user to perform karaoke while interlocking with the music providing server device 30.

Specifically, the communication terminal device 40 includes predetermined hardware such as a CPU and a memory, a Linux (registered trademark) operating system (OS), and an application driven on the operating system.
(1) a first layer configured by an interface including various kernels that function as device drivers of external devices such as the television receiver 50 and the microphone 60 and manage input / output of each data;
(2) On the first layer, a library such as a virtual machine (program (group) having a specific function having general versatility), an operating system excluding the kernel, and an API (application program interface) are configured. A second layer that controls the interface in the layer;
(3) It is configured on the second layer and is driven on the API and library or operating system (OS) to execute an application for karaoke (hereinafter referred to as “karaoke application”) 80 and other applications. The third layer,
It is driven on the basis of a platform having a hierarchical structure having a function to execute karaoke.

Then, the communication terminal device 40 receives (1) an instruction input from the user as each function for executing karaoke, and responds to the music desired by the user from the music providing server device 30 based on the acquired music data. A music data acquisition function for acquiring music data to be
(2) Based on the acquired music data, a music data reproduction display control function for reproducing accompaniment sounds on the television receiver 50 and displaying the lyrics of the music on the television receiver 50;
(3) When the voice uttered by the user according to the accompaniment sound and the lyrics is input, the dry sound data converted into the digital signal by the microphone 60 is acquired, and the effect processing based on the dry sound data is executed. Effects processing function to generate audio data,
(4) Based on the accompaniment sound data included in the dry sound data and the music data, the user's singing ability is evaluated, image data indicating the evaluation result is generated as evaluation data, and the generated evaluation data is used as the television receiver 50. Scoring function to be displayed at
It has the structure which realizes.

  In particular, when receiving the input of the dry sound data output from the microphone 60 as the input sound data, the communication terminal device 40 duplicates the input sound data that has been input to generate duplicate sound data. It has a configuration in which any one of the audio data is directly output to the television receiver 50 via the device driver and the input / output port without performing processing based on the application (that is, the karaoke application 80). ing.

  Normally, when an application such as a karaoke application is realized by the above platform structure, the karaoke including the sub-layer of each layer from the first layer that controls the actual hardware and manages the input / output of data is performed. Various processes are executed in each layer up to the highest layer (hereinafter also referred to as “application layer”) for executing the application.

  Therefore, the input audio data output from the microphone 60, that is, the dry sound data, is buffered in each layer of the platform including the sublayer, and therefore a delay of about several tens of msec each time. Will occur.

  In particular, the dry sound data output from the microphone 60 is supplied to a karaoke app that is driven by the highest application layer to perform effect processing, and further, sound data subjected to effect processing (hereinafter referred to as “effect sound data”). In addition, while synthesizing the dry sound data used therefor, predetermined processing is executed in each layer including the sub-layer, and the data is supplied from the lowest device driver to the television receiver 50.

  Therefore, even if transmission in HDMI (registered trademark) and delay in the television receiver 50 are ignored, the effect sound data and dry sound data are delayed by about 110 to 130 msec from the user's singing and utterance timing. Will be supplied to the machine 50.

  Therefore, even if the user sings based on the accompaniment sound output from the television receiver 50, the accompaniment sound and the singing sound are shifted due to the delayed output of the singing sound of the user. Listening to it often causes a sense of incongruity, and in many cases the scoring function and other attached functions related to karaoke cannot be fully exhibited.

  Therefore, in the communication terminal device 40 of the present embodiment, the input dry sound data is duplicated, and the data used for application data processing and the data output to the television receiver 50 through the minimum necessary route are separately provided. By doing so, the karaoke application can be accurately executed while reducing the output delay of the dry sound data, thereby realizing a karaoke that can prevent a decrease in user satisfaction.

  In particular, the communication terminal apparatus 40 provides a new audio library (hereinafter referred to as “new library”) in the above platform (specifically, the second layer), and the microphone 60 is stored in the new library 706. The dry sound data input from the USB port 401, the Kernel 701, and the Alsa-Lib 702 is duplicated, and the dry sound data used for the duplication or the dry sound data (duplicate audio data) generated by the duplication is transmitted to the TV. It directly outputs to the receiver 50 (pass-through output), and supplies the other dry sound data to the karaoke application 80.

  In the present embodiment, the dry sound data used for copying may be directly output to the television receiver 50 via the interface, and the copied dry sound data may be used for the karaoke application 80.

  The television receiver 50 is a consumer television receiver having at least an HDMI (registered trademark) port as a digital input port. In particular, the television receiver 50 includes a display element such as a liquid crystal panel or a plasma display panel, a drive circuit that drives the display element, a speaker, and the like, and is supplied via, for example, an HDMI (registered trademark) port of the communication terminal device 40. Based on the audio data and the image data, the singing voice, the accompaniment sound and the effect voice are output, and the lyrics and the evaluation data are displayed.

  The microphone 60 is, for example, a USB microphone, collects a user's singing voice, A / D converts the singing voice, generates voice data (that is, dry sound data), and outputs the voice data to the communication terminal device 40. To do.

  The music providing server device 30 includes: (1) music data that can be provided to the karaoke system 10; (2) a music ID for identifying the music; and (3) a genre indicating the music name of the music and the genre to which the music belongs. A request received from the communication terminal device 40 via the network 20 has a music database (hereinafter referred to as “database DB”) 300 in which information and music attribute information including artist names are recorded in association with each other. (Hereinafter referred to as “music data acquisition request”), the corresponding music data can be distributed to the communication terminal device 40.

  In particular, the music providing server device 30 searches the music DB 300 based on the music data acquisition request, generates a list of songs hit by the search (singer name list, song name list, etc.), and the corresponding communication terminal device. 40, the music data of the music selected by the user according to the list is read from the music DB 300 and distributed to the communication terminal device 40.

  The music data recorded in the music DB 300 includes lyrics data corresponding to the lyrics of the music and accompaniment sound data for karaoke. The accompaniment sound data is configured in, for example, a MIDI (Musical Instrument Digital Interface) format, and includes data corresponding to each part such as guide melody data and guitar.

[2] Communication Terminal Device Next, the configuration of the communication terminal device 40 of the present embodiment will be described with reference to FIGS. 2 is a block diagram illustrating a configuration of the communication terminal device 40 according to the present embodiment, and FIG. 3 is a diagram illustrating a platform configuration of the communication terminal device 40. FIG. 4 is a diagram illustrating a display example of a karaoke background image including evaluation data.

  As shown in FIG. 2, the communication terminal device 40 of the present embodiment includes an interface unit 400 that mediates exchange of data with an external device, a communication control unit 410, a terminal management control unit 470, and a program management control unit 480. A program management control unit that executes various applications including a display control unit 440 that generates drawing data to be supplied to the television receiver 50, an operation unit 460, a terminal management control unit 470, and a karaoke application 80 while interlocking with each other. 480. The units are connected to each other via a bus B so that data can be exchanged between the components.

  The operation unit 460 can also be realized by a remote controller. In this case, the operation unit 460 is provided with an infrared light receiving unit or a Zig Bee receiver, and a command command based on the received signal is provided. May be supplied to the bus B.

  The interface unit 400 is provided with a USB port 401 and an HDMI (registered trademark) port 402, and performs data conversion and other processing in accordance with USB and HDMI (registered trademark) standards, and includes a device driver and the like. An input / output control chip corresponding to each interface is provided.

  In particular, as illustrated in FIG. 3, the interface unit 400 according to the present embodiment includes a kernel 701 such as Linux (registered trademark) and an Alsa-Lib 702 that function as device drivers on the platform 70. Configure one layer.

  Since Kernel 701 and Alsa-Lib 702 have the same functions as those of the related art, their details are omitted. Further, for example, the interface unit 400 including the Kernel 701 and the Alsa-Lib 702 of this embodiment constitutes a first layer (interface) of the present invention.

  In addition, for example, a microphone 60 is connected to the USB port 401, and dry sound data output from the microphone 60 is output to the bus B. A television receiver 50 is connected to the HDMI (registered trademark) port 402, and image data and audio data are output to the television receiver 50.

  The interface unit 400 constitutes an interface of the present invention together with a part of the program management control unit 480, for example.

  The communication control unit 410 is a predetermined network interface, is connected to the network 20 by wire or wirelessly, and exchanges data with the music providing server device 30 via the network 20.

  The display control unit 440 is linked with the terminal management control unit 470 and the program management control unit 480, and based on the lyrics data and the evaluation data supplied from the program management control unit 480, the television receiver 50 uses the background image for karaoke. Above, drawing data for displaying an image showing lyrics and evaluation data is generated and supplied to the television receiver 50 via the HDMI (registered trademark) port 402.

  The recording unit 450 includes a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read Only Memory), an HDD (Hard Disc Drive), or an SSD (Solid State Drive), and the recording area 450 includes an operating system and a recording area thereof. (OS) A program recording unit 451 for recording a program for forming a platform, an application recording unit 452 for recording a karaoke application 80 and other applications, a music data recording unit 453, and a work area. RAM 454.

  The program recording unit 451 records various data used to construct the platform 70 for controlling the interface unit 400 having the USB port 401 and the HDMI (registered trademark) port 402 and operating the program management control unit 480. . The platform 70 of this embodiment will be described later.

  In the music data recording unit 453, music data acquired (downloaded) by the communication terminal device 40 from the music providing server device 30 is recorded in association with the corresponding music ID and music attribute information, and has already been acquired. It is made available when performing karaoke using data.

  The terminal management control unit 470 is mainly composed of a central processing unit (CPU) and includes various input / output ports such as a key input port and a display control port, and is recorded in the program recording unit 451 and the application recording unit 452. By executing the various programs and applications, the overall functions of the communication terminal device 40 are controlled in an integrated manner.

  The program management control unit 480 includes a CPU that is the same as or independent from the terminal management control unit 470. The program management control unit 480 executes a program such as an operating system (OS) recorded in the program recording unit 451 while interlocking with the RAM 454 and the interface unit 400 under the control of the terminal management control unit 470. 70 is constructed, and the karaoke application 80 is executed in an environment provided by the platform 70 to realize each function relating to karaoke.

  In particular, since the program management control unit 480 constructs the platform 70 together with hardware such as a CPU and the interface unit 400 while the communication terminal device 40 is being driven, a Linux (registered trademark) operating system (OS) such as Android is used. And a program execution unit 481 for executing other necessary programs, and an application execution unit 482 for executing the karaoke application 80 driven on the operating system.

  The program execution unit 481 is linked with the interface unit 400 at a predetermined timing such as when the power is turned on, and from the program recording unit 451 to a Linux (registered trademark) operating system (OS) such as an Android or other necessary The program is read to form a second layer on the platform 70 shown in FIG.

  In particular, the program execution unit 481 forms an HAL 703, an Audio-Finger 704, and an Open SLES 705 by executing an operating system (OS), and executes functions of the OS excluding some OS functions such as a device driver.

  Specifically, the program execution unit 481 controls the interface unit 400 (that is, a device driver), and based on the operation of the operation unit 460 or the control of the application execution unit 482, dry sound data, music data, and other various types. Data input / output is controlled, and necessary processing is executed according to various types of processing in the application execution unit 482.

  Further, as a characteristic configuration of the present embodiment, the program execution unit 481 includes a Kernel 701 and an Alsa-Lib 702 (that is, a device driver) included in the interface unit 400 from the microphone 60 via the USB port 401 on the second layer. ) Via the Alsa-Lib 702 and Kernel 701, the dry sound data supplied via the) is directly passed through to the HDMI (registered trademark) port 402, and the dry sound data is duplicated. , A new library 706 to be supplied to the karaoke application 80 is formed and its execution is controlled.

  That is, the program execution unit 481 directly inputs the dry sound data input to the USB port 401 to the HDMI (registered trademark) port 402 without interposing the HAL 703, the audio-ringer 704, and the open SLES 705 in the platform 70 of the present embodiment. It has a configuration to supply.

  In addition, by having such a configuration, in the present embodiment, it is possible to effectively suppress a delay from the user's utterance timing until the sound data is output to the television receiver 50, and the duplicate audio data. Since the effect process and scoring process based on can be performed accurately, the user's satisfaction when performing karaoke can be improved.

  In particular, as a result of measuring the delay time of the singing voice output from the television receiver 50 in the platform 70 of the present embodiment, the delay time is 75 to 85 msec, which is the delay time 110 to 110 in the conventional karaoke system. It has been confirmed that the delay time of the singing voice of about 22 to 42% can be shortened compared with the delay time of about 130 msec.

  In addition, since the time (period) required for generating the duplicate audio data can be made smaller than the sum of the buffering times in each sublayer of the platform 70, the program execution unit 481 can duplicate the dry sound data instead of the dry sound data. Audio data may be output to the HDMI (registered trademark) port 402 via the Alsa-Lib 702 and the Kernel 701.

  Further, the HAL 703, the Audio-Fringer 704, and the Open SLES 705 constituting the platform 70 are the same as the conventional functions, and thus the details thereof are omitted. For example, the HAL 703, the audio-ringer 704, the OpenSLES 705, and the new library 706 of this embodiment constitute a second layer of the present invention, and the new library 706 constitutes an input / output management means of the present invention.

  The application execution unit 482 executes the karaoke application 80 on the running OS, thereby acquiring a music data acquisition management unit 483 that acquires music data, an effect processing unit 484 that executes effect processing, and the music data. A karaoke processing unit 485 for reproducing and displaying music data and a scoring processing unit 486 for scoring karaoke are realized.

  In particular, the application execution unit 482 operates in conjunction with the program execution unit 481 to read the karaoke app 80 from the application recording unit 452 at a predetermined timing such as when the power is turned on, and to 3 layers are formed to realize each of the above parts. For example, the karaoke application 80 of the present embodiment constitutes the third layer of the present invention.

  The music data acquisition management unit 483 executes processing for acquiring music data from the music providing server device 30 via the communication control unit 410, and the music data corresponding to the music data acquired from the music providing server device 30 is used. Are recorded in the music data recording unit 453.

  The effect processing unit 484 executes audio effect processing based on the duplicate audio data supplied from the new library 706.

  For example, the effect processing unit 484 generates reverberation sound based on the replicated sound data, generates effect sound data obtained by convolving the reverberation sound, and transmits the HDMI (registered trademark) port 402 via the platform 70. Output to.

  Note that the delay of the effect sound does not affect the satisfaction level of the user, so even when the effect sound data is output to the HDMI (registered trademark) port 402 via the HAL 703, the Audio-Finger 704, and the Open SLES 705. Sound output delay due to buffering in these sublayers is not a problem.

  The karaoke processing unit 485 acquires music data from the music data recording unit 453 based on the operation of the operation unit 460, and converts the accompaniment sound data in MIDI format included in the acquired music data into sound data (hereinafter simply referred to as “sound data”). Converted to “accompaniment sound data”) and supplied to the HDMI (registered trademark) port 402 via the platform 70.

  At this time, in the platform 70, the accompaniment sound data is buffered in each sub-layer, and the Kernel 701 or Alsa-Lib 702 synthesizes the accompaniment sound data with the dry sound data to generate an HDMI (registered trademark) port. 402.

  In addition, the karaoke processing unit 485 supplies lyrics data included in the music data to the HDMI (registered trademark) port 402 via the platform 70 in conjunction with the display control unit 440.

  The scoring processing unit 486 performs scoring of the user's singing ability based on the duplicate audio data supplied from the new library 706 and the accompaniment data included in the music data acquired by the karaoke processing unit 485, and the scoring result Is generated in conjunction with the display control unit 440, and drawing data for displaying a predetermined image is generated based on the generated evaluation data.

  Specifically, the scoring processing unit 486 compares the scale state of the singing voice corresponding to the duplicated voice data with the scale, tone value, and rest included in the MIDI accompaniment sound data, and compares them. Evaluation data indicating the result is generated.

  In particular, as shown in FIG. 4, the scoring processing unit 486 sets the scale in the vertical axis direction on the staff and sets the time axis in the horizontal axis direction, and the scale, pitch value, and rest indicated by the accompaniment sound data. Evaluation data (also referred to as piano roll data) expressing the displacement state of the singing voice with a solid line is generated while expressing the note by a predetermined rectangular shape on the staff (see FIG. 4).

  That is, as shown in FIG. 4, the scoring processing unit 486 uses a scale or the like indicated by an accompaniment sound on a moving image having lyrics images or other background images for karaoke (hereinafter referred to as “karaoke background images”). Drawing data including an image indicating how much the pitch of the singing voice is shifted is generated as evaluation data.

  Then, the scoring processing unit 486 supplies the evaluation data to the HDMI (registered trademark) port 402 via the platform 70 in cooperation with the display control unit 440. Specifically, the scoring processing unit 486 controls the display control unit 440 to generate drawing data based on the lyrics data and the evaluation data supplied from the karaoke processing unit 485, and the generated drawing data is displayed. , And supplied to the HDMI (registered trademark) port 402 via the platform 70.

  When the television receiver 50 receives the drawing data output from the HDMI (registered trademark) port 402, the television receiver 50 displays an image as illustrated in FIG.

[3] Operation of Karaoke System Next, the operation of the communication terminal device 40 of the karaoke system 10 of the present embodiment will be described with reference to FIG. In addition, FIG. 5 is a flowchart which shows the process performed with the communication terminal device 40 of this embodiment.

  In this operation, it is assumed that the microphone 60 is connected to the USB port 401 of the communication terminal device 40 in advance, and the television receiver 50 is connected to the HDMI (registered trademark) port 402.

  In this operation, the operating system (OS) is executed by the program execution unit 481 in conjunction with the interface unit 400, and the karaoke application 80 is executed by the application execution unit 482, so that the platform 70 shown in FIG. It is assumed that the application execution unit 482 (specifically, the music data acquisition management unit 483) has already acquired the music data desired by the user and recorded in the music data recording unit 453.

  First, when the karaoke processing unit 485 detects a predetermined input operation to the operation unit 460 by the user (step S101), the karaoke processing unit 485 reads out the corresponding music data from the music data recording unit 453, and reproduces the music and the predetermined image. Start (step S102).

  Specifically, the karaoke processing unit 485 generates audio data in a predetermined format based on the MIDI accompaniment sound data included in the read music data, and HDMI (registered trademark) of the audio data generated via the platform 70. ) Start output to port 402.

  Then, the karaoke processing unit 485 generates data for drawing for displaying an image of the karaoke background image and the lyrics data included in the music data on the karaoke background image in cooperation with the display control unit 440. Then, output of drawing data generated via the platform 70 to the HDMI (registered trademark) port 402 is started.

  Further, when the reproduction of the music is started, each unit repeatedly executes the following processing until the music ends.

  First, when the new library 706 receives sound data (ie, dry sound data) collected by the microphone 60 output from the USB port 401 and subjected to A / D conversion via the Kernel 701 and the Alsa-Lib 702 (steps). S103), the dry sound data is duplicated to generate duplicate voice data (step S104).

  Next, the new library 706 outputs the dry sound data received via the Kernel 701 and the Alsa-Lib 702 to the device drivers of the Kernel 701 and Alsa-Lib 702, and outputs the duplicate audio data of the duplicated dry sound data to the effect processing unit 484. (Step S105).

  Next, the effect processing unit 484 generates effect sound data while performing effect processing on the duplicate sound data output from the new library 706 (step S106), and the sound data generated by the effect processing (that is, the effect data) , The effect sound data) is output to the first layer of Kernel 701 and Alsa-Lib 702 via the second layer of HAL 703, Audio-Fringer 704, and Open SLES 705 (step S107).

  Next, the device driver of Kernel 701 and Alsa-Lib 702 synthesizes the dry sound data output from the new library 706 and the effect sound data output from the effect processing unit 484 (step S108), and HDMI (registered trademark) The data is output to the television receiver 50 via the port 402 (step S109).

  Note that the effect audio data output from the new library 706 and generated by the effect processing unit 484 is further input to the device drivers of the Kernel 701 and the Alsa-Lib 702 via the second layer of the HAL 703, Audio-Finger 704, and Open SLES 705. Therefore, there is a delay from the dry sound data as described above. Therefore, the device driver synthesizes the dry sound data and the effect sound data at the acquired timing. In addition, the device driver appropriately synthesizes accompaniment sound data with dry sound data and effect sound data.

  The accompaniment sound data may be combined with the effect sound data in the second layer of the HAL 703, the audio-ringer 704, and the Open SLES 705, or may be combined with the effect sound data in the third layer that executes the application. Good. In this case, the device driver synthesizes the dry sound data, the effect sound data synthesized in advance, and the accompaniment sound data.

  On the other hand, the scoring processing unit 486 performs scoring processing based on the accompaniment sound data included in the music data and the duplicated sound data at a predetermined timing in parallel with the processing of steps S105 to S108, and the evaluation data (Step S111), and the generated evaluation data is sent to the HDMI (registered trademark) port 402 and the television receiver via the second layer of the HAL 703, the Audio-Fringer 704 and the Open SLES 705, and the first layer of the Kernel 701 and Alsa-Lib 702. 50 (step S112).

  When receiving the evaluation data, the television receiver 50 outputs the music data, dry sound data, and effect-processed data as sound while displaying the image as shown in FIG. 4 together with other image data. .

  Finally, the karaoke processing unit 485 determines the end of the accompaniment sound data or the presence / absence of the end operation in the operation unit 460 (that is, the presence / absence of the end of reproduction) (step S110). If it is determined that no end operation has been detected, the process proceeds to step S103. If it is determined that the end of the accompaniment sound data or the end operation of the operation unit 160 has been detected, this operation ends. Let

  As described above, in the karaoke system 10 according to the present embodiment, the new library 706 is provided in the platform 70 of the communication terminal device 40, the dry sound data is passed through to the Alsa-Lib 702, and the duplicate voice data is generated to generate the karaoke application 80. Therefore, the timing at which the singing voice is output from the television receiver 50 can be reduced from the time when the user speaks, and the satisfaction of the user who performs karaoke can be improved. .

[4] Modification [4.1] Modification 1
In the above embodiment, the configuration is adopted in which the effect sound output from the karaoke application 80 to the HDMI (registered trademark) port 402 is supplied to the HDMI (registered trademark) port 402 via the OpenSLES 705, the Audio-Finger 704, and the HAL 703. Alternatively, the output may be made to the HDMI (registered trademark) port 402 via only the Alsa-Lib 702 and the Kernel 701 without passing through these.

  In this case, in place of these, a new library may be provided, and audio or the like may be supplied to the Alsa-Lib 702 via the library.

  Also, with this configuration, it is possible to further improve the user satisfaction by reducing output delay of effect sound and the like.

[4.2] Modification 2
In the above embodiment, the music DB 300 is managed by the music providing server device 30. However, another computer system may be used to manage the music DB 300.

[4.3] Modification 3
The platform structure described above is applied to the communication terminal device 40 used in the karaoke system 10 of the above embodiment. However, the A / D is not limited to the communication terminal device 40 used in the karaoke system 10 or the karaoke system 10 but is collected by the microphone 60 and A / D. The present invention can also be applied to a voice input processing device such as a loudspeaker that performs input processing on the converted voice data.

  For example, such a sound input processing apparatus outputs a dry sound and processes a duplicate sound of a dry sound when data is input by a user when adjusting other sound fields such as a karaoke BOX. It is applied to a device that displays waveforms. Further, by having such a configuration, it is possible to prevent a divergence between the sound waveform display and the sound output from the speaker.

DESCRIPTION OF SYMBOLS 1 ... Karaoke network system 10 ... Karaoke system 20 ... Network 30 ... Music provision server apparatus 40 ... Communication terminal apparatus 50 ... Television receiver 60 ... Microphone 70 ... Platform 80 ... Karaoke application 410 ... Communication control part 440 ... Display control part 450 ... Recording unit 451 ... Program recording unit 452 ... Application recording unit 453 ... Music data recording unit 454 ... RAM
470 ... terminal management control unit 480 ... application execution unit 481 ... music data acquisition management unit 482 ... effect processing unit 483 ... karaoke processing unit 484 ... scoring processing unit 701 ... Kernel
702 ... Alsa-Lib
703 ... HAL
704 ... Audio-Fringer
705 ... OpenSLES
706 ... New library

Claims (11)

Receiving means for receiving an instruction input from a user who selects at least one of the music data from a plurality of music data;
Obtaining means for obtaining the selected music data;
An interface connected to an external device and outputting predetermined input data to the external device in accordance with a predetermined procedure;
When the voice uttered by the user is input, the input of the voice data is received as input voice data from the voice input means that converts the voice into voice data of a predetermined format and outputs the input voice data. To generate duplicate audio data, and to output the input audio data and either of the input audio data and the duplicate audio data to the interface;
Audio data different from the audio data output to the interface from the input / output management unit is acquired, and predetermined processing is executed based on the acquired audio data and the acquired music data. Data processing means for outputting predetermined data to the interface;
A karaoke system comprising: The karaoke system according to claim 1,
A first layer having an interface that functions as a device driver for the external device;
A second layer configured on the first layer and having the input / output management means;
A third layer configured on the second layer and having the data processing means;
Having a platform structure having at least
The second layer has sound control means for controlling the interface based on data relating to sound output from the data processing means;
The karaoke system, wherein the input / output management means directly outputs one of the input voice data and the duplicated duplicate voice data to the interface. The karaoke system according to claim 1 or 2,
The karaoke system, wherein the interface synthesizes sound data relating to sound processed by the data processing means and sound data output from the input / output management means, and outputs the result to the external device. It is a karaoke system of any one of Claims 1-3,
The karaoke system, wherein the interface converts input data into digital data of the predetermined format and outputs the converted digital data to the external device. It is a karaoke system of any one of Claims 1-4,
The data processing means is
As the predetermined process, a process of generating effect sound data corresponding to an effect sound obtained by performing an audio effect process on the input sound data,
A karaoke system for outputting the generated effect sound data to the interface. It is a karaoke system of any one of Claims 1-5,
The song data includes at least guide melody data including at least one of the scale, note value, and rest of the song when singing the corresponding song,
A karaoke system in which the data processing means executes a scoring process for evaluating the singing ability of the user based on the guide melody data and the input voice data. The karaoke system according to claim 6,
The data processing unit, as the scoring process,
The voice displacement state corresponding to the voice data or the duplicate voice data is compared with the scale, note value, rest included in the guide melody data,
Generate image data showing the comparison results as result data,
A karaoke system for outputting the generated result data to the interface. Computer
Receiving means for receiving an instruction input of a user who selects at least one of the music data from a plurality of music data;
Obtaining means for obtaining the selected music data;
An interface that is connected to an external device and outputs predetermined input data to the external device in accordance with a predetermined procedure;
When the voice uttered by the user is input, the input of the voice data is received as input voice data from the voice input means that converts the voice into voice data of a predetermined format and outputs the input voice data. The input / output management means for generating duplicate audio data by outputting the audio data of any one of the input audio data and the duplicate audio data; and
Audio data different from the audio data output to the interface from the input / output management unit is acquired, and predetermined processing is executed based on the acquired audio data and the acquired music data. Data processing means for outputting predetermined data to the interface;
A program characterized by functioning as A karaoke voice playback method for playing karaoke voice,
A first step of receiving an instruction input from a user selecting at least one of the music data from a plurality of music data;
A second step of obtaining the selected music data;
A third step of accepting input of the voice data as input voice data from voice input means that outputs the voice uttered by the user while converting the voice into voice data of a predetermined format;
A fourth step of duplicating the inputted input voice data to generate duplicate voice data;
Output either the input audio data or the duplicate audio data to the external device via the interface, and acquire audio data different from the audio data output to the external device via the interface A fifth step of executing a predetermined process based on the acquired audio data and the acquired music data and outputting the processed predetermined data to the interface;
A karaoke voice reproducing method comprising: A voice input processing device for processing input voice,
An interface connected to an external device and outputting predetermined input data to the external device in accordance with a predetermined procedure;
When the voice uttered by the user is input, the input of the voice data is received as input voice data from the voice input means that converts the voice into voice data of a predetermined format and outputs the input voice data. To generate duplicate audio data, and to output the input audio data and either of the input audio data and the duplicate audio data to the interface;
Data processing means for acquiring voice data different from the voice data output to the interface from the input / output management means, and executing predetermined processing based on the acquired voice data;
A voice input processing device comprising: A program for processing audio input by a computer,
The computer,
An interface that is connected to an external device and outputs predetermined input data to the external device in accordance with a predetermined procedure;
When the voice uttered by the user is input, the input of the voice data is received as input voice data from the voice input means that converts the voice into voice data of a predetermined format and outputs the input voice data. The input / output management means for generating duplicate audio data by outputting the audio data of any one of the input audio data and the duplicate audio data; and
Data processing means for acquiring voice data different from the voice data output from the input / output management means to the interface, and executing predetermined processing based on the acquired voice data;
A program characterized by functioning as