JP2008122888A - Karaoke machine - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a karaoke machine capable of changing parameters for generating an image object according to various parameters and displaying the image object generated as the result of changing parameters. <P>SOLUTION: According to one or more of "average level", " peak level", "frequency component of the largest energy" and "energy of a specified frequency component", which are obtained by analyzing "classification of a musical instrument", "tone height", "tone intensity", " position of sound image" and "data obtained by analog-to-digital conversion of user's voice input from a microphone" included in "karaoke accompaniment music score data stored in a memory", one or more parameters of "classification", "level", "color", "shape", "coordinates" of the image object are dynamically changed. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a karaoke apparatus, and more particularly, to a karaoke apparatus that changes an image on a television screen according to a karaoke accompaniment and a sound sung by a user.

This type of prior art is disclosed in Patent Document 1.

The prior art of Patent Document 1 captures a singer with a camera and changes the video in accordance with a sound output signal of a microphone.
JP-A-6-250683 [G10K 15/04]

Since the above-described conventional technique is to change an image obtained by photographing a human face, there is a limit to image expression.

Therefore, the main object of the present invention is to change the parameters for generating the image object according to various parameters included in the karaoke accompaniment score data, the user's voice, and the user's musical instrument performance, and the result is generated as a result. A karaoke apparatus capable of displaying an image object as a background image of karaoke is provided. Another object of the present invention is to provide an expandable karaoke apparatus rich in image expression.

The invention of claim 1 is a karaoke apparatus capable of outputting a video signal and an audio signal to a home television apparatus or the like, a microphone built in or detachable in the karaoke apparatus, a memory built in or detachable in the karaoke apparatus, And an image object expressed in two dimensions and / or three dimensions, an image composed of one or more image objects is generated as a background image of karaoke, and synthesized with a lyrics image indicating the lyrics of karaoke, Image processing means for outputting as a video signal, the image processing means including “musical instrument type”, “sound pitch (pitch)”, “sound duration” included in “karaoke accompaniment score data stored in memory” "Time", "Strength of pronunciation (velocity)", "Position of sound image (panpot)", and "User's sound input from microphone" 1 of “average level”, “peak level”, “frequency component with the largest energy”, or “energy of a specific frequency component” obtained by analyzing “data obtained by analog-to-digital conversion” The karaoke apparatus is characterized in that one or more parameters of “type”, “size”, “color”, “shape”, and “coordinate” of an image object are dynamically changed according to a plurality.

According to the first aspect of the present invention, since the karaoke accompaniment score data and the video synchronized with the user's voice are generated as the background image of karaoke, the user may feel a sense of unity between music and video when enjoying karaoke. This can provide a new karaoke experience. Also, it is not necessary to prepare a background image for each karaoke song, and the labor for producing karaoke content can be reduced.

In the invention of claim 2, the karaoke apparatus further includes a connection port to which an electronic musical instrument can be connected,
"Musical instrument performance data input from the connection port in a format similar to MIDI or MIDI""Musical instrument type", "Pitch", "Sound duration", "Sounding intensity" (Velocity), “Position of sound image (Panpot)”, and “Type”, “Size”, “Color”, “Shape”, “Coordinate” of the image object according to one or more parameters A karaoke apparatus characterized by dynamically changing one or more parameters.

According to the invention of claim 2, since the video synchronized with the user's musical instrument performance is generated as the background image of the karaoke, the user can feel a sense of unity between the musical instrument performance and the video when enjoying the karaoke, This can provide a new enjoyment of karaoke.

According to the present invention, it is possible to provide new enjoyment of karaoke.

The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

With reference to FIG. 1, the karaoke apparatus block diagram of the karaoke apparatus 1 which is an Example of this invention is demonstrated.
FIG. 1 is a configuration diagram of a system including a karaoke device 1 and devices connected to the karaoke device 1. The karaoke device 1 is connected to a home TV 3. The lyrics subtitle image and the background image are displayed on the screen of the home TV 3, and the accompaniment music of karaoke and the user's voice input from the microphone 11 are played from the TV speaker. The karaoke apparatus 1 includes a microphone 11, a key switch 13, a memory cartridge connector 12, an expansion port 17, and the like, and the shape of the housing is, for example, a microphone type.
The function of each part is described below.

Microphone 11
The user's voice is converted into an analog voice signal.

Key switch 13
It consists of a number of keys for operating the karaoke machine 1. In this embodiment, “Slow key”, “Fast key”, “b (flat) key”, “# (sharp) key”, “+ (plus) key”, “− (minus) key”, “cancel key” ”And“ Determine key ”. The function of each key is shown below.
• Slow key: Decreases the tempo of the song being played. Each time you press the key, the tempo is lowered by one step. However, if the song is already playing at the slowest tempo, nothing happens when you press this key.
・ Fast key: Increases the tempo of the song being played. Each press will increase the tempo by one step. However, if the song is already playing at the fastest tempo, nothing happens when you press this key.
• ♭ key: Decreases the pitch (pitch) of the song being played. Each time you press it, the pitch is lowered by one step. However, if the song is already playing at the lowest pitch, nothing will happen even if you press this key.
• # key: Increases the pitch of the song being played. Each press increases the pitch by one step. However, if the song is already playing at the highest pitch, nothing will happen even if you press this key.
• + key: Increases the volume of the user's voice input from the microphone 11. Each press increases the volume by one level. However, if the sound is already being played at the highest volume, nothing will happen even if this key is pressed.
-Key: Decreases the volume of the user's voice input from the microphone 11. Each press increases the volume by one level. However, if the sound is already being played at the smallest volume, nothing will happen even if this key is pressed.
・ Cancel key: Used to cancel the currently selected item or move to the menu one level higher on the menu screen or song selection screen. If this key is pressed during song playback, song playback is interrupted and the song selection screen reappears.
・ Determination key: Used to determine the currently selected item on the menu screen and song selection screen.
The four keys “♭ (flat) key”, “# (sharp) key”, “+ (plus) key”, and “-(minus) key” are arrows indicating left, right, up, and down, respectively. It is also used for the purpose of moving the selected item up, down, left and right on the menu screen and song selection screen.

Memory cartridge 2
Video data before decoding, audio data before decoding, musical score data, image object data, program code executed by CPU 102, program code executed by DSP 103, etc. are stored in NAND flash memory or mask ROM.

Memory cartridge connector 12
A connector for connecting the memory cartridge 2.

Expansion port 17
A connector for connecting electronic musical instruments. Based on UART.

External microphone 5
When multiple users enjoy karaoke, the voice of a user other than the user who inputs voice from the microphone 11 of the device is converted into an analog voice signal and output.

External microphone input terminal 18
A terminal for inputting analog audio signals from the external microphone 5.

Video output terminal 14
This terminal is used to output the composite video signal generated by the karaoke machine 1 to the home TV 3.

Audio output terminals 15, 16
This terminal is used to output the analog audio signal generated by the karaoke machine 1 to the home TV 3. It consists of an audio output terminal (left) 15 for outputting the left channel signal and an audio output terminal (right) 16 for outputting the right channel signal.

Next, with reference to FIG. 2, the screen image drawn on the television screen by the image signal output from the video output terminal 14 of the karaoke apparatus 1 of the present invention will be described.

(A) of FIG.
This is a screen image for displaying a movie as a background image for karaoke. The moving image data is MPEG-4 or H.264. It is compressed in a format such as H.264 (MPEG-4 / AVC) and stored in the memory cartridge 2. The decoded (decompressed) video data is displayed on the video screen handled by the display controller 109 described later. The subtitle image of karaoke is drawn in the frame buffer by the graphics processor 107 described later, and the display controller 109 displays the image stored in the frame buffer as an overlay on the video screen.

FIG. 2 (B)
This is a screen image for displaying an image object drawn in 3DCG (3D computer graphics) that changes in real time according to the accompaniment and the user's voice as the background image of karaoke. In this example, an image object representing “people” dances along with the accompaniment, and the mouth moves as if the image object is singing along with the voice of the user input from the microphone 11. Image object model data and animation data are stored in the memory cartridge 2. The image object is drawn in the frame buffer along with the subtitle image by the graphics processor 107. The display controller 109 displays the image stored in the frame buffer.

Next, with reference to FIG. 3, the block diagram which shows the electrical structure of the karaoke apparatus 1 of this invention is demonstrated. (Some of the parts already explained in Fig. 1 will be omitted.)

Input stage amplifier 21
The amplitude of the analog audio signal input from the microphone 11 is amplified and output to the microcomputer 22 with ADC.

Microcomputer 22 with ADC
A microcomputer (microcontroller) with two analog input terminals and ADC (analog-digital converter). The analog audio signal input from the input stage amplifier 21 and the external microphone input terminal 18 is converted into digital audio data and output to the multimedia processor SoC. It is connected to the first UART port (UART # 0) of the multimedia processor SoC100.

Key switch 13
It is connected to the first GPIO (General Purpose I / O) set (GPIO # 0 116) of the multimedia processor SoC, and the status of whether each key is pressed is input to the multimedia processor SoC100.

Expansion connector Connected to the second UART port (UART # 1) of multimedia processor SoC100. Performance information input from an electronic musical instrument is transmitted to the multimedia processor SoC100 via serial communication. The specifications of this expansion connector vary depending on the software stored in the memory cartridge 2. In addition to electronic musical instruments, it is also possible to connect a tester of the karaoke device 1 itself.

Power circuit 29
Multiple DC power supply voltages such as + 3.3V and + 1.2V are generated from dry cell 28 and supplied to each part in the equipment.

Multimedia processor SoC (System on Chip) 100
When the power of the karaoke device 1 is turned on or a system reset of the karaoke device 1 occurs, the program code stored in the boot ROM 23 is first executed, and the program code and data are loaded from the memory cartridge 2 into the main memory 24. , Start execution of the loaded program code. According to the execution of the program code, it generates images and sounds to be output to the home TV 3 according to the digital sound data input from the microcomputer 22 with ADC, the status of the key switch 13, and the input data from the electronic musical instrument 4 . The generated image is converted into a composite video signal and output through the video output terminal 14. The audio is output to the audio DAC 26 as a digital audio signal in I2S format.

Boot ROM 23
Stores the program code executed by the CPU 102 in the multimedia processor SoC 100 when the power of the karaoke device 1 is turned on or when the system reset of the karaoke device 1 occurs. It also stores the title screen image data that is displayed when the karaoke machine 1 is started and the music data that is played back when the title screen is displayed.

Main memory 24
This is the main memory of the multimedia processor SoC100. It consists of 8Mbyte to 32Mbyte SDRAM. It is divided into an area for storing program code and data loaded from the memory cartridge 2, an area as a frame buffer for storing generated images, and an area for storing texture data mapped to 3DCG.

Audio DAC26
The digital audio signal output from the multimedia processor SoC is converted into an analog audio signal and output to the output stage amplifier 27.

Output stage amplifier 27
The amplitude of the analog audio signal input from the audio DAC 26 is amplified and output to the home TV 3 through the audio output terminals 15 and 16.

Memory cartridge connector 12
Connected to the second GPIO set (GPIO # 1). The CPU 102 accesses the memory cartridge 2 by I / O access.

Next, a block diagram showing the internal configuration of the multimedia processor SoC 100 will be described with reference to FIG.

CCCP101 (Consumer Custom Solutions Platform)
It includes a bus bridge between two internal bus systems AHB104 and APB120, two sets of UART, GPIO # 1, and an external memory interface (including SDRAM controller).

AHB104
A high-speed internal bus mainly used for memory access. The CPU 102, DSP 103, DMAC 106, graphics processor 107, display controller 109, and sound processor 110 are connected as a bus master. It is also possible to access the memory bus 25 and APB120 through CCCP101.

APB120
A low-speed internal bus mainly used for control and I / O access. The DSP 103, DMAC 106, graphics processor 107, display controller 109, sound processor 110, CAU 111, power management unit 112, timer 113, mailbox 114, GPIO # 0 are connected as bus slaves.

CPU102
32-bit RISC type CPU. The program code stored in the boot ROM 23 and the main memory 24 is executed, and the overall control of the multimedia processor SoC 100 is performed.

DSP103 (Digital Signal Processor)
A processor for high-speed processing. The program code stored in the DSP local memory 105 is executed, and the DSP local memory 105 is used as a working memory area. Communication with the CPU 102 is performed through a mailbox 114 connected to the APB 120. The DMAC 106 can be used to perform DMA transfer between the main memory 24 and the DSP local memory 105.

DSP local memory 105
Stores program code executed by DSP103. It is also used as a working memory area for the DSP 103. The DSP local memory 105 and the GPU local memory 108, which will be described later, are physically the same memory block, and these two memories divide and use the memory block area.

DMAC 106 (Direct Memory Access Controller) performs DMA transfer between the main memory 24 and the DSP memory, between the main memory 24 and the GPU local memory 108, and between one area of the main memory 24 and another area.

Graphics processor 107
Geometric operations for 3DCG generation (matrix operations, affine transformations, projection transformations, etc.) It is responsible for setting up polygons that make up 3DCG and drawing polygons and 2D image objects to the frame buffer.

GPU local memory 108
This is the local memory handled by the graphics processor 107. Used for Z buffer storage when 3DCG drawing.

Display controller 109
The display screen image is generated by combining the video screen, frame buffer, and 4D 2D character screen. The video screen and frame buffer images are stored in the main memory 24. The 2D character screen is an image consisting of a two-dimensional array of characters arranged in a grid of 8 pixels by 8 pixels. The image of each character and the character placement information are stored in the main memory 24. The generated display screen image is converted to a digital data stream representing the video composite signal waveform and output to the video DAC 115.

Sound processor 110
Performs pitch conversion of instrument sounds stored as PCM data and / or ADPCM data in the main memory 24, synthesizes up to 64 instrument sounds, and plays accompaniment sounds of karaoke. The compressed PCM data decoded by the CPU 102 or the DSP 103 and stored in the main memory 24 is reproduced. The reproduced audio is output to the audio DAC 26 as a digital audio signal in I2S format.

CAU111 (Conversion Acceleration Unit)
Performs fast conversion between floating-point numbers and integer or fixed-point numbers. It is used to calculate parameters such as the coordinates of image objects drawn with 3DCG.

Power management unit 112
Reduces the power consumption of the multimedia processor SoC by controlling the operating frequency of the CPU 102 and DSP 103 in stages, and stopping the supply of clock signals to unused functional blocks.

Timer 113
The built-in counter value is updated based on the supplied clock signal. When the count value matches the predetermined value, it is possible to assert an interrupt request signal for the CPU 102 and / or the DSP 103.

Mailbox 114
It is used for message communication between CPU102 and DSP103.

GPIO # 1
Digital I / O port used for general purposes. In this application karaoke device, it is connected to the key switch 13 and the status of each key is input.

Memory bus 25
Asynchronous interface ROM, asynchronous interface SRAM, NOR flash memory, SDRAM can be connected. In this application karaoke device, boot ROM23 and SDRAM of 8MB to 32MB are connected.

Video DAC115
The digital data stream input from the display controller 109 is converted to an analog signal and output as a composite video signal.

Next, with reference to FIGS. 5, 6, and 7, the case where the decoded moving image data is displayed as a karaoke background image using the operation flowcharts of the CPU 102 and the DSP 103 will be described. S1 is an abbreviation for step S1. The same applies to S2 to S2. Further, when the step represents the operation of the CPU 102, it is described as (CPU 102), and when the step represents the operation of the DSP 103, it is described as (DSP 103).

S1 (CPU 102) See FIG.
After starting the system, the CPU 102 starts executing the program code stored at a predetermined address in the boot ROM 23. After the start of execution, initialization of the hardware settings of each functional block, setting of the memory space allocated as the DSP 103 local memory, setting of the VRAM area (frame buffer, texture data, etc.) in the main memory 24 are performed.

S3 (CPU 102)
Displays the title screen as a karaoke device. The program code and image data for display are stored in the boot ROM 23 in advance.

S5 (CPU 102)
The program code executed by the CPU 102 and the program code executed by the DSP 103 are loaded into the main memory 24 from the memory cartridge 2.

S7 (CPU 102)
Wait for program code to be loaded into main memory 24.

S9 (CPU 102)
After loading is complete, CPU102 starts executing the loaded program code.

S11 (CPU 102)
The program code executed by the DSP 103 is transferred to the DSP 103 local memory by DMA.

S13 (CPU 102)
Wait for DMA transfer to complete.

S15 (CPU 102)
After the transfer is complete, the DSP 103 is instructed to start executing the program code transferred to the DSP local memory 105.

S17 (DSP103)
The DSP 103 initializes the execution environment of the DSP 103 immediately after the execution of the program code is started.

S21 (CPU 102) See FIG.
The CPU 102 checks whether or not a predetermined amount of pre-decoding moving image data and a predetermined amount of pre-decoding audio data need to be loaded into the main memory 24. When this step is executed for the first time, since both are not loaded into the main memory 24, it is determined that both need to be loaded. If the CPU 102 determines that it is necessary to load, the process proceeds to S23, and if not, the process proceeds to S25.

S23 (CPU102)
If it is determined that either a predetermined amount of pre-decoding video data, a predetermined amount of pre-decoding audio data, or both need to be loaded, the data is loaded from the memory cartridge 2 to the main memory 24.

S25 (CPU 102)
Check the current presentation time. The presentation time is cleared to 0 at the start of playback, and is updated based on the vertical synchronization of the video signal or the count value by the timer 113.

S27 (CPU 102)
The CPU 102 decodes the audio data loaded in the main memory 24. The audio data loaded into the main memory 24 is compressed in a format such as AAC or MP3. The CPU 102 decompresses it and generates a linear PCM audio data stream.

S29 (CPU102)
Instructs the sound processor 110 to play back the decoded audio data (linear PCM format).

S31 (CPU 102)
Next, it is confirmed whether the image frame data to be displayed is already stored in the main memory 24 by the DSP 103. If the CPU 102 determines that it is stored in the main memory 24, the process proceeds to S33, and if not, the process proceeds to S41.

S33 (CPU102)
Check whether the current presentation time has reached the timing for displaying new image frame data in the next video frame. If the CPU 102 determines that it is the timing, the process proceeds to S35, and if not, the process proceeds to S41.

S35 (CPU 102)
In order to improve the display quality of the video, the image data may be dithered. The color data format handled by the display controller 109 is a maximum of 16 bits per pixel. Therefore, if the decoded moving image data (usually 24 bits per pixel) is reduced to 16 bits as it is, the color boundary stripes (Mach band) will stand out. If random noise is added to the color data of each pixel in the decoded video data and then the accuracy is reduced, the Mach band will not be noticeable to the human eye. This process is called dithering. When performing dithering processing, it may be necessary to convert the color space. H. For video codecs such as H.264 and MPEG-4, the color space format of the decoded video data is the color difference signal YCbCr. Since dithering is usually performed in the RGB color space, color space conversion processing from YCbCR to RGB is required before dithering. In this embodiment, the color space conversion process is performed by the CPU 102. As the image frame data to be displayed next, the color space of each pixel of the data stored in the main memory 24 by the DSP 103 is converted from YCbCr to RGB.

S37 (CPU 102)
Dithering the image frame data after color space conversion. Based on the display XY coordinates of each pixel in the image frame data, noise data is extracted from the data table and added to the color data of each pixel.

S39 (CPU 102)
Set the image frame data that has been dithered to be displayed on the video screen in the next video frame.

S41 (CPU 102)
Generate a display list of subtitle images. The subtitle image is drawn in the frame buffer and displayed on the video screen by the display controller 109 when the video signal is output. In this embodiment, each character data of the subtitle image is drawn in the frame buffer as a sprite (rectangular two-dimensional graphics object). The CPU 102 creates a display list including parameters such as drawing coordinates, color depth, and color lookup table specification for each sprite.

S43 (CPU 102)
A rendering instruction is given to the graphics processor 107 so that the subtitle image is rendered in the frame buffer based on the display list created in S41. The VRAM area of the main memory 24 is equipped with two sets of frame buffers, which can be switched between drawing and display at any time. The graphics processor 107 performs drawing on the frame buffer set for drawing.

S45 (CPU 102)
Wait for the start of the vertical blanking period in the composite video signal.

S47 (CPU 102)
If the update of the frame buffer and / or video screen has been completed, the display controller 109 switches the frame buffer and / or video screen data to be read for display.

S48 (CPU 102)
The presentation time is updated based on the start of the vertical blanking period of the composite video signal or the count value of the timer 113.

Next, the processing of the DSP 103 will be described.
S49 (DSP103)
The DSP 103 checks whether or not a predetermined amount of pre-decode video data has already been loaded into the main memory 24. Then wait for it to load.

S51 (DSP103)
Transfers pre-decoded video data from the main memory 24 to the DSP local memory 105 using DMA.

S53 (DSP103)
Decodes the video. In this embodiment, the DSP 103 is H.264. Runs H.264 decoder software or MPEG-4 decoder software.

S55 (DSP103)
The image frame data to be displayed next is transferred from the DSP local memory 105 to the main memory 24 using DMA.

S57 (DSP103)
The mail box 114 is used to notify the CPU 102 that the image frame data to be displayed next is already stored in the main memory 24.

S59 (DSP103)
Check whether the pre-decode video data required for the next decoding unit exists in the DSP local memory 105. If it already exists, you can continue decoding the next image frame data. (Return to S49) Otherwise, return to S53.

Referring to FIG. 7, a flowchart showing the contents of processing performed by CPU 102 and DSP 103 when a CG character that changes in accordance with the accompaniment of karaoke and the user's voice input from microphone 11 is displayed as a karaoke background image. Will be explained. (Synchronized CG) The initialization part processing (S1 to S17) is the same as the moving image playback flowchart of FIG. However, the content of the program code executed by the DSP 103 is different from the case of moving image playback.

S61 (CPU 102)
Audio data input from the microphone is received from the microcomputer 22 with ADC and stored in the main memory 24. When two users' voices are input using the external microphone 5, the two audio data streams are interleaved and input from the microcomputer 22 with ADC.

S63 (CPU102)
Waiting for the completion of receiving the specified amount of audio data. The processing performed by the DSP 103 requires a predetermined number of samples of PCM audio data.

S65 (CPU 102)
Using the mail box 114, the DSP 103 is notified that a predetermined amount of audio data is already stored in the main memory 24.

S67 (CPU102)
Updates the value of the sound processing counter. In this example, karaoke accompaniment is based on MIDI-based score data. In this case, the CPU 102 performs management of the playback timing of the tempo and each note based on the sound processing counter.

S69 (CPU 102)
The musical score data is “musical instrument type”, “playback start timing”, “playback stop timing”, “pitch (pitch)”, “velocity (sounding intensity)”, “panpot (sound image Information such as “Position” ”is stored in chronological order. The CPU 102 manages a pointer indicating the reading position of the score data, and controls how far the pointer is advanced according to the value of the sound processing counter. In this step, the musical score data at the position pointed to by the pointer is read and interpreted.

S70 (CPU 102)
The pointer value is updated from the musical score data to the position to be read next.

S71 (CPU 102)
Based on the value of the sound processing counter, it is determined whether or not the pointer value has been read and interpreted all the musical score data to be processed this time. If it has been read, the process proceeds to S73, and if not, the process returns to S69.

S73 (CPU102)
"Musical instrument type", "Playback start timing", "Playback stop timing", "Pitch (pitch)", "Velocity (sounding strength)", "Panpot (sound image position)" The "type", "size", "color", "shape", and "coordinate" parameters of the image object to be drawn are calculated according to the information of "".

S75 (CPU102)
Check the mailbox 114 and determine whether the DSP 103 has notified the microphone input voice parameters. If the parameter has been notified, the process proceeds to S77, and if not, waits.

S77 (CPU 102)
According to the parameters “average level”, “peak level”, “frequency component with the largest energy”, and “energy of a specific frequency component” of the microphone input sound notified from the DSP 103, the “type” of the image object to be drawn, Change the "Size", "Color", "Shape" and "Coordinate" parameters.

S79 (CPU 102)
Based on the image object parameters calculated in S73 and S77, a display list of image objects to be displayed as a background is created. A display list of subtitle images is also created. Of the image objects displayed as the background, those reproduced with 2DCG are drawn in the frame buffer as sprites and those reproduced with 3DCG as polygons. Each character that makes up a subtitle image is drawn as a sprite in the frame buffer, just like when playing a movie. The CPU 102 displays a display list including parameters such as “drawing coordinates of each polygon, vertex color, texture pattern, texture color depth, vertex texture coordinates, etc.” and “sprite drawing coordinates, color depth, color lookup table designation, etc.” Create a.

S81 (CPU 102)
Based on the display list created in S79, the graphics processor 107 is instructed to draw the image object displayed as the background and the subtitle image in the frame buffer.

S83 (CPU102)
Wait for the start of the vertical blanking period in the composite video signal.

S84 (CPU102)
If the frame buffer update is complete, the display controller 109 switches the frame buffer to be read for display. After S84 ends, the process returns to S61.

On the other hand, the operation of the DSP 103 will be described.
S85 (DSP103)
Check whether a predetermined amount of microphone input audio data is already loaded in the main memory 24. If it has been loaded, the process proceeds to S87, and if not, it waits.

S87 (DSP103)
Using DMA, the loaded microphone input audio data is transferred from the main memory 24 to the DSP local memory 105.

S89 (DSP103)
Calculates “average level” and “peak level” of microphone input voice data. The average level is calculated from the average deviation or standard deviation after calculating the average value of all samples. The peak level is obtained from the maximum deviation from the average value of all samples.

S91 (DSP103)
Performs FFT (Fast Fourier Transform) processing for microphone input audio data. From the processing results, calculate the “frequency component with the largest energy” and the “energy of a specific frequency component”.

S93 (DSP103)
Using the mailbox 114, the CPU 102 is notified of the “average level”, “peak level”, “frequency component with the largest energy”, and “energy of a specific frequency component” of the microphone input voice data calculated in S89 and S91. . After S93 ends, the process returns to S85.

Next, an example of application software using the karaoke apparatus of the present invention will be shown.

1. An example DSP 103 for synthesizing a moving image background and synchro CG is H.264. H.264 or MPEG-4 decoder software is executed, and the compressed moving image data is decompressed and stored in the main memory 24.

The CPU 102 instructs the sound processor 110 to reproduce karaoke accompaniment based on the MIDI-based score data.

Further, the CPU 102 calculates the parameters of the 3DCG image object based on the note parameters in the score data and the microphone sound parameters. Further, based on the calculated parameters, a display list of each polygon constituting the 3DCG image object is generated and passed to the graphics processor 107.

Further, the CPU 102 generates a display list of each sprite constituting the subtitle image based on the lyrics data embedded in the score data and the lyrics display control data, and passes it to the graphics processor 107.

The graphics processor 107 draws a 3DCG image object composed of polygons and a subtitle image composed of sprites in a frame buffer based on the display list.

The display controller 109 displays the decompressed moving image data stored in the main memory 24 on the video screen by overlaying the image stored in the frame buffer.

As a result, it is possible to use, as a karaoke background, an image in which a CG character that changes in real time according to accompaniment or microphone sound is synthesized on a previously recorded video.

2. Display example of synchro CG karaoke “Human” is displayed as a 3DCG image object represented by polygons.

The position of the “human” hand moves up and down according to the “pitch (pitch)” of the “piano” notes in the accompaniment score data. If the pitch is high, the hand position is high, and if the pitch is low, the hand position is low.

“Human” repeatedly jumps on the screen. The jump height is determined according to the “peak level” of the microphone sound data. If the peak level is high, the jump height is high, and if the peak level is low, the jump height is low.

3. A large number of short karaoke video background examples that change interactively according to accompaniment and microphone sound are stored in the memory cartridge 2 in advance.

The CPU 102 determines which moving image data is to be reproduced next before the reproduction of the currently reproduced moving image data based on the note parameters in the musical score data and the microphone sound parameters, and the moving image data before decoding. Is loaded from the memory cartridge 2 into the main memory 24.

The DSP 103 sequentially decodes (decompresses) the moving image data stored in the main memory 24.

Thereby, for example, a moving image whose story changes according to accompaniment or microphone sound can be displayed as a karaoke background.

Furthermore, the karaoke apparatus of the present invention may have the following features.
4). For example, when “voice training using a karaoke device” is executed, it can be realized using the following configuration.

The DSP 103 mounted on the multimedia processor SoC 100 performs FFT (Fast Fourier Transform) on the user's voice input from the microphone and converted into digital data, and calculates a frequency component having the largest energy.

The CPU 102 mounted on the multimedia processor SoC 100 gives instructions to the sound processor 110 while reading out MIDI-based score data from the memory cartridge 2 and reproduces the karaoke accompaniment.

The score data includes “voice guide data”. The voice guide data is data indicating the pitch and timing of each note of the scale to be sung by the user. The CPU 102 reads out the voice guide data from the musical score data, but does not necessarily reproduce using the sound processor 110.

The CPU 102 instructs the graphics processor 107 to display one or more first score images corresponding to each note of the voice guide data on the screen. The display position of the note image is associated with the pitch of the note of the voice guide data. For example, if the pitch of the sound is high, the vertical display position of the note image on the screen becomes high.

The graphics processor 107 is instructed to display on the screen a second score image corresponding to the first score image in the vicinity of the note image corresponding to each note of the voice guide data. The first score image and the second score image are different in color or shape, for example, so that the user can identify the difference on the screen. The display position of the second note image is associated with the height of “frequency component with the highest energy of the user's voice” transmitted from the DSP 103. For example, if the pitch of the sound is high, the vertical display position of the second note image on the screen becomes high.

The user can recognize in real time the difference between the pitch of the musical note to be sung and the pitch of the current singing sound from the difference in the display position of the two types of note images on the screen. The effect can be expected.

5. By adding a game / speech recognition function using speech and an English pronunciation rating function, it is possible to provide game content that is difficult to achieve with conventional devices in combination with 3DCG and the like.
According to the present invention, music performance and various music games can be executed by connecting a microphone, maracas, tyco, etc. to the expansion port 17.

6). By connecting to the accompaniment instrument main body expansion port 17, accompaniment using an instrument becomes possible. In addition to playing maracas, drums, etc., playing instruments minus 1 (playing the instrument at the right timing will produce a sound with the timing and tone of that timing) will also play an instrument with a pitch such as a guitar or piano Is possible.

7). By inserting a wireless LAN module into the communication karaoke ROM cartridge slot, a PC-free communication karaoke can be realized.

8). “Kara-drama” and “Kara-Anime” utilizing the video display function
Since secondary use of existing video content is possible, it is possible to provide “Kara-drama” and “Kara-Anime” content that can become characters in drama and anime. ("Kara-drama" and "Kara-Anime" are contents that allow you to enjoy the drama and anime characters' telephony.) By muting only the voice of a specific person, by telcoing that person's dialogue , You can get the fun as if it were on the show. Furthermore, by using voice recognition, it is possible to provide a variety of games that were difficult to achieve with conventional devices, such as changing the development of dramas depending on the contents of the user's ateleco.

The characteristics of the karaoke apparatus of the present invention are summarized as follows.
By using the present invention, a home karaoke device connected to a television can be realized at low cost. The present invention not only incorporates functions necessary for a karaoke apparatus such as a CPU, a graphics processor 107, and a sound processor 110, but also decompresses moving image data compressed by the DSP 103 installed, Sound processing etc. can be performed.
The features of the karaoke device of the present invention are as follows.

1. Background of Karaoke by Moving Image The CPU 102 and the DSP 103 mounted on the karaoke apparatus of the present invention make it possible to H.264 or MPEG-4 video can be played back as a karaoke background. H. H.264 can achieve QVGA (320 × 240 pixels) / 15 fps, and MPEG-4 can achieve QVGA / 30 fps.

2. Correspondence to various audio codecs As middleware of DSP103, various audio codecs (decoders) such as MP3 and AAC SBR are already prepared. This has made it possible to play “audio tracks with vocals” and “tracks recorded with live performances” as accompaniment to karaoke, which could not be played with conventional MIDI-based home karaoke equipment.

3. Accompaniment, 3D graphics synchronized to mic sound By combining the excellent 3D graphics generation function of the GPU installed in the karaoke apparatus of the present invention with middleware such as FFT executed by the DSP 103, "Karaoke accompaniment" and It is possible to display 3D graphics synchronized with “user's voice input from microphone” as the background of karaoke.

FIG. 1 is a connection diagram of the karaoke apparatus of the present invention. FIG. 2 is a display example of a video signal output from the karaoke apparatus of the present invention. FIG. 3 is a block diagram showing an electrical configuration of the game apparatus shown in FIG. FIG. 4 is a block diagram showing an electrical configuration of the multimedia processor SoC shown in FIG. FIG. 5 is a flowchart showing the operations of the CPU 102 and the DSP 103. FIG. 6 is a flowchart showing the operations of the CPU 102 and the DSP 103. FIG. 7 is a flowchart showing the operations of the CPU 102 and the DSP 103.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Karaoke apparatus 2 ... Memory cartridge 3 ... Home television 4 ... Electronic musical instrument etc. 5 ... External microphone 11 ... Microphone 12 ... Memory cartridge connector 13 ... Key switch 14 ... Video output terminal 15 ... Audio | voice output terminal (left)
16: Audio output terminal (right)
17 ... expansion port 18 ... external microphone input terminal 21 ... input stage amplifier 22 ... microcomputer 23 with ADC ... boot ROM
24 ... main memory 25 ... memory bus 26 ... audio DAC
27 ... Output stage amplifier 28 ... Dry battery 29 ... Power supply circuit 100 ... Multimedia processor SoC
101 ... CCCP
102 ... CPU
103 ... DSP
104 ... AHB
105 ... DSP local memory 106 ... DMAC
107 ... Graphics processor 108 ... GPU local memory 109 ... Display controller 110 ... Sound processor 111 ... CAU
112 ... Power management unit 113 ... Timer 114 ... Mailbox 115 ... Video DAC
116 ... GPIO # O
120 ... APB

Claims (2)

A karaoke device capable of outputting a video signal and an audio signal to a home TV device,
A microphone built into or detachable from the karaoke device;
A built-in or detachable memory in the karaoke device, and an image object expressed in two and / or three dimensions are generated, and an image composed of one or a plurality of the image objects is generated as a karaoke background image And image processing means for synthesizing with a lyric image showing lyrics of karaoke and outputting as a video signal,
The image processing means includes
“Musical instrument type”, “pitch”, “sound duration”, “sounding intensity (velocity)”, “sound image” included in the “karaoke accompaniment score data stored in the memory” "Panpot", as well as "Data obtained by analog-to-digital conversion of user's voice input from microphone", "Average level", "Peak level", "Most energy" Depending on one or more of “large frequency component” and “energy of specific frequency component”, one of “type”, “size”, “color”, “shape”, and “coordinate” of the image object A karaoke apparatus characterized by dynamically changing a plurality of parameters. The karaoke device
In addition, it has a connection port that can connect electronic musical instruments
“Musical instrument performance data input from the connection port in a format similar to MIDI or MIDI”, “Musical instrument type”, “Pitch”, “Sound duration”, “Strong pronunciation” “Type”, “size”, “color”, “shape”, “coordinate” of the image object in accordance with one or more parameters of “sales (velocity)” and “sound image position (panpot)” The karaoke apparatus according to claim 1, wherein one or a plurality of parameters are dynamically changed.

Images