JPH08123448A - Image processor using waveform analysis of sound signal - Google Patents

Abstract

PURPOSE: To control the display position of an object image of a game equipment by speech input by obtaining scale information on an input speech, generating object image data, and generating image data for display. CONSTITUTION: A main CPU 30 generates scale data on the text that a singer currently sings on the basis of information reported by a sound CPU 43 and stores it in an object VRAM 36. Then an object CONT 37 generates the object image data to be superimposed on a KARAOKE background image on the basis of the reported scale information and the currently played part and stores the data in an object VRAM 26, and also controls a read of the object RAM 36. Further, a priority control circuit 33 puts the background image from a scraping VRAM 32 and the object image from the object VRAM 36 together and inputs the result to a D/A converter 34. Then a video signal which is converted into an analog signal is displayed on a monitor 35.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus which analyzes a waveform of an input voice signal by using a digital signal processing technique of a voice signal and controls a display image based on the analysis result.

[0002]

2. Description of the Related Art In a video game device or the like as an image processing device, a player can enjoy a game by controlling a display position and movement of object data on a screen by using a manual input device such as a keypad. It is common.

On the other hand, due to the widespread use of laser disks and ISDN lines, so-called karaoke devices (musical tone playing devices)
Are likely to become more popular due to their lower cost and abundance of software, but they also need to differentiate themselves in the market by increasing their entertainment.

Further, with the progress of digital signal processing technology, it has been the subject of various studies to use voice as an input of a computer instead of an input from a keyboard device.

Therefore, instead of a manual input device such as a keypad, it is possible to give the player more variability by enabling input to the video game device by voice.

Alternatively, by using an image processing apparatus that is a combination of a video game device and a karaoke device, it is considered that the entertainment property of the karaoke device is further enhanced. That is, while singing a song, the game can be enjoyed on the TV monitor.

For this purpose, it is possible to control the display position or the movement of the object (sprite) on the display screen by using the voice information of the player of the video game or the voice information of the player as the singer as an input control signal. Therefore, it is considered to be suitable.

[0008]

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide an image processing device for controlling the display position or movement of an object (sprite) on a display screen by a player's voice input.

A further object of the present invention is to provide a video game device as an image processing device which enables voice input instead of a manual input device such as a keypad.

It is another object of the present invention to provide a video game device combined with a karaoke device as an image processing device.

[0011]

An image processing apparatus according to the first aspect of the present invention for achieving the above object is a scale information creating means for obtaining a frequency of an input voice waveform and obtaining scale information of the input voice, and scale information. Based on the object image, a position where the object data should be displayed on the background image is obtained, object image creating means for creating the object image data and the object image are prioritized, and the object image and the background image are combined and displayed. Display data creating means for creating image data for use.

A second aspect of the present invention further comprises sound pressure information creating means for detecting the peak value of the input speech waveform and creating sound pressure information, wherein the object image creating means converts the scale information into scale information. In addition, an image processing apparatus using waveform analysis of a sound signal, characterized in that object image data is created based on sound pressure information.

In the third invention, the combination of the scale information and the sound pressure information is made to correspond to each key of the key input device.

[0014]

In the present invention as described above, when used as a video device, the operation of the sprite on the display screen can be controlled by voice input instead of input from the keypad.

When used as a karaoke device, the scale information of the singer is obtained by digital signal processing of the voice signal, and the display position of the object data on the screen is controlled by the scale information (pitch). That is, the display position of the object data on the screen can be changed depending on whether the singer is singing in the correct scale according to the score of the song being played or not.

[0016]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram illustrating a case where a player's analog voice waveform input from a microphone is quantized and converted into a digital voice signal according to the present invention.

FIG. 2 shows the digital audio signal of FIG.
FIG. 3 is a block diagram of an embodiment of the present invention for obtaining a fundamental frequency and a peak level of input voice. Referring to FIG.
Referring to FIG. 2, a method according to the present invention for obtaining scale information and sound pressure information by digital signal processing technology will be described.

As shown in FIG. 1, the voice input waveform generally has a shape close to a sine wave, and its fundamental frequency is 150 to 300 Hz for a female and 100 for a male.
It is known to be ~ 200 Hz.

Based on the sampling theorem, such an input speech waveform is limited to a band below the Nyquist frequency by a filter, and a fixed sampling period (t1, t
Quantize at timing 2). If the quantized amplitude value is represented by an 8-bit digital signal, P
CM coding is completed.

Such a PCM code has a code bit, and for example, a PC at the timing of t1 and t2 shown in FIG.
In the M code, the positive and negative amplitudes are different, and t1 → t
2, the sign bit is inverted at the timing of tn → tn + 1 and tm → tm + 1.

Therefore, it is possible to detect the frequency of the input voice waveform by inverting the sign bit, that is, detecting the zero-cross point of the input voice waveform in a predetermined period. Therefore, the scale information may be obtained from the detected frequency information as to which scale of the eight scales it belongs to.

On the other hand, for the sound pressure information of the audio signal, the audio input waveform of FIG. 1 may be subjected to absolute value processing to detect the peak level.

Specifically, the frequency and sound pressure of the input voice waveform can be obtained with the configuration shown in FIG. Microphone 2
The audio information input from the above is input to the A / D converter 3 via the low pass filter LPF1 and is quantized and PCM encoded. Here, the low pass filter LPF1 is for removing high frequency components of the audio signal.

The PCM-encoded digital voice signal sequentially has a zero-cross detection circuit 11 and an absolute value processing circuit 5.
Is input to The zero-cross detection circuit 11 has a zero-cross point (t1 → t) that changes from the negative polarity to the positive polarity shown in FIG.
2, tm → tm + 1), that is, the timing at which the code bit of the PCM code is inverted from a bit having a negative polarity (usually 1) to a bit having a positive polarity (usually 0) in a predetermined period.

The zero-cross detection circuit 11 has such a P
When the inversion of the CM code is detected, the counter A13 is cleared. The counter A13 performs a counting operation in synchronization with the internal clock 17. Therefore, the counter A13
Detects the number of clocks in one cycle of the input speech waveform shown in FIG. Therefore, the frequency of the input voice waveform can be known from the count value of the counter A13 and the oscillation frequency of the internal clock 17.

In order to make such frequency information more accurate, in the example of FIG. 2, the detection accuracy is increased by obtaining a plurality of count values of the counter A13 and averaging the count values. That is, the output of the zero-cross detection circuit 11 is
It is also input to the counter B15.

The counter B15 is a zero cross detection circuit 1
While counting the zero-cross detection signal from 1,
An address signal is sent to the memory 19 corresponding to the count value, and the count value of the counter A13 is stored in the memory 19. The average value detection circuit 21 detects the average value of each count value stored in the memory 19 and sets it in the register 23.

Since such a PCM code appears every 125 μsec according to the sampling theorem, the frequency of the input speech waveform can be obtained by using the internal clock 17 having a cycle several times faster than 125 μsec.

Next, the peak detection circuit detects the maximum value of the amplitude information of the input voice waveform obtained by the absolute value processing circuit 5. The output of the zero-cross detection circuit 11 is added to the peak detection circuit 7 in order to detect the maximum value in one cycle of the input speech waveform. Upon receiving the signal from the zero-cross detection circuit 11, the peak detection circuit 7 sets the PCM code at that time in the register 9.

Although the frequency and sound pressure of the input voice waveform can be obtained as described above, the functional block diagram of FIG. 2 can be easily realized by a commercially available digital signal processor (DSP). In the concrete example shown in FIG. 3, a configuration using a DSP will be described.

FIG. 3 is a block diagram of an embodiment in which the image processing device of the present invention is applied to a video game device having a karaoke (musical tone performance) function.

The portion connected to the bus 55 is an image processing unit, and the portion connected to the bus 56 is an audio processing unit.
The image processing unit and the audio processing unit are the system controller SYS.
The signals on the respective buses are interfaced by the CONT 41.

The main CPU is attached to the bus 55 of the image processing unit.
30, a work RAM 53, a ROM 54 in which a game program and a system control program are stored are connected. Further, a scroll image (background image) VRAM 32 and an object image (sprite image) VRAM 36, which are controlled by the main CPU 30 via the scroll controller 31 and the object controller 31, are connected.

In the SYSCONT 41, a player in an ordinary video game device inputs a control signal for controlling the display of sprites on the display screen, and in the present embodiment having a karaoke (musical performance) function, the singer is A keypad 42 is connected for inputting the title of a song to be sung and for setting performance speed, key pitch, echo amount, and the like.

It is well known that the keypad 42 may be a cordless remote controller.

When the device is used as a video game device, the SYSCONT 41 is operated by inputting from the keypad 42 as in the case of a normal video game device, in which the moving direction (horizontal movement, vertical movement) and moving speed of the splice on the display screen. It is input to the main CPU 30 as a signal for controlling.

The main CPU 30 controls execution of a game program stored in the ROM 54 or a disk ROM 39 such as a CD (compact disk) / LD (laser disk). In the course of executing the game program, the game is progressed by controlling the movement of sprites and the like on the display screen of the monitor 35 in accordance with the control signal input from the keypad 42.

In this display, the scroll image from the scroll image (background image) VRAM 32 and the object image from the object image (sprite image) VRAM 36 are passed through the priority circuit 33 to the D / A conversion circuit 3
4, the analog video signal is converted and displayed on the monitor 35.

Further, as described above, the VRAMs 32 and 34 are controlled by the scroll controller 31 and the object controller 31, respectively.

On the other hand, on the bus 56 of the voice processing unit, a sound CPU 43 for voice and a work R for the sound CPU 43 are provided.
An AM 57, a ROM 58 in which a program executed and controlled by the sound CPU 43 is stored, and a sound RAM 44 in which performance information is stored are connected.

Further, a DSP 10 for processing a voice signal having a function of processing scale information and sound pressure information according to the principle of the present invention described with reference to FIGS. 1 and 2 is connected to the bus 56. A voice signal input by the microphone 49 is input to the DSP 10 through the A / D conversion circuit 48.

The output of the DSP 10 is output as a voice output to the left and right speakers 47 and 48 through the left and right D / A conversion circuits 45 and 50 and the amplifiers 46 and 47.

Next, when the apparatus of the embodiment shown in FIG. 3 is used as a karaoke apparatus, the music title selected and input by the keypad 42 is transmitted to the center (not shown) via the ISDN line, and performance information is received from the center. To do. Alternatively, it is possible to instruct reading of the performance information stored in the ROM disk 39 from the keypad 42.

The performance information transmitted from the center or the performance information read from the disk ROM 39 includes PCM-encoded karaoke video and performance tone data. Of the received performance information, the musical tone data for performance is downloaded to the sound RAM 44.

Further, various setting information input from the keypad 42 is set in the DSP 10 to control the keys of the musical composition, the echo amount of the voice of the singer, the musical performance speed, and the like. Regarding the playing speed, if there is no instruction from the input pad 42, the standard speed data (ISD
The SYSCONT 41 receives the performance information from the N line or the disc ROM 39. ) May be adopted.

The voice information of the singer is
It is input to the / D converter 48 and is quantized and PCM encoded. This voice information is input to the DSP 10, and the count value / peak value information of the internal clock is obtained according to the principle described in FIG. DSP10 is also sound R
The performance information downloaded via the ISDN line is read from the AM 44.

The DSP 10 digitally processes the performance information based on the set value notified from the sound CPU 43 so as to match the key / performance speed desired by the singer. The digitally processed performance information is transferred to the D / A converter 4
5 is converted into an analog signal, amplified by the amplifier 46, and reproduced by the speaker 47.

Further, the voice of the singer input from the microphone 49 is D
The analog information is converted into analog information via the A / A converter 50 and the amplifier 51 and reproduced by the speaker 52.

As described above, the player can enjoy the karaoke by uttering his own singing voice through the microphone 49 from the speaker 52 against the background of musical performance as an ordinary karaoke device.

Here, a case where a karaoke video image is displayed on the monitor 35 as an image when used as a normal karaoke apparatus will be described.

The SYSCONT 41 is the IS described above.
Karaoke video video sent from the center through the DN line or read from the disk ROM 39 is downloaded to the work RAM 53. Karaoke video video downloaded to the work RAM 53 is the main CP
It is transferred and stored in the scroll VRAM 32 under the control of U30.

The main CPU 30 further includes an input pad 42.
Performance speed data set from the above or standard speed data determined for each performance music (performance information from the center or disk ROM 39 through ISDN line together with S
The YSCONT 41 receives it. ), The timing information for reading the karaoke video video transferred and stored in the scroll VRAM 32 is provided to the scroll controller SCO.
Notify NT31. Therefore, the karaoke video image is displayed on the monitor 35 in synchronization with the musical performance.

Based on the scale information and the sound pressure information obtained according to the principle explained in FIG. 2 according to the present invention, in addition to the function of using the apparatus of the embodiment shown in FIG. 3 as a normal video game apparatus and a karaoke apparatus as described above. A technique for controlling the display image on the monitor 35 will be described below.

First, when used as a video game device, according to the present invention, the display control of the sprite displayed on the monitor 35 does not depend on the input from the keypad 42.
It can be performed by voice input from the microphone 49.

That is, as described above, the DSP 10
According to the principle described with reference to FIGS. 1 and 2, the scale information and the sound pressure information of the voice sampled at a predetermined timing input from the microphone 49 are output.

On the other hand, information as shown in FIG. 4 in which the combination of the scale information and the sound pressure information is associated with each key of the keypad 42 is stored in the work RAM 53 in advance. In the example of FIG. 4, when the scale and the sound pressure are classified into 1 and 1, for example,
The up arrow key ↑ corresponds to the scale and sound pressure of 4, 1
In the case of being classified into, the key of the left direction arrow ← corresponds.

Therefore, the player switches the input from the keypad 42 before starting the game by using the main CPU 30.
In response to the player's voice input from the microphone 49, the main CPU 30 replaces the input from the keypad 42 or in common with the input from the keypad 42 during the video game execution process.
Based on the scale information and the voice information analyzed by 0, it is possible to process as an input from the equivalent keypad 42 according to the correspondence relation of FIG.

After the voice input from the microphone 49 is equalized and processed as the input from the keypad 42, the operation is the same as that in the normal video game apparatus as described above.

Next, as another embodiment of the present invention, an example in which the karaoke device function and the video game device function are combined to function as a novel game device will be described. That is, based on the voice of the player as the singer, the scale information corresponding to the lyrics of the singer is displayed on the karaoke video, thereby realizing a device that makes karaoke more enjoyable.

When used as the karaoke apparatus described above, the main CPU 30 is the sound CPU 43.
Based on the scale information and sound pressure information of the voice of the singer notified by the above, scale data of the lyrics currently being sung by the singer is created and stored in the object VRAM 36. The CONT 37 for object is based on the timing notified from the main CPU 30 (the timing at which the object image is read by the VRAM 36 for object to be combined with the background image at the timing when the singer is currently singing).
The reading of the object VRAM 36 is controlled.

The object CONT 37 is the main C
Based on the scale information notified from the PU 30 and the currently playing part, object image data to be described later, which should be superimposed on the karaoke background image, is created, and the object V is created.
VRA for object while storing in RAM36
Controls reading from M36.

The priority control circuit 33 is a scroll VRAM.
The background image from 32 and the object image from the object VRAM 36 are combined and input to the D / A converter 34.

The priority control circuit 33 preferentially inputs the object image to the D / A converter when the images are combined. The video signal converted into the analog signal is displayed on the monitor 3
5 is displayed.

FIG. 5 is an operation flowchart of the SYSCONT 41 shown in FIG. The SYSCONT 41 is a control unit for storing external input information in the image processing unit, the SVRAM 32 of the sound processing unit, and the sound RAM 44. First,
Whether or not there is an input from the input pad 42 is confirmed (step S60).

If there is new input information, take it in,
The information is set in the register for notifying the sound CPU 43 and the main CPU 30 (step S6).
1). If there is new song title information in this input information,
In order to notify this music piece number to the center connected via the ISDN line, the ISDN line is captured (steps S62, 63).

When the ISDN line is seized, the center is called, the music number is notified, and the performance information corresponding to this music number is prepared for reception. That is, frame synchronization for error-free reception of performance information, error detection / correction, and sequential storage in the sound RAM 44 under direct memory access control are performed (steps S65, 6).
6).

FIG. 6 is an operation flowchart of the sound CPU 43. First, it is read whether or not there is new input information in the SYSCONT 41 (step S7).
0). If the setting information such as the playing speed, the playing key, and the echo amount has changed, this value is set in the DSP 10 (step S71).

The DSP 10, based on the performance speed information,
It controls the reading speed of the performance information from the sound RAM 44. Performance information is input to the DSP 10 from the sound RAM 44 by direct memory access control. The DSP 10 uses the performance key information to generate sound RA
Digital signal processing is performed so as to control the high / low of the scale of the performance information from M44 (steps S71, 7).
2).

The sound CPU 43 further reads the frequency information of the voice of the singer analyzed by the DSP 10 in response to the voice input from the microphone 49 (steps S74, 7).
5).

In the ROM 58, in addition to the program executed by the sound CPU 43, data for changing the frequency information of the voice to scale information is stored in a table. The sound CPU 58 accesses the ROM 58 based on the frequency information read from the DSP 10 and reads the corresponding scale information.

Generally, it is sufficient to have scale information of about 2 octaves, so that the ROM 58 may have a table of correspondence relationships between voice frequencies and scales of 2 octaves.

The sound CPU 43 also sets the scale information in the SYSCONT 41. The above processing is repeated until the performance of one piece of music is completed or the signal of forced termination is input from the input pad 42.

FIG. 7 is an operation flowchart of the main CPU 30 that performs image processing and overall system control. Like the sound CPU 43, the main CPU 30 also reads various setting information into the SYSCONT 41.

On the basis of this setting information, the karaoke video video temporarily stored in the RAM 53 is read out, and the karaoke image displayed on the monitor 35 through the scrolling VRAM 32 contains the musical score portion corresponding to the part of the music being played. As displayed, SVRAM3 of SCONT31
The timing information for reading 2 is created.

Similarly, the object VR of the object CONT 37 is synthesized so that the scale information of the vocal part currently being sung by the singer is correctly synthesized on the score part.
Timing information for reading the AM 36 is created (steps S100, 101).

Here, in the scroll VRAM 32,
Music score information as shown in FIG. 8 is transferred and stored from the RAM 53 as karaoke video data. Therefore, the scrolling CONT 31 reads out from the scrolling VRAM 32 the karaoke video data having the musical score information corresponding to the part of the song currently being played.

Further, in the object VRAM 36,
Object data read from the ROM 54 by the CPU 30 is transferred and stored. As shown in FIG. 9, this is based on the scale information of the vocal part that the singer is currently singing, and scale data corresponding to this is stored.

This scale data stores only the scale of the part where the singer is currently singing [corresponding to each of FIGS. 9 (a), (b) and (c)]. Therefore, by synthesizing the first measure of the musical notation data shown in FIG. 8 and the scale of FIG. 9 (a), the singer is currently singing and the vocal is:
It is observed that the pitch is not out of order.

Alternatively, by synthesizing the second measure of the musical notation data shown in FIG. 8 and the scale shown in FIG. 9B, it is understood that the vocal currently sung by the singer has a wrong pitch. In this case, the scale data as shown in FIG. 9 which is the object data has a further visual effect when controlled so as to be displayed in a color different from that in FIG. 8 (steps S102, 103, 104).

The above processing is repeated until the performance of one piece of music is completed or the signal of forced termination is input from the keypad 42.

[0081]

As described above, according to the present invention,
Since the display position of the object image on the game machine or the like can be controlled by the voice input, particularly when applied to the video game device, the manual input by the keypad 42 can be changed to the voice input, so that the sense is further different. You can enjoy video games.

Further, when applied to a karaoke device,
You can enjoy the song with a sense of game. That is, as described in the embodiment, when the pitch is removed, it is visually displayed as a musical score image in real time, so that it is possible to make the audience other than the singer interested. Moreover, it seems that the mood will get more exciting.

[Brief description of drawings]

FIG. 1 is a diagram for explaining a case where an analog voice waveform is quantized and converted into a digital voice signal.

FIG. 2 is a principle configuration diagram for obtaining a fundamental frequency of an input voice and a peak level of sound pressure from the digital voice signal of FIG.

3 is a block diagram of an embodiment for realizing an image processing apparatus of the present invention to which the principle of FIG. 3 is applied.

FIG. 4 is a diagram illustrating a correspondence relationship between scale / sound pressure information and input keys used as an embodiment of the present invention.

5 is an operation flowchart of SYSCONT 41 of FIG. 3. FIG.

6 is an operation flowchart of a sound CPU 43 of FIG.

7 is an operation flowchart of the main CPU 30 of FIG.

FIG. 8 is a diagram showing a musical score portion of karaoke video data.

9 is an example of object image data displayed overlaid on a musical score portion of the karaoke video data of FIG. 7.

[Explanation of symbols]

 1 Low-pass Filter 2 Microphone 3 A / D Converter 5 Absolute Value Circuit 7 Peak Detection Circuit 9, 23 Register 11 Zero Cross Detection Circuit 13, 15 Counter 19 Memory 21 Average Value Detection Circuit

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Office reference number FI technical display location H04N 5/272

Claims (3)

[Claims] 1. A scale information creating means for obtaining a frequency of an input voice waveform to obtain scale information of the input voice, and a position where object data should be displayed on a background image based on the scale information to obtain object image data. And a display data creating means for creating a display image data by synthesizing the object image and the background image by giving priority to the object image. Image processing device using the waveform analysis of the. 2. The sound pressure information generating means for detecting a peak value of an input speech waveform and generating sound pressure information according to claim 1, wherein the object image generating means includes sound pressure information and sound pressure information. An image processing apparatus using waveform analysis of an audio signal, characterized in that object image data is created based on information. 3. The image processing apparatus according to claim 2, wherein the combination of the scale information and the sound pressure information is made to correspond to each key of the key input device.