JP2000200088A - Karaoke sing-along machine - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance the actual feel of participation in a KARAOKE sing-along event via motion, such as choreography, by detecting the movements of motion detecting markers disposed on the body surface of a singer and forming and outputting singer motion data, such as choreography motion, in real time in synchronization with KARAOKE sing-along playing in accordance with the detection information thereof. SOLUTION: Color cameras 15 detect the movements of the plural motion detecting markers disposed at the prescribed positions on the body surface of the singer GAL. A singer motion data forming means forms the singer motion data, such as choreography motion, which the singer GAL carries out while singing in real time in synchronization with the KARAOKE sing-along playing. A singer motion data outputting means outputs the singer motion data in synchronization with the KARAOKE sing-along playing. If the singer GAL, thereupon, induces the choreography motion, this motion is transmitted to a computer graphic image CG and the same motion as the motion of the singer GAL is carried out on the screen of a monitor 109.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a karaoke apparatus.

[0002]

2. Description of the Related Art As is well known, a karaoke apparatus is such that when a predetermined music piece is selected, the music piece is played from a speaker, and the lyrics telop of the music piece is displayed on a monitor together with a background image. In a general karaoke apparatus, a moving image related to the contents of the lyrics is often displayed on a monitor as a background image. However, recently, a model called “choreography karaoke” or “flicker” for short has recently been used. It has begun to spread and has attracted the interest of enthusiasts. This displays a computer graphic (hereinafter, computer graphic is abbreviated as "CG") moving image of the choreographic operation of the song instead of a general background image. You can enjoy the fun of choreography. In this case, for example, a CG moving image of an animation character faithfully replicating the dancer's motion can be obtained by capturing the model choreography motion of a professional dancer with a motion capture system and using a known CG technique using the motion data. . This can be used as an example of a choreography operation performed by a user by reproducing a moving image in synchronization with a karaoke performance.

[0003]

However, in the above-mentioned conventional karaoke apparatus, the CG of the choreography operation together with the music performance is performed.
The moving image only flows unilaterally, and the user of the karaoke apparatus is at best satisfied with watching the CG moving image and performing choreography while imitating the moving image. In addition, an enthusiastic fan, such as a specific idol singer, may well be able to perform choreography while dancing and immerse himself in the dreaming world in order to fulfill his idol desire. However, in the above-mentioned conventional karaoke apparatus, it is not a real idol that choreography is performed on the spot, but it is an irreplaceable singer, and it is an fact that it is impossible to obscure it. To do so requires a fairly strong vanity and a sustained imagination. Also, since what is displayed on the display device is a CG movie that captures the movements of professionals who have been sofitified, the qualitative gap from the dance of one's own will be amplified rather than looking at it. Can also occur. In other words, even if you sing while dancing the choreography, there are too many elements to bring the user back to reality, and it is far from being able to experience virtual reality (virtual reality) that made yourself idle. .

On the other hand, if a user is highly motivated, CG
Watching the movements of professional dancers appearing in the video, you may try to practice choreography. In this case, it can be said that ordinary users want to know how much their dance has improved. However, the above-mentioned conventional karaoke apparatus does not have a function of objectively evaluating the content of a user's dance, and none of the conventional karaoke apparatuses has a function of immediately performing the evaluation for each karaoke performance. Japanese Patent Application Laid-Open No. Hei 10-301586 also proposes a device capable of video recording a singing figure of a singer together with a singing voice. It does not evaluate. All of these drawbacks of the conventional karaoke apparatus are rooted in a common cause that the operation of the user himself is not reflected in the function of the apparatus at all.

[0005] An object of the present invention is to compare the karaoke device with a conventional karaoke device by capturing the user's own operation and reflecting it on the function of the device, thereby realizing participation in a karaoke event through operations such as choreography. It can be much more enhanced, for example, performing a rich effect of virtual reality effect such as turning the user into an idol singer, or objectively evaluating the choreographic movement of the user while singing, etc. It is an object of the present invention to provide a karaoke apparatus that can achieve a novel effect not found in the apparatus.

[0006]

SUMMARY OF THE INVENTION The present invention provides a karaoke performance means, a lyrics display means, a microphone for singing while watching the lyrics displayed on the lyrics display means, and an input from the microphone. In order to solve the above-mentioned problem, in a karaoke apparatus provided with a singing voice output unit for outputting a selected singing voice, an operation of detecting the motion of a plurality of motion detection markers provided at predetermined positions on the body surface of the singer Based on the detection means and the detection information detected by the motion detection means, singer motion data which is singer motion data, which is data of motion such as choreography performed by the singer while singing, is created in real time according to the karaoke performance It is characterized by comprising creator means and singer operation data output means for outputting the singer operation data according to the karaoke performance.

In the karaoke apparatus of the present invention, the movement of the movement detecting marker provided on the body surface of the singer (which may be a clothing surface) is detected by the movement detecting means, and based on the detected information. The singer's movement data, such as choreography, is created and output in real time according to the karaoke performance. By utilizing the output singer operation data, it is possible to reflect the operation of the user of the karaoke apparatus in various forms on the function of the apparatus, and for example, a karaoke event through an operation such as choreography. The feeling of participation in the karaoke can be greatly enhanced as compared with the conventional karaoke apparatus.

According to the karaoke apparatus of the present invention, a computer graphic moving image (singer-compatible CG) reflecting the singer operation based on the output singer operation data is provided.
CG moving image synthesizing means for singers for synthesizing moving images),
A singer-compatible CG moving image output means for outputting the singer-compatible CG moving image in accordance with the karaoke performance can be provided. As a result, the following novel effects can be achieved.

For example, if a CG moving image corresponding to a singer is a moving image of a human model such as a singer or a dancer, the movement of the singer himself such as choreography is reflected in the movement of the CG moving image of the singer or the dancer. Will be output and displayed. By singing while watching the CG moving image of the admired singer or dancer move according to his / her own movement, the user can enjoy karaoke as if he / she was the singer or dancer represented by the CG moving image. it can. In other words, the virtual reality effect of making the user himself an idol singer can be greatly enhanced. Also, if a CG moving image of an animated character is used instead of a human model, the user can enjoy singing and choreography while immersing himself in an animation world that can never enter the real world.
You will have a quirky experience.

On the other hand, although it is a somewhat strange idea, for example, a CG moving image of a singer or an animation character of the opposite sex as viewed from the user can be used as a CG moving image corresponding to a singer. For example, in recent years, the number of men who have a hobby as a transvestite is increasing, but it takes considerable courage to actually put the transvestite into practice. It appears that there is a considerable number of aspirants (so-called "hidden transvestites"). Therefore,
For example, a female idol singer's CG video is converted to a singer's CG
If used as a moving image, the female idol singer can enjoy the feeling of a female idol singer without actually dressing up, so that the above-mentioned hidden transvestite applicant can reduce drinking. In this case, the singing voice output unit that outputs the singing voice input from the microphone can be provided with a voice conversion unit that converts the input voice waveform into a voice waveform different from the original voice by modulating the input voice waveform. For example, by converting a male voice waveform into a feminine high-pitched voice waveform, the singing voice is converted into a feminine voice and output, which is more effective.

[0011] A model operation moving image data storage means for storing moving image data of a model operation performed by a professional singer or dancer is provided, and a model operation moving image based on the model operation moving image data is stored as a CG video corresponding to a singer. An image representing the position of the motion detection marker provided on the singer side, or a contour image of the singer synthesized by CG or the like corresponding to the position of the motion detection marker is displayed in a superimposed manner. By operating while looking at this display, the user can check in real time the deviation between the exemplary operation moving image and his / her own operation. As a result, a guideline for motion correction can be obtained, which is very effective in choreography practice and the like.

The karaoke apparatus has an accompanying moving image data storage for storing image data of one or more moving images of a human model or an animation character (hereinafter referred to as an accompanying moving image) to be combined with a CG moving image corresponding to a singer. Means and an accompanying moving image synthesizing output control means for synthesizing the accompanying moving image based on the accompanying moving image data with the singing person corresponding CG moving image and outputting the synthesized image to the singer corresponding CG moving image output means. it can. For example, when performing a choreography operation while singing a song of a popular idol pair or idol group, a specific one of the members is represented by a CG video corresponding to the singer, and the remaining members are synthesized and displayed as an accompanying video. Singing and choreography can be performed happily with the intention of becoming a member of the group. Also, a choreography motion in which a plurality of members fit together can be realistically reproduced by combining the operation of the accompanying moving image with the CG moving image corresponding to the singer, even though the singing is performed alone.

The singer's CG moving image output means can also be used as the lyrics display means. According to this configuration, it is possible to easily confirm the lyrics while confirming the user's choreography operation based on the singer's CG moving image. As a result, it is possible to sing and dance satisfactorily without being disturbed by a song whose lyrics are memorized, and to enjoy a three-dimensional karaoke effect.

Next, in the karaoke apparatus of the present invention, reference operation data storage means for storing operation data (reference operation data) of a singer operation as a reference to be performed by the singer, the reference operation data and the singer A singer motion evaluator for evaluating a singer motion based on comparison with the motion data and a singer motion evaluation result output device for outputting the evaluation result can be provided. By comparing the singer's motion data with the reference motion data, it becomes possible to objectively evaluate a choreographic motion or the like performed by the user. Then, by outputting the evaluation result, for example, the user can quantitatively confirm the ability of his or her own choreography operation, and can greatly refer to the subsequent choreography practice and the like. Further, in a gathering such as a karaoke party, the choreography operation is performed while singing, and the evaluation result is output, thereby greatly enlivening the place. The singer motion data includes information on the position of the motion detection marker at each time,
The reference motion data may include information on a standard position (hereinafter, referred to as a standard marker position) where the motion detection marker should be positioned at each time, and the singer motion evaluation unit performs motion detection at each time. It is determined whether or not the marker position and the standard marker position satisfy a predetermined positional relationship, and points are added and / or subtracted based on the determination result, and the singer's motion is evaluated based on the magnitude of the point value. Can be performed. By evaluating the singer's motion based on the magnitude of the point value, the evaluation result can be made easier to understand.

Next, the motion detecting means sets the singer as a subject, attaches a color marker as a motion detection marker to the subject, or sets a part of the subject as a color marker, and By integrating one or a plurality of color cameras for photographing a color video signal based on a video output from the color camera and integrating color information included in each video signal into a specific number of color groups through a decoding process, A color integrated image data generating means for generating image data having a smaller number of data than the color video signal (hereinafter referred to as color integrated image data); and the generated color integrated image data is designated in advance as a color of a color marker. Color (hereinafter referred to as the marker designated color)
A marker-specified color extracting means for extracting pixel data of the subject; calculating a color marker position of the photographed subject based on the extracted marker-specified color pixel data; and calculating the marker position data obtained as a result of the calculation. Marker position data calculation / output means for outputting as operation data.

A color image of a subject including a color marker captured by a camera is composed of a huge amount of color information of, for example, an RGB video signal, and the amount of information is enormous. It is inconvenient. Therefore, by integrating and aggregating the image information of the color markers on the subject into a specific number of color groups, the amount of information is dramatically reduced, real-time processing is facilitated, and the hardware configuration of the entire karaoke apparatus is simplified. Can be.

The motion detection means uses a color marker as a motion detection marker whose surface is divided into a plurality of areas colored by two or more different marker colors to sing. With a person as a subject, the color marker is attached to the subject, or in a state where a part of the subject is defined as a color marker, a color camera that captures the motion of the subject, and a moving image of the captured subject, Marker image specifying means for specifying, as a marker image, an area in which two or more specific colors corresponding to the marker colors are present in the image data of each frame, and the pixels of the marker pixels constituting the specified marker image Marker position data generation means for generating position data of a marker image represented as a set of marker pixels based on position information on a plane , The position data of the generated marker image for each video frame can be made and a motion data output means for outputting the operation data of the subject.

According to the above arrangement, the color of the marker attached to the body of the subject is such that the surface of one color marker is divided into a plurality of areas colored by two or more different marker colors, An area where two or more specific colors corresponding to the colors are present is specified as a marker image. As a result, markers with different color combinations are identified as different markers even if some areas are colored in the same color, so the number of markers can be significantly increased compared to conventional motion capture systems. Becomes possible. For example, when eight colors are used as identification colors, a maximum of 28 markers can be used for a combination of two colors.
With a marker combining three colors, it is possible to identify as many as 56 types of markers.

The motion detecting means sets a mask area of a predetermined shape and size for the image data of each frame so as to be movable on a pixel plane of the image data. Means, and two or more colors among a plurality of colors set for each pixel are designated as discrimination designation colors for the marker color, and the number of discrimination designation color pixels in the mask area as a pattern condition;
A mask including at least one of the position of the discrimination-designated color pixel in the mask area and, when a plurality of discrimination-designated color pixels are included, at least one of their relative arrangement relations, is set on the pixel plane. Pattern matching determining means for determining whether or not a pixel of each determination designated color in the area matches the pattern condition; mask area scanning means for scanning the mask area on a pixel plane; and mask area at each scanning position And a suitable pixel extracting means for extracting a pixel determined to be suitable for the pattern condition (hereinafter, referred to as a suitable pixel) among the target pixels corresponding to. It can be configured to generate marker position information based on position information on a plane.

In the above configuration, two or more discriminating designated colors are simultaneously designated for a mask area set to be movable on a pixel plane, and pixels of each discriminating designated color coexist in the mask area as a pattern condition. Is defined as mandatory,
In the mask area set on the pixel plane, it is determined whether or not the pixel of each designated color matches the pattern condition. Thereby, 2 in the mask
Since pattern matching (adaptation) is performed in such a manner that pixels of the above colors are simultaneously detected, it is possible to directly and accurately recognize an image in which two or more colors are mixed. In addition, since the discrimination conditions include at least one of the number and position of the discrimination-designated color pixels and, when a plurality of discrimination-designated color pixels are included, their relative arrangement relations, the processing contents Can be simplified, and application to real-time processing can be easily achieved. The number, position, or relative position of pixels may be detected by focusing only on pixels of the same color, or the total number, relative position, or the like of pixels having different colors may be detected.

In this case, one or a plurality of pixels satisfying a predetermined positional relationship with the set mask area are set as target pixels, and a specific output state is defined for the target pixel in accordance with the result of the determination by the pattern matching determining means. Target pixel data generating means for generating target pixel data to be generated. The target pixel data generating means, when it is determined that the arrangement of the pixels of the designated color for determination matches the pattern condition,
While generating data for setting the output state of the target pixel to the first output state, if it is determined to be non-conforming, the output state of the target pixel is changed to a second output state different from the first output state. It is possible to generate data to be set. This configuration is particularly effective when it is desired to recognize a pixel set that meets the pattern condition as a binary image area.

Further, in the moving image processing apparatus of the present invention, the mask area scanning means for scanning the mask area on the pixel plane and the target pixel corresponding to the mask area at each scanning position are determined to be suitable for the pattern condition. A suitable pixel extracting means for extracting a pixel (hereinafter, referred to as a suitable pixel) can be provided. This makes it possible to easily extract a region composed of pixels that meet the pattern condition. More specifically, an extracted image data output unit that outputs the data of the extracted suitable pixels as data of an extracted image represented as a set of suitable pixels is provided. If it is desired to extract two or more types of image areas in a form that is distinguished from each other due to a difference in color or a combination of colors, information for specifying each type is added to the matching pixel data. Can be generated by:

Further, there may be provided extracted image position data generating means for generating position data of an extracted image represented as a set of suitable pixels based on the position information of the extracted suitable pixels on the pixel plane. According to this, by extracting a specific object (for example, a color marker) in a moving image as an extracted image and generating position data thereof,
The movement of an object in a moving image can be easily converted into data.

By integrating and aggregating the color information included in each video signal into a specific number of color groups through decoding of the original video signal, color integrated image data having a smaller number of data than the video signal before decoding is generated. When the color integrated image data generating means is provided, the marker image specifying means
A marker image can be specified in the color integrated image data. For example, by integrating and integrating the number of colors to the same number as the set marker colors, it is possible to save the necessary and sufficient amount of information for marker identification and to reduce the amount of subsequent data. It is easy to respond to

The mask area setting means arranges the color data storage memory sections corresponding to the respective pixels in the mask area in chronological order of pixel transfer, and stores the color data storage memory sections in the color data storage memory sections in synchronization with the pixel transfer clock. On the other hand, it can be configured to include a memory circuit unit with a transfer control function having a transfer control function of sequentially transferring and storing the color data of each pixel from the upstream side of the array. According to this, the setting and movement of the mask on the pixel plane are performed in a hardware manner by pipeline processing synchronized with the pixel transfer clock, so that the processing speed is dramatically improved, and the operation of the singer is photographed. While real-time processing can be easily realized.

The pattern matching determining means has a color data input terminal corresponding to each pixel in the set mask area for binaryly inputting whether or not the pixel is set to the specified color. And a hardware logic determination circuit that generates and outputs a binary determination result in a one-to-one correspondence with a combination of the input states of the color data input terminals. According to this configuration, since the pattern matching determination unit is configured by hardware logic that does not impose a burden on a control unit configured by a CPU or the like, the processing speed is dramatically improved, and the operation of the singer is photographed while being shot. Real-time processing can be easily realized. Note that the hardware logic determination circuit can be configured by, for example, a logic circuit or a programmable logic device (PLD), and furthermore, a look-up table memory or the like.

When two or more types of designated colors are designated, the same number of discriminating circuits as the designated colors can be provided. In this case, the target pixel data generating means includes a plurality of determination result input terminals to which the determination result from the determination circuit for each determination designation color is binary-input, and one-to-one corresponding to a combination of the input states of the determination result input terminals. In a corresponding manner, the present invention can be configured to include a hardware logic data generation circuit that outputs one of the first output state and the second output state as target pixel data. According to this configuration, the target pixel data generation unit sets C
The processing speed is drastically improved because of the hardware logic that does not burden the PU, and real-time processing while photographing the singer's movement can be easily realized.

The moving picture processing apparatus according to the present invention comprises: an information separating means for separating a color video signal into luminance information as luminance information and color information as color component information through decoding processing; And color information correcting means for correcting color information. in this case,
The color integrated image data generating means may generate color integrated image data by mainly integrating and correcting the corrected color information into a specific number of color groups. Thereby, the system configuration is simplified and the system configuration is less likely to be affected by a change in light amount.

A video signal digital conversion means for converting the color video signal into a digital signal; and an image data generating means for generating the image data of the subject by capturing the digitally converted color video signal at predetermined time intervals. Can be provided. In this case, the information separation means
With respect to the color information of the pixel of the image data, the color information is divided into luminance information composed of a single component, and a first color component and a second color component.
And color information composed of color components, and the color integrated image data generating means, when the first color component information and the second color component information are divided into a specific number of groups, respectively, Each combination of the information group and the second color component information group and the integrated color information (hereinafter, referred to as
And a color integration look-up table memory storing the correspondence relationship with the integrated color information. For each pixel, the integrated color information corresponding to the first color component and the second color component is stored in the color integration lookup table. It can be configured such that it is read out from the up-table memory in a character-by-character manner and is used as color data of the pixel. According to this configuration, the integration processing of the two color components of the color image data is performed by
This process can be performed quickly using a look-up table (hereinafter, sometimes abbreviated as LUT) memory, and the amount of data involved in the calculation can be greatly reduced. Can be done quickly.

More specifically, the color information correcting means, when the integrated color information and the luminance information are divided into a specific number of groups, respectively, includes each combination of the integrated color information group and the luminance information group; A correction look-up table memory storing the correspondence relationship with the color information after the luminance correction is provided. For each pixel, the corrected color information corresponding to the integrated color information and the luminance information corresponds to the corrected color information of the pixel. The color data may be read from the correction look-up table memory in a text-based manner. In this configuration, the amount of information can be reduced more effectively by performing correction by incorporating the luminance information into the integrated color information by using the correction look-up table memory. Therefore, the accuracy of the generated marker position data also increases.

In the above-described configuration, after integrating two kinds of color information, the renormalization (correction) of the luminance information is performed. However, the order is reversed, that is, the color information is corrected by the respective luminance. Then, they may be configured to be integrated. Specifically, the color information correction means
For each of the first color component and the second color component,
When each of the color information and the luminance information is divided into a specific number of groups, a first and a second storing a correspondence relationship between each combination of the color information group and the luminance information group and the color information after the luminance correction. A correction look-up table memory for each pixel, and for each pixel, the corrected color component information corresponding to each color information and the luminance information is used as the corrected color component information of the pixel as the corresponding correction lookup table. It shall be read out from the table memory. Then, in the color integrated image data generating means, the first color component information and the second color component information each used after the luminance correction are used.

The extracted image position data generating means for generating the position data of the extracted image represented as a set of matching pixels, calculates and outputs the extracted image position data based on the designated color pixel data for each image data. It can be configured such that the transfer is performed sequentially in synchronization with the transfer cycle of the pixel (for example, the cycle of the transfer clock pulse of the pixel). According to this configuration, the calculation of the extracted image position data of each image frame is performed at a high speed in a pipeline manner in synchronization with the transfer cycle of the pixels (for example, the frequency of the transfer clock pulse is about 8 to 30 MHz). Therefore, for example, in the case of a motion capture system, it is possible to create operation data including marker position data for each frame in real time during shooting. In particular, 10 MHz (for example, 14 to
By using a clock pulse frequency of 15 MHz or higher, a high-speed processing environment particularly suitable for real-time creation of operation data can be realized.

In general, color image data is described in a case where each color component is described as 8-bit (256 steps) data in RGB three-primary color display (the same applies when a luminance component and two kinds of color components are each described in 8 bits). ) ²⁸ ×
2 ⁸ × 2 ⁸ = 16,770,000 patterns of enormous amount of information, and real-time operation is almost impossible with the processing ability of inexpensive CPUs used in personal computers and the like. However, by reducing the amount of information by integrating and aggregating color information as in the above-described method of the present invention, a low-cost C
The real-time operation is also possible in the PU (computer).

In this case, the extracted image position data generating means generates, for each frame of the image data, the sum Σx of the horizontal coordinate components x of the pixels included in the extracted image area and the sum of the vertical coordinate components y similarly. It can be configured to calculate and output extracted image position data including Σy and the total number n of pixels in each area. According to this, based on the obtained data of Σx, Σy, and n, the center of gravity G of the extracted image (for example, the image of the color marker) is calculated as (Σx / n,
Σy / n). Since the center of gravity is not affected by, for example, the rotational movement of the color marker, it is easy to handle as motion data, and is useful in the present invention.

Further, Δx and Δy have the advantage that they can be easily calculated, for example, by diverting the image synchronization control unit always provided in the video photographing apparatus system using the ordinary scanning method. That is, the video signal from the photographing device is decomposed into pixels, separated for each scanning line, and sequentially transferred. At the time of this transfer, the horizontal and vertical synchronization signals are usually inserted into the pixel data string. Therefore, if the pixel transfer clock cycle is constant, the image receiving side receives the horizontal synchronization signal. By counting the number of transfer pixels from, the horizontal coordinate of each pixel on the screen can be specified. Since the transfer pixel count can of course use the pixel transfer clock pulse, it goes without saying that quick processing is possible. On the other hand, by counting the number of scanning lines (e.g., equivalent to the number of horizontal synchronization signals inserted for each scanning line) from the reception of the vertical synchronization signal to the intended pixel position, the vertical coordinates are also calculated. Can be identified. Also, Σ
Since x, Σy and n are all simple addition amounts,
A very simple circuit configuration using an adder enables quick processing, which is advantageous in performing the above-described real-time calculation.

Next, when a lookup table memory is used in the present invention, read target data specifying information such as first and second color component information, luminance information or integrated color information is specified by an address line. By doing so, the read target data specified by the read target data specifying information can be read from the memory cell specified by the address in a text-based manner. Also,
It is possible to provide read control means for sequentially performing read processing of read target data from the look-up table memory for each image data in synchronization with a pixel transfer cycle. By adopting the look-up table memory having the above configuration, the read target data can be quickly accessed by specifying the read target data specifying information by the address line. As a result, high-speed processing synchronized with the pixel transfer cycle is realized, and the above-described real-time processing can be performed without difficulty.

[0037]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the karaoke apparatus according to the present invention will be described with reference to an embodiment shown in the drawings. FIG. 1 is a block diagram showing an example of the overall electrical configuration of a karaoke apparatus 1 according to one embodiment of the present invention. The apparatus is roughly divided into a karaoke apparatus main body 100 and an operation data creation system (singer operation data creation means) 2 connected thereto.
Consists of The operation data creation system 2 is roughly divided into an image input / output unit (image data generation unit) 4, an image extraction mechanism unit 5, an extracted image data output control unit 6, and a marker pixel counting unit (extracted image position data generation unit, operation data The output unit 7 includes four parts, and is connected to the karaoke apparatus main body 100 at an I / O port 9. Also,
The operation data creation system 2 is provided with a clock pulse generation circuit that supplies a pixel transfer clock pulse, and a system controller 3 that performs various controls in the image input / output unit 4.

As shown in FIG. 30, the karaoke apparatus 1
Users of karaoke equipment (singers) in using karaoke
The GAL attaches a plurality of color markers MK at predetermined positions on the body. As the color markers, those which are entirely colored in a single color may be used, but in the present embodiment, the surface of one marker is divided into a plurality of regions colored with two or more different marker colors. Use something. As shown in FIG. 32, in this embodiment, for example, the surface of a spherical marker is divided into four regions by two great circles crossing each other at an angle of about 90 °, and these are alternately changed into two colors. Markers that are painted separately.

The mounting position of the marker MK may be appropriately set in accordance with the degree to which the singer's choreography is accurately captured. For example, in order to obtain motion data capable of expressing the movement of the whole body without discomfort, It is desirable to disperse and attach markers to each part of the body separated by joints. In this embodiment, one location on the head (FIG. 35: M1),
Two places on the torso (one place near the chest (FIG. 35: M2), one place near the abdomen (FIG. 35: M3)), and two places on the left and right arms (one place each with the elbow joint in between (FIG. 35) : M4 to M
7)), markers are attached to the left and right legs at a total of 11 locations, each at two locations (one at each location across the knee joint (FIG. 35: M8 to M11)), but the present invention is not limited to this. For example, more markers may be attached to capture the movement of the singer GAL more precisely.

The marker MK can be attached to the singer GAL by various detachable fixing means. For example, for an arm or a leg, a marker MK is attached to an intermediate portion of a string-shaped or band-shaped flexible member 131 as a detachable fixing means,
A method of wrapping the flexible member 131 around an arm or a leg and tying it, or attaching a marker MK to a ring-shaped rubber band
A fitting method is possible on an arm or a leg. As for the head,
Although the hair band 130 to which the marker MK is attached is fitted, a hat 132 to which the marker MK is attached may be worn as shown in FIG. On the other hand, the torso marker can be fixed to the clothing of the singer GAL using a safety pin 133 as shown in FIG. 32, a double-sided adhesive tape, a pressure-sensitive fastener, or the like as a detachable fixing means. Although not shown, a special suit to which the marker MK is attached from the beginning is prepared at the shop where the karaoke apparatus 1 is installed, and the singer GAL can borrow and wear the suit each time. Conceivable.

Hereinafter, another embodiment of the color marker usable in the present invention will be described. FIG. 24 (a) to (o)
Shows various examples of the marker MK formed in a plate shape. As shown in the figure, various shapes of the marker can be adopted, and the coloring pattern is also divided into parallel divisions ((a), (g), (h)), radial divisions ((b),
(C), (d), (f), (i), (k), (l),
(M), (n)), concentric divisions ((e), (j),
(O)) and so on. this house,
(A), (b), (e), (f), (k) and (n) are 2
The color markers (c), (g), (j), and (l) are examples of three-color markers, and (h), (m), and (o) are examples of four-color markers.
Further, it can be said that it is desirable to use a marker having a maximum dimension of about 2 to 8 cm (for example, about 5 cm) from the viewpoint of preventing the subject from feeling uncomfortable wearing and ensuring the accuracy of marker recognition.

On the other hand, in the case of a plate-shaped marker, for example, if the marker is photographed from the lateral direction, it may enter a blind spot and the marker color cannot be identified. In order to solve such a problem, it is effective to use a three-dimensional marker including those already shown in FIG. In this embodiment, since the color on the marker is set as the marker designated color, it is determined that the marker is detected only when all the colors are detected at the same time.
A coloring pattern in which only a part of the color appears depending on the viewing angle is not desirable because the marker may not be detected. Therefore, it can be said that it is desirable that each color region is formed in a dispersed or intricate manner so that all marker colors can be visually recognized in an arbitrary direction. 25 to 27 show some examples of the three-dimensional marker. For example, in FIGS. 25 and 26, the entire marker is formed in a spherical shape, and in FIG. 25, the spherical surface is completely filled with a hexagonal region and a pentagonal region,
A soccer ball pattern is used.
(A) is an example of a two-color marker, (b) is an example of a three-color marker,
(C) shows an example in which each area is radially divided and further divided into three colors.

FIGS. 26 (a) and 26 (b) show the sphere surface divided by meridian dividing lines V, and FIGS. 26 (c) and (d) An example in which a color or three colors are applied is shown. (E) shows a background area M,
An example is shown in which an irregularly shaped dispersion region D dispersed therein is painted in three colors. The three-dimensional marker is
Not only a spherical shape but also various shapes can be adopted. For example, FIG. 27 shows an example in which a plurality of angular (or needle-like) protrusions have an outer shape protruding from a spherical main body surface, and the outer surface of each protrusion is painted in three colors.

The marker M painted in a plurality of colors as described above.
By employing K, the number of markers that can be distinguished from each other can be significantly increased as compared with the case where a single-color marker is used. For example, consider a case where an arbitrary color is selected from the six marker colors (COR1 to COR6) and used as the marker color.
As shown, only six types of markers equal to the number of colors are possible. In the table, “1” indicates a color to be adopted, and “0” indicates a color not to be adopted. MK1 to MK6 each represent a marker number.

[0045]

[Table 1]

However, in the case of a two-color marker, as shown in Table 2, the number of combinations for selecting from 2 to ₆ ( ₆ C ₂ )
Can be used. Also,
As shown in Table 3, the three-color markers allow ₆ C ₃ = 20 markers to be used. For example, MK # 2 in Table 3 corresponds to COR
It means that the marker surface is separately painted in three colors of 1, COR2 and COR4.

[0047]

[Table 2]

[0048]

[Table 3]

As shown in FIG. 33, the singer GAL wearing the color marker MK has a plurality of (two in this embodiment) color cameras 15 from different directions.
Photographed at As the color cameras 15, for example, well-known CCD cameras can be used. Further, in this embodiment, the microphone 116 for singing is of a hanging type, but may be mounted on a stand or the like (not shown) or a pendant microphone may be mounted on the body. Good. Alternatively, the user may sing while holding the microphone 116 in his hand.

FIG. 2 is a block diagram showing details of the image input / output unit 4. The color camera 15 is a video decoder 16
It is connected to the. As shown in FIG. 10, a video switch 3 operated by a command from the karaoke apparatus main body 100.
Are connected to a plurality of color cameras 15, and when a predetermined color marker on a subject in a certain camera 15 enters a blind spot, the camera 15 is appropriately switched to a camera 15 not in the blind spot. Also, two cameras 15, 1
By using 5, the position of the color marker is photographed in stereo,
The position of the three-dimensional center of gravity of the marker may be calculated.

R (red signal), G of the color camera 15
(Green signal) and B (blue signal) video signals (or video composite signals) are 1
It is decoded into a digital signal in Y, Cr, Cv, which is an expression format. In this embodiment, the number of data bits of each signal is set slightly smaller at 6 bits in order to make the concept of the device easier to understand, but the number of data bits of the signal is naturally limited to this. The basic configuration of the device is not changed at all even if a signal other than 6 bits is used (generally, signals of each color are often 8 bits). Is).

Here, since the image expression format using Y, Cr, and Cv is known, detailed description is omitted, but Y is a luminance signal, and Cr and Cv are specified for chroma and hue. It is a signal of color information, and forms a two-dimensional coordinate space in which hue and saturation are independent parameters (corresponding to first and second color component information). By using such an expression format, colors are represented in a two-dimensional space of hue and saturation, making it easy to specify colors. Also, since the luminance is independent, processing that is not easily affected by lighting can be performed. There are advantages.

The decoded Y, Cr, and Cv digital signals are stored in a look-up table memory group 18 (hereinafter, referred to as a look-up table memory group 18).
The look-up table memory is sometimes abbreviated as “LUT”). The LUT group 18 includes two L
UT circuits 19 and 20 are provided. As is known,
The look-up table memory has a function of registering calculation results for all input data in the memory in advance and outputting the calculation results in a subscripted manner by selectively outputting the registered data using an address line for data input. It has.

Generally speaking, the flow of data processing is as follows. Image data is divided and integrated into a limited number of color groups by the color integration LUT circuit 20, and a predetermined type (for example, 64 types for 6 bits, (8 bits, 256 types). Also, the LUT circuit 19 for incorporating a luminance
Corrects the integrated color data with a luminance signal. First, Cr data and Cv data are input to the color integration LUT circuit 20. Then, they are integrated (aggregated) into multiple types of color groups used as color markers,
The luminance is incorporated into the necessary color groups including the luminance in the luminance-incorporating LUT circuit 19. Here, COR (color)
The information is output as 6-bit color group number information of 1 to COR6.

For example, in the case of 8 bits, Cr
Since the component is 8 bits and the Cv component is 8 bits, there are 60,000 thousands of specified color components, which are divided and integrated into 256 types of color groups. Conceptually speaking, 256 necessary color groups from 1 to 256 are set (registered) in advance, and are registered in the luminance-incorporated LUT circuit 19. All combinations of Cr data, Cv data, and the like are sorted (classified) by the LUT circuits 20 and 19 so as to belong to any of these 256 color groups. In other words, the operation results (256 color groups) of all input data (individual 16-bit data) of color data (color or luminance) are registered in the memory, and registered in advance using address lines for data input. By selectively outputting the data (that is, any one of the 256 color groups), the 16-bit color components are integrated into the 8-bit color group. FIG.
Schematically shows an example of a memory cell portion of the LUT circuit, and 16-bit input data is converted to data A
By selecting an address line corresponding to those contents as the data B and the data B, the registration data DR11, D
R2 の is a two-dimensional table to be read.

Note that a static ram (SRAM) is used for the LUT circuits 19 and 20, and 256 kinds of color data can be arbitrarily set. The SRAM does not require a circuit section for refreshing, thus contributing to simplification of the entire system. For example, input image 24
The amount of information can be reduced by integrating and dividing the color representation of about 16.77 million colors (Y: 8 bits, Cr: 8 bits, Cv: 8 bits) for each color camera and outputting 256 types of color group signals. An easy calculation can be realized by reducing the number to about 1 / 60,000.

Each LUT circuit 19 or 20 has the same hardware configuration, which is shown in more detail in FIG.
It is. 3 constitutes one LUT circuit, and corresponds to, for example, the color integration LUT circuit 20 in FIG. In FIG. 3, the input signals of data A and data B correspond to Cr and Cv, respectively, and are output as 6-bit signals of OD. The LUT memory 160 includes an LUT memory controller 159 (hereinafter, LMCN).
In this embodiment, the D-type flip-flop circuits 161 and 162 are provided on the input side of the data.
However, a D-type flip-flop circuit 164 is provided on the output side (hereinafter, the D-type flip-flop circuit is referred to as a D-latch in this specification). Each of these D-latches has an output control terminal (inhibit input terminal) OC.

The data bus from the karaoke apparatus main body 100 is connected to the L through the bidirectional bus transceiver buffer 166.
Connected to UT memory 160. The address bus from the karaoke apparatus main body 100 has a bus buffer 168,
169 are connected to the address for data A and the address for data B in the LUT memory 160, respectively.

When performing initial setting for the LUT circuit, the karaoke apparatus main body 100 uses the LUT memory control signal (LMCN).
Send a bus request command to LMCN 159 using T). According to this, the LMCN 159 sets the D latches 161, 16
2, 164 output control signals (OC) to make the outputs of the data flip-flops 161 to 165 inactive and to control the output enable signal G of the bidirectional bus transceiver buffer 166 and the bus buffers 168, 169. Then, these outputs are activated. As a result, the karaoke apparatus main body 100 can read and write to the LUT memory 160, and initialization is performed.

After the data input to the LUT memory 160 is completed, the karaoke apparatus main body 100 sends a bus release command (operation execution command) to the LMCN 159 using the LMCNT. As a result, the LMCN 159 controls the output enable signal G of the bidirectional bus transceiver buffer 166 and the bus buffers 168 and 169, and the D latches 161 and 16
2, 164 output control signals (OC), and the outputs of the D latches 161, 162 are activated. As a result, the calculation result obtained by the input data A and the data B to the LUT memory 160 is output to the OD. Here, the D latches 161 and 162
In other cases, high-speed arithmetic processing is realized by using a transfer clock pulse of a pixel to be processed and operating in synchronization therewith. Note that CK in each D latch is a clock pulse input terminal, and DIR of the bidirectional bus transceiver buffer 166 is a bus direction control signal input terminal. As described above, a pixel transfer clock pulse generated by a pulse generation circuit incorporated in the system controller 3 (FIG. 2) is used as the clock pulse.

In this way, the amount of information can be reduced.
Image data compiled into 6-bit information of ~ COR6
In FIG. 2, after being temporarily stored in the frame memory 21,
It is input to the image extraction unit 5 of FIG. Frame memory 21
Then, the video signal output from the camera output by the interlaced scanning is stored in the odd-numbered scanning memory 22 and the even-numbered scanning memory 23, respectively, and is synthesized to perform scan conversion into a sequential scanning signal. The LUT memory group 18
The control of reading and writing data to and from the frame memory 21 is as follows.
Control signals LTCNT and F from system controller
Performed by MCNT.

Hereinafter, the contents of the processing performed by the image extracting unit 5 will be briefly described, and then, how the processing functions will be realized by hardware will be described. The image extracting unit 5 (FIG. 1) determines whether or not the image between the colors exists on the photographed image according to the following flow, and extracts the extracted image data of the marker. Generate and output. That is, as shown in FIG. 12, a mask area (hereinafter, also simply referred to as a mask) MK of a predetermined size and shape is placed on the pixel plane SC of the moving image frame output from the frame memory 21 of FIG. Set to be movable vertically and horizontally on a plane. More specifically, the mask MSK moves horizontally from, for example, the left end to the right end of the scanning line, and when it reaches the right end, moves down by one scanning line and returns to the left end. Is scanned on the pixel plane SC.

For example, consider the case where the mask MSK exists at a certain position on the pixel plane SC as shown in FIG. Here, as the mask MSK, three pixels in the horizontal direction (direction parallel to the scanning line) and three pixels in the vertical direction (direction orthogonal to the scanning line), that is, nine pixels P0 to P9 are arranged in a matrix. Although the arrangement is adopted, the form of the mask MSK is not limited to this. For example, various arrangements may be employed, such as a arrangement in which pixels are arranged in a horizontal line as shown in FIG. be able to.

For example, when a marker of two colors consisting of COR1 and COR2 (for example, as shown in FIG. 24 (a)) is used, if the image is taken, the pixel plane as shown in FIG. On the SC, an image area of COR1 (hereinafter referred to as COR1 area) and an image area of COR2 (hereinafter referred to as COR2 area)
Should appear adjacent to each other. For example, when the mask MSK is located so as to extend over the boundary BD between the COR1 region and the COR2 region, the COR
One pixel and the COR2 pixel are detected simultaneously. On the other hand, when the mask MSK is located outside the boundary BD, COR1 and COR2 are not detected at all, or only one of them is detected. However, even if one color is detected, it cannot be determined whether or not it is derived from the image of the marker. This is because another marker MSK may use the color. Therefore, in such a case, the mask M
K determines that the marker MK has not been detected.
Note that the background portion of the marker MK must be a different color (for example, COR3) from any of the marker colors COR1 and COR2 to be able to be identified as a marker. Therefore, it is obvious that the image data to be used is one in which pixels of at least three colors are mixed.

The simplest way to determine whether the mask MSK has detected the marker MK is to use the mask MSK.
It is sufficient to check whether or not K contains at least one pixel of each marker-specified color. in this case,
If at least one pixel of the designated color exists in the mask, it should satisfy any one of the pattern conditions shown in FIG. In the drawing, hatched pixels indicate pixels of the designated color, and the others indicate pixels of other colors. Therefore, among the color data for each pixel, it is only necessary to sequentially check whether or not the pixel including the color data of the designated color applies to any of the patterns in FIG. 17 for all the colors included in the marker MK. It is. In this check, for example, if the color data of the designated color is binary data, the setting state of the designated color for each of the pixels P0 to P8 is x
It can be easily checked by taking the logical sum of (0) to x (8).

However, if an attempt is made to check only whether or not one pixel of the designated color exists, FIG.
As shown in (a), there is a risk that a marker is erroneously detected under the influence of an isolated pixel PC suddenly generated due to the influence of noise or the like. In this case, for example, it is determined whether at least two or more pixels of the specified color exist in the mask, or as a more severe condition, whether or not a plurality of pixels exist in a form satisfying a specific positional relationship. Try to find out. For example, FIG. 18 shows a pattern condition for checking whether two or more pixels of the designated color are adjacent to any of the upper, lower, left, and right sides. If such a pattern condition is adopted, it is determined that the marker is detected as shown in FIG.
This is a pattern like 9 and, and it is determined that there is no marker. It should be easily understood that by including the diagonally adjacent pixels in the pattern condition, the pattern of FIG. 19 can also be determined as the marker detection.

By performing the above-described pattern matching on the pixel array of the designated color in the mask MSK, if it is determined that the marker has been detected, the mask M
A pixel (target pixel) representing the position of the SK is extracted as a marker pixel (extracted pixel), and data of a marker number that matches the detected combination of the designated colors is output to the address of the pixel. As shown in FIG. 13, the target pixel SPC is defined as the lower left pixel P6 in the mask MSK in the present embodiment, but is not limited to this. If the condition for holding is satisfied, the target pixel SPC is masked MS
It may be set outside K.

For example, when focusing on a specific marker number, if the data of the marker number is stored for the address of a certain target pixel on the pixel plane, the output state of the target pixel becomes the first output state. , And if they are not stored, the output state of the target pixel can be considered to be the second output state. Then, the target pixel in the first output state is the marker pixel of the marker number.

In the case of the marker MK shown in FIG.
When the mask MSK is scanned on the pixel plane SC while performing the matching under the pattern conditions of FIG. 18 using R1 and COR2 as the marker MK designated colors, the marker pixel becomes as shown in FIG.
It is extracted in the form as shown in (b). These marker pixels are
An extracted marker image is formed by arranging three vertically and two horizontally along the left side of the boundary BD in FIG. This extracted marker image is a boundary BD of the image of the original marker MK.
Is obtained by extracting only a part of the region along.

FIG. 22 shows examples of extracted marker images EI for various markers ((a) to (c) are FIGS. 24 (a) to (c), and (d) is FIG. (A) shows the case where each marker is used.) In addition,
G is a marker centroid position calculated from the position coordinates of each pixel of the extracted marker image EI by a method described later.

On the other hand, if the purpose of detecting a marker of a plurality of colors (or use as a motion capture) is deviated and, for example, only one specific color is set as a designated color, a pixel having a certain positional relationship with the mask MSK is maintained. Is set as a target pixel, and when pattern matching is performed, this is set as an extraction pixel, so that the image extraction unit 5 (FIG. 1)
It can be used as a kind of image filter for a designated color (this can be applied to, for example, industrial inspection technology). For example, if the pattern condition of FIG. 18 is adopted, the pattern of “ya” in FIG.
As shown in FIG. 19B, no matter how the mask MSK is aligned, no pattern matching is performed.
As shown in (b), the extracted pixel does not appear, so that the noise is removed. In the other pattern, the same pixel is commonly matched to the mask MSK at a different position, so that the pixel is supplemented and the image is enhanced. For example, the hole opened in the center of the pattern is shown in FIG.
As a result, the extracted pixel EPC appears at the mask MSK position shown in FIG.

FIGS. 20 (a) to 20 (i) show how the filtering process is performed on the pattern of FIG. 19 by scanning the mask MSK. In each of the right diagrams, the mask MSK is represented by a solid line, the target pixel is hatched to the lower right, and the designated color pixel before filtering is indicated by the upper right hatching (the target pixel and the designated color pixel before filtering are indicated by hatching right, respectively). If they overlap, they are both hatched.) Further, in the left-hand drawing, the extracted pixels are represented by hatching falling to the right.
In the step (c), since the pattern does not match any of the patterns in FIG. 18, the state of the extracted pixel has not changed from the previous step (b). The other steps are all cases where pattern matching is performed, and it can be seen that corresponding new extracted pixels are increasing.

The function of the image extracting unit 5 as described above can be realized by, for example, the hardware configuration shown in FIGS. 5 and 7. First, the six color data COR1 to COR6 from the frame memory 21 in FIG. 2 are input to the mask setting processing circuit 30 in FIG. The mask setting processing circuit 30 functions as a mask area setting unit. The data storage memory unit 32 corresponding to each pixel in the mask MSK stores the data in the time series of pixel transfer.
That is, it has a configuration in which the color data (COR1 to COR6: output setting state data) of each pixel is arranged upstream in the data storage memory unit 32 in synchronization with the pixel transfer clock. It operates as a memory circuit unit with a transfer control function having a data transfer control function for sequentially transferring and storing data from the side.

More specifically, a plurality of pixels (pixels arranged in the scanning direction: in this case, three pixels P0 to P2, P3 to P5, and P6 to P8 in FIG. ), The data storage memory sections 32 are configured as one type of shift memory circuit (therefore, they themselves function as a memory circuit section with a transfer control function). ). Specifically, a plurality of memory cells 33 corresponding one-to-one to the pixels arranged in the scanning direction are provided, and each time a pixel transfer clock is received, the color data of the corresponding pixel is sequentially stored while being memory-shifted in the transfer time series. It acts as something to do. The color data of each memory cell 33 of each shift memory circuit 32 is used as the color data of the corresponding pixel in the mask as the data output unit 3.
4 is output. By adopting the shift memory, the pixel transfer relative to the mask MSK in the scanning line direction, that is, the relative movement processing of the mask MSK in the scanning line direction is realized in the form of pipeline processing. Thereby, the mask MSK can be moved quickly on the pixel plane,
This is more advantageous for real-time processing.

The individual memory cells 33 of the shift memory circuit 32 are arranged in series such that the data output terminal is located on the upstream side in the color data input direction, but is connected to the data input terminal on the downstream side. And a D-type flip-flop circuit (D-latch) in which a pixel transfer clock (PCK) is input to each clock input terminal. Then, a data output line 34 as a data output unit is branched from a data output terminal of each D latch. A pipeline processing circuit is realized at low cost by combining the D latches. Reference numeral 31 denotes a D-type flip-flop circuit for input latch of color data.

On the other hand, pixels arranged adjacent to the moving image frame at right angles to the scanning direction (pixels arranged in the scanning orthogonal direction: here, P0, P3, P6, P1, P4, P7, P2, P2 in FIG. 13)
In order to set the mask area in such a manner as to include three mask areas (5 and P8 each), each data storage memory section 32 has a memory area other than the memory area (32 (A)) located on the input topmost side. Line (subsequent memory unit: 32 (B), 32 (C)), which is provided individually for each of the following memory circuits and stores color data of a pixel on a scanning line immediately before a pixel to be stored in each subsequent memory circuit. The memory circuit 35 (LN-1, LN-2) is provided.
These line memory circuits 35 are
The color data of the pixel whose transfer position on the scanning line corresponds to the pixel input to (A) is read from the corresponding memory area, and the corresponding subsequent memory circuit 32 (B), 32 (C) is read.
Forward to By temporarily saving the color data of the pixel one scanning line before in the line memory, the mask setting over a plurality of scanning lines and the moving process thereof can be realized at low cost and at high speed.

In FIG. 5, in order to avoid complication of the drawing, the data output line 34 as a data output unit is shown.
Shows only outputs C10 to C18 related to COR1. FIG. 6 shows a corresponding position in the mask MSK of each output.
Is shown in However, actually, the data output lines 34 are also branched from the outputs C20 to C28,..., C60 to C68 of the COR2 to COR6, and the masks MS of the corresponding colors are respectively provided.
It goes without saying that K is formed.

FIG. 15 shows the flow of data in the mask MSK and the line memory circuit 35 (LN-1, LN-2) when focusing on one specific one of COR1 to COR6.
1 shows the data storage state of the data. In (a), 1
This shows a state where the first three pixels of the second scanning line have entered the mask MSK. In this state, the line memory circuit LN-1 of the succeeding memory circuit 32 (B) is
The data of the scanning line is sequentially written. When one new pixel is transferred in this state, the color data of the oldest pixel in the mask is replaced with the color data of the pixel newly entering the mask MSK because the data storage memory unit 32 is a shift memory. The memory shift is performed in an extruded manner, and the mask MSK moves rightward on the pixel plane by one pixel.

FIG. 15B shows a state in which the pixel transfer of the first line is completed, the horizontal synchronization signal is received, and the pixel transfer of the second line is started. As shown in FIG. 5, each line memory circuit 35 receives the horizontal synchronizing signal HSNC as a reset signal and outputs the horizontal synchronizing signal HSN.
When a pixel of a new line is transferred in response to C, the storage address of the pixel is reset and, as in the line memory circuit LN-1 in FIG. Data is written. Further, as shown in FIG. 5, the data of the line memory circuit LN-1 is serially transferred to the line memory circuit LN-2 and copied. As a result, the line memory circuit LN-2 stores data two lines before as seen from the top memory circuit 32 (A). Then, as shown in FIG. 15C, every time the pixel transfer of the second line advances by one, the corresponding data of the previous line is read out from the line memory circuit LN-1, and is read out from the subsequent memory circuit 32 (B ).

FIG. 15D shows a state in which pixel transfer on the third line has been started. Data transfer from the line memory circuit LN-1 to the succeeding memory circuit 32 (B) is performed as described above. Data transfer from the memory circuit LN-2 to the subsequent memory circuit 32 (C) is performed. FIG. 16E shows a state in which the data of the first three pixels on the third line have entered the mask MSK. As shown in FIG. 16A, the upper left nine pixels on the pixel plane SC have the mask MSK. Corresponding to the shape covered by In this state, each time one pixel is transferred, the mask MSK moves one pixel at a time in the scanning direction. When the mask MSK reaches the right end and receives the horizontal synchronizing signal, it returns to the left end at a position one line lower, Start scanning again. When the frame reaches the lower end of the screen and receives the vertical synchronization signal (VSNC), the frame is cleared, and the same process for the next frame proceeds in real time in synchronization with the moving image frame switching.

The color data of the mask MSK for each color (the number of bits corresponding to the number of pixels in the mask: 9 bits each) is transmitted from the mask setting processing circuit 30 to the pattern shown in FIG. The data is input to a hardware logic determining circuit 41 (pattern matching determining means) in the matching determining / outputting circuit 40. The hardware logic determination circuits 41 are provided by the number corresponding to the types of designated colors (COR1 to COR6), and the mask setting processing circuit 30
, And has a data input terminal 44 (here, 9 inputs) into which the color data of each pixel is binary-inputted. It generates and outputs results.

The hardware logic discrimination circuit 41 can be specifically constituted by a logic circuit. For example, when the pattern condition shown in FIG. 17 is used, the pattern condition can be easily constituted by one 9-input OR circuit. On the other hand, as shown in FIG. 18, when it is determined whether or not the colors of two specific pixels are set as the discrimination designated colors, as shown in FIG. 9-input AND circuit 47 in which the input of the pixel to be formed and the input of the other pixel are inverted.
Can be exemplified by the types of pattern conditions to be examined. Such a logic circuit may be realized by, for example, a programmable logic device (PLD).
Also, a plurality of discriminating circuits corresponding to each pattern condition are incorporated so that any one of the plurality of pattern conditions can be selected and used, and selector inputs (here, 2-bit inputs of SL1 and SL2) are provided. It is convenient if it can be switched by. In this embodiment, the selector input to each hardware logic determination circuit 41 is provided by the karaoke apparatus main body 100 via the register 43.

On the other hand, the hardware logic determination circuit 41
As shown in FIG. 8B, all combinations P1 to PN of the color setting state of each pixel (here, 9 bits, N =
LUT circuit 48 storing the output OP corresponding to (512)
May be used. Here, a two-dimensional LUT circuit is used so that the pattern condition can be switched for each selector input. In this case, the input (and the selector input) of the color data of each pixel from the mask setting processing circuit 30 is LU
An address line is input to the T circuit 48.

Returning to FIG. 7, the judgment result output from each hardware logic judgment circuit 41 (that is, each of the colors COR1 to CORC)
The pattern matching result of OR6) is input to the hardware logic data generation circuit 42. The hardware logic data generation circuit 42 receives the discrimination result from the discrimination circuit of each discrimination designated color as a binary value, and outputs the first output state in a form corresponding to the combination of the input states of the discrimination result input terminals on a one-to-one basis. And the second output state is output as target pixel data. Here, in the case of marker detection, the data of the marker number is output as the first output state, and in the case of not detecting the marker, an output corresponding to none of the marker numbers is output as the second output state. The hardware logic data generation circuit 42 is composed of a decoder of 6-bit input and 5-bit output, and the setting of the marker designation color is selected by the selector inputs SL1 and SL2 (that is, setting of up to four color markers is possible. ), COR1-C
Marker number data is output based on the determination result output of OR6. For example, in Table 2, when COR1 and COR2 are selected as the marker designation colors by the selector inputs SL1 and SL2, ￥, and when the discrimination input between COR1 and COR2 is active, the marker number 1 is output. In this embodiment, CO1 to COR6 of CO1
The R1 side is prioritized so that erroneous encoding due to multiple detection is suppressed.

The data of the marker number thus generated is sent to the extracted image data output control unit 6 and the marker pixel counting unit 7 in FIG. 1 as the data of the extracted image (marker image). As shown in FIG. 9, the marker pixel counting unit 7 includes three adders 55 for calculating the sum Σx of the x coordinate values of the pixels of the designated marker number, the sum Σy of the same y coordinate values, and the number of pixels n. ~ 57. The output total of each adder is stored in the data memory 58, and the result is output to the karaoke apparatus main body 100. The data memory 58 includes a $ x-memory, a $ y memory, and an n memory.

The $ x memory is initially cleared to zero. The sub-controller 54 has a built-in address controller for designating the address of the storage area of the $ x data for each marker color based on the address information from the karaoke apparatus main body 100. Select an area. When the first processed pixel comes in, the marker number data of that pixel is input to the address line of the $ x-memory via the address controller.

Then, based on the horizontal synchronizing signal generated by the image transfer synchronizing control unit (built-in to the system controller 3 in FIG. 2), the x counter 52 operates until the target pixel is transferred. Is transferred to the adder 55 as a horizontal coordinate value of the pixel on the screen. The transfer control of the pixels is controlled according to a clock pulse signal generated by a clock pulse generator in the system controller 3 of FIG. On the other hand, the current value stored in the Σx-memory is fed back to the adder 55, and the count value (ie, the x coordinate value of the new pixel) from the x counter 52 and the cumulative addition value of the Σx-memory are added to the adder 55. 55, and are written again to the same address of the $ x-memory through a buffer or the like (not shown) to update the added storage value.
The processing for one pixel ends.

Next, an adder 56 for calculating Σy of a pixel is paired with Σy-memory. The operation is substantially the same as that of the arithmetic system of が x, except that the input to the adder 56 is from the y-coordinate value of the pixel, that is, from the reception of the vertical synchronization signal from the synchronization control unit to the target pixel position. Is the count output value of the y counter 53 for counting the number of scanning lines (for example, the number of horizontal synchronization signals inserted for each scanning line), and the added output value is the cumulative addition value of the y coordinate. is there.

The adder 57 for calculating n is
It forms a pair with the n memory that stores the output total. The operation is
The operation system is substantially the same as that of the calculation system of Σx, but what is input to the adder 57 is the output of a pixel counter (not shown) that emits one count pulse each time one pixel of the designated marker number enters. The addition output value is the total number n of pixels of the image area (marker image) of the corresponding color marker.

When this series of operations is performed on the entire screen of the currently operating color camera 15 (FIG. 2), the sum x of the x coordinates of each color marker (CM) is stored in the Δx-memory,
Similarly, a state is obtained in which the sum Σy of the y-coordinates is stored in the 、 y-memory and the total number of pixels n is stored in the n-memory as a calculation result. The stored values are respectively transferred to the karaoke apparatus main body 100. When the processing of the image data of one frame is completed, the contents of each memory are cleared by the signal from the frame counter, and the same processing is repeated for the image data of the next frame. Thus,
The values of に よ り x, 及 び y, and n of each color marker are calculated and generated in real time as marker position data with respect to image data sequentially generated by photographing with the color camera 15 and output to the karaoke apparatus main body 100. Becomes

For example, as shown in the flowchart of FIG. 11, the karaoke apparatus main body 100 of FIG. 1 outputs the number (address) of the color marker designated in S1 and S2
In S4, the values of Σx, Σy and n of the color marker are read. Then, by using this, as shown in, for example, FIG. 22, the barycentric coordinate G of the extracted marker image is calculated, and this can be acquired as choreography motion data performed by the singer.

The circuit configuration of the operation data creation system 2 is not limited as long as it is suitable for real-time data creation.
It is not limited to the above.

On the other hand, the extracted image data output control section 6 accumulates the marker number data in the frame memory 21 in a form associated with the pixel address to form marker image data. Transfer according to. The karaoke apparatus main body 100 can perform, for example, shape analysis of the marker image on the basis of the marker image data.

When using markers that are painted in three or more colors, it is also possible to ignore some of the marker colors and apparently recognize them as markers painted in fewer colors. Such processing can be easily performed only by reducing the number of designated colors set in the hardware logic data generation circuit 42 in FIG. FIG. 23 shows an application example of such processing. That is, the marker MK shown in (a) is originally three of COR1 to COR3.
They are painted in different colors, of which COR1 and C
If only OR2 is adopted, it is determined that the area of COR3 does not belong to the marker MK.
Appears near the boundary between COR1 and COR2, and the barycentric position G1 is also calculated as that in the marker image EI1. On the other hand, CO
If only R2 and COR3 are adopted, the area of COR1 is the marker M
Since it is determined that the image does not belong to K, the marker image E
I2 appears near the boundary between COR2 and COR3, and the center of gravity G
2 is calculated as that in the marker image EI2. Since these barycentric positions G1 and G2 have different coordinate values in the original marker MK, for example, G1 on the image
The attitude of the marker MK with respect to the camera can be detected from the values of 1, G2. For example, FIG. 23B shows an example in which the inclination angle θ of the marker MK with respect to the horizontal line X is obtained from the inclination angles of the line segments G1 and G2 with respect to the horizontal line X, and FIG.
Shows an example in which the inclination angle φ in the front-rear direction with respect to the vertical plane z (coincident with the paper surface) is obtained from the length of the line segment G1G2.

Next, returning to FIG.
The electrical configuration of 0 will be described. Karaoke device body 1
00 is an I / O port 9 and a CPU 10 connected thereto.
2, a main control unit 101 mainly including a ROM 103 and a RAM 104; The main control unit 101 further includes a choreography model CG synthesizing unit (singer CG moving image synthesizing unit) 105,
Compressed video decoder 106, lyrics telop synthesizing section 110,
Frame image synthesizing section 107 and music data decoder 11
1 are connected to the I / O port 9 respectively.
A monitor control unit 108 is connected to the frame image synthesizing unit 107, and a monitor 109 serving as a main body of a lyric display unit, a singer operation data output unit, a singer CG moving image output unit, and a singer operation evaluation result output unit. Is connected.

The music data decoder 111 is connected to an amplifier mixer 112, to which a speaker 113 and
The microphone 116 described above is connected. The music data decoder 111, the amplifier 112, and the speaker 113 constitute a karaoke playing means.

The I / O port 9 has an input unit 117 having keys (not shown) for music selection and the like, and a remote controller for receiving a radio signal from a remote controller 119 for performing the input operation by radio remote control. Receiving unit 118 and storage device 120 storing various data necessary for karaoke performance
Is connected.

FIG. 28 schematically shows the contents of the ROM 103, the RAM 104, and the storage device 120. R
The OM 103 stores a control program 103a that controls the entire karaoke apparatus 1. Also, RAM
A work area 104 used when the CPU 102 (FIG. 1) executes the control program 103a is provided in the work area 104.
a, a score counter 104b, and a music selection memory 104c. The storage device 120 stores karaoke data 120a, choreography model CG data 120b, compressed background moving image data 120c, and standard choreography data 120d.

As shown in FIG. 29, the karaoke data 12
0a stores music data, lyrics data, a choreographic index, and a background image index of a karaoke song in association with a song number. The music data is digital compressed sound data obtained by digitizing and compressing audio waveform data of a karaoke performance, or MIDI (Musical Instrument Diode).
Gital Interface) data, and serves as a performance sound source for each karaoke song. The music data is shown in FIG.
Is transmitted to a music data decoder 111, where it is converted into a karaoke performance sound signal. For example, in the case of digital compressed sound data, the music data decoder 111 plays a role of decompressing the digital compressed sound data and further converting it into an analog karaoke performance sound signal. On the other hand, in the case of MIDI data, the music data decoder 111
Is composed of a MIDI sequencer or a MIDI-compatible instrument or synthesizer incorporating the MIDI sequencer.
It plays a role of synthesizing a karaoke performance sound signal based on signal data such as pitch, sound intensity, sound length, and timbre of each sound included in the data. This sound signal is supplied to the amplifier mixer 112.
Is output from the speaker 113 through Also, microphone 1
The singer's voice signal from 16 is mixed with the above karaoke performance sound signal in the amplifier mixer 112 and output from the speaker 113 in the same manner.

Next, the lyrics data describes the contents of the lyrics telop of each karaoke song to be displayed on the monitor 109 using character codes or the like. This is sent to the lyrics telop synthesizing unit in FIG. 1, where it is converted into a video signal of the characters included in the lyrics telop using the character font data corresponding to the character code.

Returning to FIG. 29, the choreography index is given only to karaoke songs that can use the choreography mode, which is also a feature of the karaoke apparatus of the present invention. This is for specifying various data. On the other hand, the background image index is for specifying moving image data of a background image to be sent together with the lyrics telop to the monitor 109 in accordance with the music.

Next, the choreography model CG data 120b is
Model CG data for each song and choreography mode moving image data as accompanying moving image data used in place of the background image when the choreography mode is selected are stored in association with the choreography index. The model CG data and the choreography mode moving image data form a pair, and as shown in FIG.
This is for displaying on the monitor 109 a moving image of a female idol pair PF, which has recently been gaining in popularity (a real image or a processed image thereof, or an animated character using this as a motif). Among them, the model CG data is computer graphic image data for representing a computer graphic image CG (CG moving image corresponding to a singer) of one girl of the pair PF, and the moving image data for choreography mode is the other girl. This is moving image data for representing the moving image (accompanying moving image) SI. By performing coordinate transformation on at least one of the model moving images CG and SI with respect to the three-dimensional moving image data, this is output as a moving image in a state where the viewpoint of the model is changed (for example, when viewed from above or behind). You can also. Thereby, the singer can refer to the movement of the model on the moving image from various angles, so that the singer can check his own movement by looking at the moving image CG, and can check the model movement more closely by looking at the SI, for example. Becomes

As shown in FIG. 35, the computer graphic image CG of the girl has eleven parts P1 corresponding to the color markers M1 to M11 worn by the singer GAL.
To P11, and mark positions G1 to G11 are respectively defined. The model CG data is composed of a set of three-dimensional image data representing these parts P1 to P11. The choreography model CG synthesis unit 105 in FIG. 1 monitors the monitors 109 of the parts P1 to P11 based on the position data of each color marker sent from the motion data creation system 2.
The display position and the angle on the display screen of the computer graphic image C are determined by using the three-dimensional image data.
It plays a role of generating G data for each frame. On the other hand, the choreographic mode moving image data is configured as compressed moving image data, and is decompressed by the compressed moving image decoder 106 in FIG. The frame image synthesizing unit 107 includes a frame memory (not shown) and its peripheral control circuit and the like. As shown in FIG. 38, the computer graphic image CG, the accompanying moving image SI, the lyrics telop image TP Are frame synthesized. The synthesized image information in the frame memory is converted into a video signal via the monitor control unit 108, and the monitor 109 receives a vertical synchronization signal for each frame.
Is output to

Next, in FIG. 29, the compressed background moving image data 120c stores compressed moving image data of various background images in association with a background image index. If the choreography mode is not selected or a song that does not support the choreography mode is selected from the beginning, the corresponding compressed moving image data is read out with reference to the background image index assigned to the song number, and FIG. After being decompressed by the compressed video decoder 106, the image is synthesized with the video TP of the lyrics telop by the frame image synthesizing unit 107, and
Is output to the monitor 109 via

The standard choreographic data 120d shown in FIG.
33, a professional singer or dancer (for example, a girl of the female idol pair PF in question) may wear a color marker MK in exactly the same form as in FIG. 2, which is described as a set of barycentric position coordinates (hereinafter referred to as reference position coordinates) of each marker MK for each frame. As will be described later, this is used when performing a score evaluation for the singer's choreography. Note that FIG.
In FIG. 9, moving image data of a model performing an exemplary choreography operation (hereinafter, referred to as exemplary choreography moving image data: FIG. 29) may be stored in association with the standard choreography data 120d.

The above data can be distributed from the server 122 via the modem 121 and the communication line in FIG. 1 in accordance with, for example, the addition of a new song or a change in the choreography mode correspondence. .

Hereinafter, the operation of the karaoke apparatus 1 will be described with reference to a flowchart. FIG. 41 shows the flow of the music selection process. S101 is a music selection waiting state. When a music number is input by the remote controller 119 or the input unit 117 in FIG. 1 and music selection is performed, the state of the music selection memory 104c (FIG. 28) is confirmed in S103. If the memory is full, the process proceeds to step S104, and a memory full error is notified by displaying a screen on the monitor 109 or the like. On the other hand, if the memory is not full, the process proceeds to S105, and the input station number is written in the last cell of the music selection memory 104c (FIG. 28) formed as a shift memory. The user further inputs by a key operation of the remote controller 119 or the input unit 117 whether to select the choreography mode (S106) or to select the scoring mode (S109). If each mode is selected, the selection and setting of the choreography mode (S108) and the selection and setting of the scoring mode are performed (S110). As shown in FIG. 28, the setting contents are set together with the song number in the song selection memory 1.
04c. If a choreography mode is selected for a song that does not support the choreography mode, the process proceeds from step S107 to step S111, and a notification that the choreography mode cannot be selected is made by displaying a screen on the monitor 109 or the like. Thus, the music selection processing ends.

FIG. 42 shows the flow of the karaoke performance process. First, in S201, the music selection memory 104c
The first area (FIG. 28) is read. If the music number is stored here, the process proceeds to S203, and the storage device 120
To read the karaoke data 120a (FIG. 29) corresponding to the song number (S203, S20).
4). If the choreography mode is not selected in S205, the process proceeds to S206, and the compressed moving image data of the corresponding background image is read out with reference to the background image index (FIG. 29). Then, the process proceeds to S207, where the karaoke performance based on the music data of the karaoke data 120a (S2
07a) is performed, and the lyrics telop based on the lyrics data is also combined with the background image and displayed on the monitor 109 (FIG. 1) (S207b, S207c). The singer sings using the microphone 116 while watching this. When the music performance is completed, the process proceeds to S213, where the music selection memory 104c performs a memory shift, and returns to S201 to repeat the processing of the next music in the same manner. The above is substantially the same processing as that of a known karaoke apparatus.

On the other hand, if the choreography mode is selected in S205, the process proceeds to S208, where the karaoke data 120a
The moving image data for choreography mode (FIG. 28) for creating a necessary accompanying moving image is read out with reference to the choreography index in FIG. If the scoring mode is selected in S209, the process proceeds to S211. If the scoring mode is not selected, the process proceeds to S210. The only difference between these flows is whether or not the scoring process is performed. Therefore, the flow of S211 for performing the scoring process will be described below.

The music performance in S211a and S211b
Is the same as in the case of choreography mode selection. And in the choreography mode,
As shown in FIG. 36, the singer GAL wears the color marker MK on his body and dances choreography in front of the cameras 15 and 15. As already described in detail, the movement of each color marker MK is photographed by the cameras 15, 15, and the marker center position at each time is created as operation data in the operation data creation system 2. FIG. 43 shows the flow of a choreography model CG creation process using this (S211c in FIG. 42). In this case, before starting the choreography operation, the singer G
The AL takes a pose determined in front of the cameras 15 and 15 (for example, a pose of a natural body with its legs slightly opened and both hands laid down as shown in FIG. 30). Motion data creation system 2
Generates the center-of-gravity position data of each marker MK at this time and transfers it to the karaoke apparatus main body 100 (FIG. 1). In the apparatus main body 100, as initial setting processing, landmark positions G1 to G11 of the parts P1 to P11 when a corresponding natural pose is taken with respect to the corresponding model CG data (FIG. 35).
To the center of gravity of each of the markers M1 to M11 on the singer side in the natural pose. As a result, even if the height and body shape differ depending on the singer, the model C
G can be created.

Referring back to FIG. 43, after the initial setting as described above, when the singer GAL starts the choreography operation while singing along with the music, in S301, the barycentric position of each marker is set at the time corresponding to the frame fetch timing. Is taken in from the operation data creation system 2. And S
At 302, the reference positions G1 to G11 of the parts P1 to P11 of the computer graphic image CG (choreography model CG) in FIG.
Is superimposed on the position of the center of gravity of each color marker sent from the motion data creation system 2. And S303
Then, the necessary rotation processing is performed on the three-dimensional image data of each of the parts P1 to P11, and the model (idle pair P
Computer graphic image C of one girl in F)
G data is generated, for example, a choreography model CG synthesis unit 10
5 is stored in the CG memory. Then, when a frame update is instructed by a vertical synchronization signal or the like, S30
Proceeding to 5, the CG memory is cleared, and the process returns to S301 to repeat the same processing for the next frame.

Returning to FIG. 42, the computer graphic image CG synthesized in this way is obtained in S211d.
As described above, the image is synthesized with the accompanying moving image SI (FIG. 36 and the like). The accompanying moving image SI is reproduced irrespective of the movement of the singer GAL in synchronization with the reproduction of the karaoke song. For example, when this movement is substantially the same as the choreography operation to be performed by the singer GAL (that is, in FIG. 34, when two girls of the idle pair PF perform almost the same choreography operation), the singer GAL performs The choreography operation can be performed while referring to this as an exemplary operation. In addition, the model choreography moving image data may be separately prepared in the standard choreography data 120d of FIG.

If the singer GAL performs a choreography operation, this is transmitted to the computer graphic image CG, and the same operation as the singer GAL is performed on the screen of the monitor.
For example, as shown in FIG. 36, when the singer GAL stands still, the computer graphic image CG
Will continue to stand, but if the choreography is performed correctly, the computer graphic image CG will be displayed as shown in FIG.
Also performs a choreography operation in accordance with the singer GAL.

On the other hand, in FIG.
Is executed in parallel with the computer graphic image synthesis processing described above. FIG. 44 shows an example of the processing flow. First, in FIG. 29, standard choreography data corresponding to the choreography index of the karaoke data 120a is read. Then, in S401, the reference position coordinates of each marker of the current frame are read. And S402-S
At 407, as shown in FIG. 39, for all the markers (or some defined markers) MK of the singer,
Judgment area J that satisfies a predetermined positional relationship with reference position coordinates SG
It is determined whether or not the center of gravity MG of the marker MK is included in A, and if it is, the score counter 104b of FIG.
Is added, and if not included, the point is not added, and this process is repeated for all (or a predetermined part of) moving image frames. as a result,
As the number of times that the center of gravity position MG falls within the determination area JA is larger,
The accumulated point value in the score counter 104b increases,
It is determined that the choreography is excellent. Note that a fixed holding point is set, and the center of gravity position MG is determined from the determination area JA.
May be adopted for deducting points each time the value is deviated.

When the music performance is completed, the process proceeds from step S408 to step S409, in which the accumulated point value in the score counter 104b is read out and converted into a score as it is or relative to a predetermined full score (for example, 100). Then, a score is calculated.

Returning to FIG. 42, if the score is calculated, S212
Then, as shown in FIG. 40, for example, as shown in FIG.
To display that point. Instead of displaying the score, the evaluation output may be performed visually (for example, output of an evaluation comment), aurally (for example, applause or booing), or a combination thereof, corresponding to the score value.

In FIG. 42, the process proceeds to S213 to shift the memory, and returns to S201 to repeat the same processing.

As shown in FIG. 45, instead of displaying the computer graphic image CG, the model operation image MI based on the model choreography moving image data described above is displayed, and the singer is displayed as a singer-supported CG moving image. A marker image MJ indicating the position of the attached marker (or
A skeleton image SK obtained by connecting the marker images MJ as shown in (a) or a contour image SH which is fleshed out as shown in (b) may be superimposed on the model operation image MI. Good. In this way, the singer can easily visually confirm how much his or her choreography motion deviates from the model motion, which is convenient for practice and the like.

The karaoke apparatus of the present invention employs a technique such as comparing a singer's singing waveform with a standard waveform.
A known singing evaluation function for evaluating singing can be added. In this case, if a comprehensive evaluation of singing and choreography is performed in combination with the above-described choreography evaluation, a more novel effect can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an overall electrical configuration of a karaoke apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram showing the contents of the image input / output unit.

FIG. 3 is a circuit diagram showing a specific configuration example of a look-up table memory circuit.

FIG. 4 is a diagram conceptually showing data contents of a two-dimensional lookup table.

FIG. 5 is a circuit diagram showing an example of a mask setting processing circuit of the image extracting unit.

FIG. 6 is an explanatory diagram showing a correspondence relationship between a position of a data output line and a pixel position in a mask area.

FIG. 7 is a circuit diagram showing an example of a pattern matching determination / output circuit of the image extraction unit.

FIG. 8 is a diagram illustrating an example in which a hardware logic determination circuit is configured using a gate or a lookup table memory.

FIG. 9 is a circuit diagram showing a configuration example of an extracted image data output control unit and a marker pixel counting unit.

FIG. 10 is a block diagram illustrating an example of a switching circuit of a plurality of cameras.

FIG. 11 is a flowchart illustrating an example of calculating the center of gravity of a color marker.

FIG. 12 is an explanatory diagram showing an example of a mask area setting state on a pixel plane and a marker extraction state based on the state.

FIG. 13 is an explanatory diagram of a mask region.

FIG. 14 is an explanatory diagram showing another example of a mask region.

FIG. 15 is an explanatory diagram of the operation of a mask area and a line memory.

FIG. 16 is an explanatory diagram showing a state of scanning a mask area on a pixel plane.

FIG. 17 is an explanatory diagram showing some examples of pattern conditions when performing one-pixel detection.

FIG. 18 is an explanatory diagram showing some examples of pattern conditions when detecting two adjacent pixels.

FIG. 19 is an explanatory diagram showing some examples of moving image filter processing by pattern matching.

FIG. 20 is an explanatory diagram showing an example of the processing flow.

FIG. 21 is an explanatory view showing another example.

FIG. 22 is an explanatory diagram showing various examples of marker extraction.

FIG. 23 is an explanatory diagram showing a method of recognizing a marker of three or more colors as a marker of a smaller number of colors, and some usage examples thereof.

FIG. 24 is a front view showing various examples of a plate-shaped marker.

FIG. 25 is a front view showing various examples of a three-dimensional marker.

FIG. 26 is a front view showing some other examples.

FIG. 27 is a front view showing still another example.

FIG. 28 is a diagram schematically showing contents of a ROM, a RAM, and a storage device in FIG. 1;

FIG. 29 is a schematic view showing the contents of a storage device in more detail;

FIG. 30 is a perspective view showing an example of a fixed form of a color marker.

FIG. 31 is a diagram showing a modified example thereof.

FIG. 32 is a diagram showing another modified example.

FIG. 33 is a perspective view illustrating a state in which a singer with a color marker attached is photographed.

FIG. 34 is a diagram showing an example of a display mode of a moving image on a monitor.

FIG. 35 is a diagram showing a correspondence relationship between a color marker attached to a singer and a computer graphic image.

FIG. 36 is an operation explanatory view of the karaoke apparatus of the present invention.

FIG. 37 is another operation explanatory view.

FIG. 38 is an exemplary view for explaining how a computer graphic image, an accompanying moving image, and a lyrics telop are combined;

FIG. 39 is an explanatory diagram showing an example of a scoring processing method.

FIG. 40 is an explanatory diagram showing an example of a score display mode.

FIG. 41 is a flowchart showing the flow of an example of a music selection process.

FIG. 42 is a flowchart showing the flow of an example of a karaoke performance process.

FIG. 43 is a flowchart showing the flow of an example of a choreography model CG creation process.

FIG. 44 is a flowchart showing a flow of an example of a scoring process.

FIG. 45 is an explanatory view showing some modified modes of the singer-supported CG moving image.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Karaoke apparatus 2 Motion data creation system (singer motion data creation means) 4 Image input / output control section (color integrated image data creation means) 7 Marker pixel counting section (extracted image position data creation means,
Operation data output unit) 15 Color camera (operation detection unit) 30 Mask setting processing circuit (mask area setting unit, mask area scanning unit) 32 Data storage memory unit (memory circuit unit with transfer control function) 33 D-type flip-flop circuit ( Memory cell) 34 data output line (data output unit) 35 LN-1, LN-2 line memory circuit 40 pattern matching judgment / output circuit 41 hardware logic judgment circuit (pattern matching judgment means) 42 hardware logic data generation circuit ( MSK mask area SC Pixel plane C10 to C18 Pixel SPC Target pixel MK Color marker 100 Karaoke apparatus main body 101 Main control unit (singer motion evaluation unit) 102 CPU 103 ROM 104 AM 105 Choreography model CG synthesizing unit (CG moving image synthesizing unit corresponding to singer) 106 Compressed background moving image decoder 107 Frame image synthesizing unit (combined moving image synthesizing output control unit) 108 Monitor control unit 109 Monitor (lyric display unit, singer operation) Data output means, singer-response CG moving image output means, singer motion evaluation result output means) 110 lyrics telop synthesis section 111 music data decoder (karaoke performance means) 112 amplifier (karaoke performance means) 113 speaker (karaoke performance means) 114 A / D converter 115 Microphone amplifier 116 Microphone 117 Input unit 118 Remote control receiving unit 119 Remote control 120 Storage device (combined moving image data storage unit, reference operation data storage unit) 121 Modem 122 Server

Claims (10)

[Claims] 1. Karaoke performance means, lyrics display means,
In a karaoke apparatus provided with a microphone for singing while watching the lyrics displayed on the lyrics display means and a singing voice output unit for outputting a singing voice input from the microphone, a predetermined position on the body surface of the singer Motion detecting means for detecting the movement of a plurality of motion detecting markers provided in the singer, based on the detection information detected by the motion detecting means, the singer motion being data of a choreography motion or the like performed by the singer while singing. A karaoke apparatus comprising: singer operation data creating means for creating data in real time in accordance with a karaoke performance; and singer operation data output means for outputting the singer operation data in accordance with the karaoke performance. . 2. A singer-compatible CG moving image synthesizing unit that synthesizes a computer graphic moving image (hereinafter, referred to as a singer-corresponding CG moving image) reflecting the singer operation based on the output singer operation data. The karaoke apparatus according to claim 1, further comprising: a singer-compatible CG moving image output unit that outputs the singer-compatible CG moving image in accordance with the karaoke performance. 3. The karaoke apparatus according to claim 2, wherein the CG moving image corresponding to the singer is a moving image of a human model such as a singer or a dancer or a moving image of an animation character. 4. An accompanying moving image data storage means for storing image data of one or a plurality of moving images of a human model or an animation character (hereinafter, referred to as an accompanying moving image) to be combined with the CG moving image corresponding to the singer; 3. An output control means for synthesizing an accompanying moving image based on the accompanying moving image data with the CG moving image corresponding to the singer and outputting the synthesized image to the CG moving image corresponding to the singer. Or the karaoke apparatus according to 3. 5. The karaoke apparatus according to claim 2, wherein said singing person corresponding CG moving image output means is also used as said lyrics display means. 6. A reference operation data storage means for storing operation data (hereinafter, referred to as reference operation data) of a singer operation as a reference to be performed by the singer, and the reference operation data and the singer operation data The karaoke apparatus according to any one of claims 1 to 5, further comprising: a singer motion evaluation unit that evaluates the singer motion based on the comparison; and a singer motion evaluation result output unit that outputs the evaluation result. 7. The singer motion data includes information on the position of the motion detection marker at each time, and the reference motion data includes a standard position at which the motion detection marker should be positioned at each time. (Hereinafter referred to as a standard marker position), and the singer motion evaluation means determines whether the motion detection marker position and the standard marker position satisfy a predetermined positional relationship at each time. 7. The karaoke apparatus according to claim 6, wherein the karaoke apparatus is configured to make a determination, add and / or subtract points based on a result of the determination, and evaluate the singer's movement based on the magnitude of the point value. 8. The motion detection means, wherein the singer is a subject, and a color marker as the motion detection marker is attached to the subject, or a part of the subject is defined as a color marker,
One or a plurality of color cameras for photographing the movement of the subject, and a color video signal based on a video output from the color camera are integrated with color information included in each video signal into a specific number of color groups through decoding processing. A color integrated image data generating means for generating image data having a smaller number of data than the color video signal (hereinafter referred to as color integrated image data); A marker designation color extracting means for extracting pixel data of a color designated in advance as a marker color (hereinafter referred to as a marker designation color); and a color marker of a photographed subject based on the extracted marker designation color pixel data. A marker position for calculating a position and outputting marker position data obtained as a result of the calculation as motion data of the subject Karaoke apparatus according to any one of claims 1 to 7 and a chromatography data calculation and output unit. 9. The motion detecting means uses, as the color marker as the motion detection marker, a color marker whose surface is divided into a plurality of regions colored with two or more different marker colors. With the singer as a subject, either attaching a color marker to the subject or defining a part of the subject as a color marker, a color camera for photographing the motion of the subject, A marker image specifying means for specifying, as a marker image, an area in which two or more specific colors corresponding to the marker colors exist in the image data of each frame of the moving image; and forming the specified marker image. Generating position data of a marker image represented as a set of the marker pixels based on the position information of the marker pixels on the pixel plane 9. The karaoke machine according to claim 1, further comprising: marker position data generating means; and motion data output means for outputting position data of the marker image generated for each moving image frame as motion data of the subject. apparatus. 10. A mask area setting means for setting a virtual mask area of a predetermined shape and size for image data of each frame so as to be movable on a pixel plane of the image data. 1 or 2 among a plurality of colors set for the pixel
The above are designated as the discrimination designated colors for the marker color, and the number of discrimination designated color pixels in the mask area, the position of the discrimination designated color pixels in the mask area, and the plurality of discrimination designated color pixels are specified as the pattern conditions. In the case where it is included, a pixel that includes at least one of their relative arrangement relations is determined in advance, and in the mask area set on the pixel plane, the pixel of each discrimination designated color matches the pattern condition. Pattern matching determining means for determining whether or not there is a mask area; mask area scanning means for scanning the mask area on the pixel plane; and determining that the target pixel corresponding to the mask area at each scanning position matches the pattern condition. Pixel extraction means for extracting the extracted pixels (hereinafter referred to as “adapted pixels”). Wherein the karaoke device according to claim 9 and generates the marker position information based on position information on the pixel plane adaptation pixels.