JP5800247B2 - Karaoke system and karaoke device - Google Patents

Description

  The present invention relates to a karaoke system for synthesizing and reproducing a plurality of in-performance images photographed in synchronism with music reproduction and a karaoke apparatus used therefor.

  A normal musical piece is a form in which a plurality of performance parts (singing parts) are performed in synchronization, and in recent karaoke systems, one specified performance part is excluded for the purpose of singing a specific part or practicing musical instrument performance. Minus-one performances that play only with other performance parts are widespread.

  In Patent Document 1, the state of a player who is performing the minus one performance is photographed with a camera or the like on the karaoke apparatus side, and the images and images (hereinafter referred to as images) during the performance are transmitted to a server via a communication network such as the Internet. A technique for recording, accumulating, and synthesizing so as to collaborate by extracting images during performance corresponding to the same music piece is disclosed. As a result, it is possible to synthesize a co-starring image that appears as if the performers of different performance parts are co-starring.

JP 2012-73428 A

  Conventionally, when the co-star images are combined into a plurality of images, the co-star images are combined in an arrangement in which the extracted images during performance are arranged in time series in the respective recording orders. However, when a musical piece is co-starred in an actual live performance or the like, there are many cases where the standing position of the musical instrument player corresponding to each part is customarily determined. As described above, the co-star images in which the playing images of each part are arranged in the order of recording as described above, while there is customary arrangement regularity of each part according to the genre and organization form of the music to be played, etc. I felt uncomfortable when I saw it.

  The objective of this invention is providing the karaoke system and karaoke apparatus which can implement | achieve natural arrangement | positioning on a musical instrument performance in a co-starring image.

In order to achieve the above object, the present invention provides a part type timbre designation that automatically designates a part type of a minus one performance part to be subjected to minus one performance with respect to a predetermined music from a pronunciation of a musical instrument. A karaoke apparatus comprising: means; and a part image photographing means for photographing a performing image of a performer of the minus one performance part; and the playing image photographed by the part image photographing means of the karaoke apparatus; And a server having part image recording means for recording in association with the piece identification information of the piece of music and the part type information of the minus one performance part, wherein either the karaoke apparatus or the server On the other hand, when performing a minus one performance by specifying a song and a minus one performance part, the song identification of the designated song And the performance image already recorded corresponding to the part type information of the part other than the specified minus one performance part is extracted from the part image recording means, and the extracted performance image is extracted. A co-star image synthesizing unit that synthesizes co-star images arranged in a grid pattern according to a predetermined arrangement rule, and the co- star image synthesizing unit determines the extracted images during performance according to their number and part type information. The position and size in the display area of the co-star images are arranged and synthesized to be changed, and the display position of the playing image representing the performance by the instrument designated by the part type tone designating means represents the performance by vocal. It determines an image being played as a reference, the karaoke apparatus, played image display means for displaying the played image synthesized by the co-star image synthesizing means Characterized in that it comprises.

In order to achieve the above object, the present invention provides a part type timbre designation that automatically designates a part type of a minus one performance part to be subjected to minus one performance with respect to a predetermined music from a pronunciation of a musical instrument. Means, a part image photographing means for photographing the performance image of the performer of the minus one performance part, and the performance image photographed by the part image photographing means with respect to an external server. A karaoke apparatus having a song identification information and an in-playing image transmission means for sending the recorded information in association with the piece type information of the minus one playing part, wherein the song to be played minus one and the minus one playing part are designated. The part type of the part corresponding to the music identification information of the specified music and other than the specified minus one performance part. In response to distribution to extract the performance in an image that has already been recorded, the performance in an image obtained by these extracts, the position and size in the display area of the played image depending on their number and part type information If the performance image includes a vocal performance and a guitar performance, the performance image representing the guitar performance is more to the right than the performance image representing the vocal performance. A co-starring image requesting unit for requesting the server to receive a co-starring image that is arranged and synthesized so as to change the position and size in the display area so as to be arranged, and the co-starring image is received from the server and displayed. Co-starring image display means ,
It is characterized by having.

  According to the present invention, it is possible to realize a natural arrangement on musical instrument performance in a co-star image.

It is explanatory drawing which shows the main structures of a karaoke apparatus. It is a functional block diagram which shows the main structures of the control system of a control apparatus. It is a figure which shows the specific process of the characteristic process in a karaoke system. It is a figure explaining the timbre recognition process performed when setting a performance part. It is a figure which shows the basic form of the arrangement | positioning rule of each performance part at the time of a collaboration image being synthesize | combined. It is a flowchart which shows the processing content each performed by CPU of a karaoke apparatus and CPU of a server at the time of performing a minus one performance. It is a figure which shows the data format of each information transmitted / received and recorded in a flow, and those relationship. It is a figure which shows the example of the arrangement | positioning pattern in the case of changing suitably the position and magnitude | size in those collaboration images according to the kind and number of performance parts.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, a case will be described in which the user mainly plays the electric guitar by himself / herself along with the performance of the karaoke apparatus.

<Main configuration>
FIG. 1 is an explanatory diagram showing a main configuration of a karaoke system including a karaoke apparatus according to the present embodiment. As shown in FIG. 1, a karaoke apparatus 10 includes a monitor television (hereinafter abbreviated as a monitor) 13 for displaying lyrics telop, co-starring images described later, various operation screens, etc. on a CRT, a small monitor 14, and the like. Karaoke music composed of a musical instrument connection board 8 for connecting various electronic musical instruments (only an electric guitar is shown) including an electric guitar 4, an electric bass, an electronic drum, a keyboard, etc. (Referred to as “song” as appropriate) and music selection control such as reservation of playback of the music, transmission of a request signal to the server 58 including a request for transmission of the selected music, and the music indicated by the request signal Communication control such as reception of response signals including corresponding music data, etc., musical instrument of the type to be played minus minus (to be described later) A control device 20, such as specifying the guitar 4), is provided.

  Further, in this example, the karaoke apparatus 10 includes a mixing circuit 9 that mixes a musical instrument performance signal input from the musical instrument connection board 8, an audio signal input from the microphones 17 and 18, and a music reproduction signal. An amplifier 16 that performs volume balance between sound and music, echo adjustment, delay adjustment, amplification of mixing signal, pitch control of reproduced music, control of treble and bass, and the amplified signal output from this amplifier 16 As a floor type set of speakers 11 and 11, a set of speakers 12 and 12 for hanging the ceiling, a remote control 30 for remotely operating the control device 20, and a performance of a singer or a minus one performance to be described later And a camera 25 (corresponding to part image photographing means) for photographing a person.

  Although not shown, the musical instrument connection board 8 has input terminals (phone jacks) for connecting the output terminals (phone plugs) of the electric guitar 4 and an electronic drum, keyboard, electric bass, etc. (not shown), and input terminals. And an output terminal for outputting performance signals input to the. The musical instrument connection board 8 automatically detects that each musical instrument is connected by connecting the output terminal of each musical instrument to the input terminal, selects one performance signal of the connected musical instrument, and selects the selected performance signal. The performance signal can be output to an input terminal 44 of the control device 20 (see FIG. 2 described later). For example, a performance signal output from the electric guitar 4 is input to the musical instrument connection board 8 via a phone jack 4 a (see FIG. 2 described later) for connecting a phone plug of the electric guitar 4, and is output from the output terminal of the musical instrument connection board 8. The output performance signal is input to the input terminal 44 of the control device 20.

<Control system>
The configuration of the control system of the control device 20 will be described with reference to FIG. FIG. 2 is a functional block diagram showing the main configuration of the control system of the control device 20. The control device 20 includes a communication terminal 40 that connects the LAN line 15 to the device housing, an input terminal 44 that is connected to the output terminal of the musical instrument connection board 8, and an audio output terminal 41 that is connected to the audio input terminal of the amplifier 16. The video output terminal 42 connected to the video input terminal of the monitor 13 and the video output terminal 43 connected to the video input terminal of the monitor 14 are provided. The control device 20 is provided with a CPU 45 that executes various controls according to a control program. The CPU 45 includes a RAM 46 for temporarily storing data transmitted from the remote controller 30, song name data as song identification information of the selected song, song title data of a reserved song, a program executed by the CPU 45, and necessary A ROM 47 in which various data tables are stored is connected.

  Further, the CPU 45 has a video RAM 48 in which character video data for displaying lyrics telop and various message videos on the monitors 13 and 14, music data transmitted from an external server 58, and video data of co-star images ( A LAN board 50 for receiving various data such as lyrics data and video data via the communication terminal 40 and the communication network 15, and music data received by the LAN board 50 and moving images of co-star images. A RAM 49 for temporarily storing data and the like is connected.

  Further, the CPU 45 is connected to a MIDI sound source board 51 that inputs MIDI data included in the song data read from the RAM 49 and outputs a sound source signal from a sound source specified by the input MIDI data. . Further, the CPU 45 receives the output sound source signal and converts it into a signal that can be amplified by the amplifier 16, and a CD-ROM player 53 that reads background video data indicating a general background video separately. And are connected.

  Further, the CPU 45 is connected to a video control circuit 54 that performs video control for fitting and displaying the above-mentioned co-star image received from the server 58 and a minus one performance image captured by the camera 25.

  Further, the CPU 45 includes a conversion circuit 55 that converts an optical signal from the remote controller 30 received by the light receiving unit 38 of the control device 20 into a digital signal, and various buttons and keys 39 (tenkeys) provided on the housing of the control device 20. From the camera 25, a display circuit 56 that outputs a display signal to the LED 37 that lights when the music selection button is pressed, an input circuit 57 that inputs a switching signal generated when the various buttons and key 39 are pressed, and so on. Is connected to an input circuit 59 for inputting the photographic signal. The whole of the karaoke apparatus 10 and the server 58 corresponds to the karaoke system (karaoke system 1) described in each claim.

<Features of this embodiment>
As described above, the karaoke system 1 of this embodiment includes the karaoke apparatus 10 and the server 58 that are connected to each other via the communication network 15 so as to be able to send and receive information. In addition, a normal musical piece is a form in which a plurality of performance parts are played in synchronization, and in the karaoke system 1 of the present invention, a minus one performance in which only the other performance parts other than one performance part designated by the user are played together. I do. Then, the part type of the minus one performance part to be subjected to the minus one performance is designated by manual operation of the remote controller 30 of the karaoke apparatus 10 for a predetermined music specified separately, and the performance of the minus one performance part is designated by the camera 25. Take a picture of the person performing. When the performance is finished, the moving image data of the captured performance image is transmitted to the external server 58, and is associated with the song name of the designated music and the part type information of the minus one performance part. The moving image data is recorded in a large-capacity storage device 58a (equivalent to part image recording means) such as an HDD provided in the server 58.

  By recording and accumulating images during performance corresponding to each part type in the server 58 in this way, when performing a minus one performance by specifying a song name and a minus one performance part again after the next time, a co-star image (hereinafter referred to as a collaboration image). Image, and appropriately abbreviated as a collaboration image). Specifically, the moving image data of the performance image already recorded corresponding to the part type information of the part other than the designated minus one performance part corresponding to the music title of the designated music is stored in the large capacity of the server 58. A collaboration image can be synthesized by combining the extracted moving image data of the performance image extracted from the storage device 58a. This combining process itself may be performed by either the karaoke apparatus 10 or the server 58, but in this example, it is performed by the server 58. The collaborative image is displayed on the monitors 13 and 14 on the karaoke apparatus 10 side, so that the performer of the minus one performance is performed together with the performers of other parts (hereinafter referred to as a collaboration performance, and appropriately abbreviated as a collaboration performance). The visual effect is obtained.

  As a comparative example, a technique of synthesizing a collaboration image in an arrangement in which the extracted images during performance are arranged in time series in respective recording orders has been taken. However, when a musical piece is collaborated in an actual live performance or the like, there are many cases where the standing position of the performer corresponding to each part is conventionally determined.

  For example, when actually performing rock music in a band format, the vocal is located in the center of the stage, the bass is on the left side, the guitar is on the right side, and the drum is on the back side of the vocal. Each is generally located. The reason for this customary arrangement of each part is, for example, the historical theory that the band first became famous for the arrangement of such a part organization (such as the Beatles), and the drum that plays the role of rhythm. There are various theories such as the ease of taking the base eye contact, the high ears of the guitar with the right ear, and the theory that it is easier to hear the bass bass with the left ear. In (especially rock music), the positions of various musical instruments displayed in an image obtained by photographing the performance state are visible to the audience as a state in which the left and right and the vertical relationship with each other are determined.

  In this way, the performance images of each part are arranged in the recording order as described above, while there is customary arrangement regularity of each part according to the genre of the music to be played and the organization of the performance parts. I felt uncomfortable when I saw the collaboration image.

  On the other hand, in the karaoke system 1 of the present embodiment, the playing image extracted from the large-capacity storage device 58a of the server 58 is displayed in accordance with a customary arrangement rule according to the genre or organization form of the music as described above. The collaborative images arranged in a shape are synthesized.

<Specific processing contents>
In FIG. 3, the specific process of the said characteristic process in the karaoke system 1 of this embodiment is shown. First, as shown in FIG. 3A, the user designates the performance part by causing the musical instrument of the performance part to be subjected to minus one performance to sound. At this time, a display prompting the user to sound the musical instrument to be subjected to the minus one performance is displayed on the monitors 13 and 14, and the user is connected to the musical instrument connection board 8 (the electric guitar 4 in this example). Is pronounced. As a result, the control device 20 to which the sound generation signal of the musical instrument is input via the musical instrument connection board 8 recognizes the type of the connected musical instrument by the timbre recognition processing (see FIG. 4 described later), and the corresponding performance part (instrument) Name) as the type of minus one performance part. At this time, a song name is also designated on a separate operation screen (not shown). As a result of the reservation operation performed in this way, the karaoke apparatus 10 transmits a request signal including information on the music title and the performance part to be subjected to minus one performance to the server 58 (see A in FIGS. 6 and 7 described later).

  Receiving the request signal, the server 58 extracts from the large-capacity storage device 58a the performance-in-progress images of the parts other than the designated performance part corresponding to the designated music title, and synthesizes the collaboration image using them. This composition may be performed in advance by the server 58. The moving image data of the collaboration image is returned as a response signal to the karaoke apparatus 10 together with the song data (MIDI data) corresponding to the designated song name (see B in FIGS. 6 and 7 described later).

  The karaoke apparatus 10 that has received the response signal from the server 58 reproduces the song data together with the moving image data of the collaboration image received from the server 58 as shown in FIG. In the example of the collaborative image shown in the figure, only the already-recorded vocal playing image is synthesized, and the arrangement location (upper right) corresponding to the target guitar for the minus one performance is blank.

  During the performance, as shown in FIG. 3C, the performance image of the performer of the guitar 4 in this example photographed by the camera 25 is displayed in the blank arrangement position. During this time, a performance image captured by the camera 25 is recorded, and when the performance of the music is finished, the video data of the recorded performance image is transmitted to the server 58 and stored in the mass storage device 58a. It is recorded (see D in FIGS. 6 and 7 described later). Thereby, when the same music is reserved for minus one performances other than vocals and guitars after the next time, a collaboration image of vocals and guitars is synthesized and displayed.

  Here, the timbre recognition processing performed when setting the performance part in FIG. 3A will be described with reference to FIG. The tone generation signal of the instrument input to the control device 20 via the instrument connection board 8 is a signal that periodically repeats a complicated waveform unique to the instrument. The repetition frequency determines the pitch of the sound, and the unique waveform determines the tone color of the instrument. In the example of the present embodiment, the control device 20 includes a band filter group (BPF in the drawing) having different frequencies, and discriminates one sound generation signal according to the frequency band of each band filter to analyze each amplitude level. In addition, level distribution data for each frequency band of the timbre used as a model for each instrument is separately prepared as comparison data.

  The control device 20 compares the level distribution data for each band, which is the analysis information of the input sound signal, and the level distribution data for each band of the comparison data, and the musical instrument that is the sound signal source from the comparison data with the best level distribution. Can be specified. Through such tone color recognition processing, the karaoke apparatus 10 automatically designates the part type of the minus one performance part from the pronunciation of the musical instrument. Vocal or chorus may be specified as a minus one performance part.In this case, the vocal type signal is input instead of the musical instrument performance, and the part type is determined by the frequency analysis of the same timbre recognition process. Specific to chorus. In addition, in order to ensure the accuracy of the comparative analysis, it is preferable that the sound of the same reference height (for example, A sound of 440 Hz) is sounded when any type of musical instrument is sounded. Further, the analysis of the level distribution for each band of the sound generation signal may be performed by spectrum analysis such as so-called fast Fourier transform.

  Further, the arrangement of the performance parts when the collaboration image is synthesized by the server 58 is assumed to follow the basic form of the arrangement rule as shown in FIG. 5, for example. In the example shown in the figure, the entire collaboration image is divided into a grid pattern of 2 rows × 3 columns. The base is in the first row and the first column, the vocal in the first row and the second column, the guitar in the first row and the third column, and the second row. A chord is assigned to the keyboard in the first column, the drum in the second row and the second column, and the chorus in the second row and the third column. With this arrangement, the bass is placed on the left side, the guitar is placed on the right side, and the drum is placed on the back side in a pseudo manner, as if it were the actual live performance stage. Close placement relationship.

In the example of this embodiment, the position of the performance image of each performance part in the display area of the collaboration image is fixed to the specified position in the basic form as it is, and is displayed in the same size. For performance parts for which no images are being recorded on the server 58 yet, the part names as shown in the figure may be displayed at the corresponding positions, or the images of the corresponding musical instruments may be displayed, or as blanks. Also good. As described above, on the server 58 side, the position of the performance part that is the subject of the minus one performance this time is blank, and other than that, the performance image of the performance part that can be extracted from the large-capacity storage device 58a is combined. Synthesize (may be synthesized in advance). On the karaoke apparatus 10 side, the playing image of the minus one performance photographed by the camera 25 is inserted into the corresponding position in the collaboration image and reproduced.
The fitting of the other image into one image, or the fitting of the image, may be performed using a technique such as video overlay or superimposing, and is not described in detail because it is well known.

  The basic form of the arrangement of each performance part shown in FIG. 5 is an example corresponding to the rock music in which the reserved music is played in a band format. You may prepare the basic form according to the usual arrangement rule.

<System control procedure>
When the user plays the electric guitar 4 while reproducing the karaoke music as described above, the processing contents respectively executed by the CPU 45 of the karaoke apparatus 10 and the CPU (not shown) of the server 58 are shown in FIG. The flowchart will be described.

  In FIG. 6, this flow is started when, for example, the user presses the power button of the control device 20 of the karaoke device 10 to turn on the power of the control device 20. In conjunction with the rising of the power supply of the control device 20, the power supply of the amplifier 16, the musical instrument connection board 8 and the monitors 13, 14 is started up. Note that, as described above, in this example, a case where the user performs with the electric guitar 4 connected to the musical instrument connection board 8 will be described. The server 58 is always on. Further, in the figure, in order to avoid the complexity of illustration, the contents related to the music data are omitted, and each term is abbreviated.

  In FIG. 6, first, in step SK10, the CPU 45 accepts a reservation operation for a piece of music that the user wants to play minus one. Specifically, an operation in which the user inputs the song name or song selection number of the song that the user wants to play minus one with the remote controller 30 is accepted.

  Next, the process proceeds to step SK13, where the CPU 45 of the karaoke apparatus outputs a display for urging the user to play the musical instrument to play minus one on the monitors 13, 14, and inputs the musical instrument sound signal via the musical instrument connection board 8. Accept.

  Next, the process proceeds to step SK17, and the CPU 45 of the karaoke apparatus specifies the type of musical instrument of the sound generation signal source as a minus one performance part by the frequency analysis of the timbre recognition process described with reference to FIG.

  Next, the process proceeds to step SK20, and the CPU 45 of the karaoke apparatus requests a collaboration image request signal including the reserved music received in step SK10 and the minus one performance part information specified in step SK17 (see A in FIG. 7 described later). ) To the server 58.

  Next, proceeding to step SS10, the CPU of the server 58 that has received the request signal is moving image data corresponding to the minus one designation of the request signal, that is, the playing image of the designated minus one playing part corresponding to the designated music piece. The collaboration image from which the image is missing is acquired from the mass storage device 58a. Then, the acquired moving image data of the collaboration image and the separately acquired music data are returned to the karaoke apparatus 10 of the request source (request signal transmission source) as a response signal (see B in FIG. 7 described later).

  Next, moving to step SK30, the CPU 45 of the karaoke apparatus downloads the moving image data and music data of the collaboration image, which is a response signal from the server 58.

  Next, the process proceeds to step SK40, and the CPU 45 of the karaoke apparatus determines whether minus one performance is designated in step SK10. In particular, when the minus one performance is not designated or when the vocal is designated as the minus one performance part, the determination is not satisfied and the routine goes to Step SK50.

  In step SK50, the CPU 45 of the karaoke apparatus sets so that the main vocal is designated as minus one in the same manner as in a normal karaoke performance. Then, the process proceeds to step SK70.

  On the other hand, if it is determined in step SK40 that a minus one performance part is designated, the determination is satisfied, and the routine goes to step SK60.

  In step SK60, the CPU 45 of the karaoke apparatus performs setting so that the musical instrument or chorus designated by the OFF operation is designated as minus one. Then, the process proceeds to step SK70.

  In step SK70, the CPU 45 of the karaoke apparatus synchronously reproduces the moving image data and the music data of the collaboration image downloaded in step SK30, and at the placement location corresponding to the song / instrument set as minus one in the collaboration image. Is displayed. It also records camera images.

  Next, the process proceeds to step SK80, and the CPU 45 of the karaoke apparatus transmits the recorded data (see D in FIG. 7 described later) to the server 58. Specifically, the main contents of the recording data to be transmitted are the moving image data of the playing image recorded by the camera 25 this time, the musical instrument information (corresponding to the part type information) which is the performance part played this time minus one. This is identification information (moving image ID) of moving image data of a collaboration image.

  Next, moving to step SS20, the CPU of the server 58 that has received the above data collaborates from the musical instrument of the performance part that was originally synthesized in the current collaboration image and the musical instrument of the performance part that was performed this time minus one. Re-determine where to place each performance part in the image. This re-determination of the arrangement location is performed according to the basic arrangement rule shown in FIG.

  Next, the process proceeds to step SS30, where the collaboration image is re-synthesized from the moving image data of the original collaboration image, the recording data of the currently-playing image recorded this time, and the information on the arrangement location re-determined in step SS20. Then, the collaboration image is recorded in the mass storage device 58a in association with the song title or performance part. Then, this flow ends.

  In the above, the procedure of step SK17 corresponds to the part type tone color designating means described in each claim, the procedure of step SS30 corresponds to the co-starring image synthesizing means described in each claim, and the procedure of step SK70 is described in each claim. This corresponds to the co-star image display means, and the procedure of step SK80 corresponds to the in-performance image transmitting means described in each claim.

  FIG. 7 shows the data format of each piece of information sent and received and recorded in the above flow and the relationship between them. In addition, in order to avoid the complexity of illustration, description of music data (MIDI data) is omitted. A in FIG. 7 indicates a data format of a request signal transmitted from the karaoke apparatus 10 to the server 58. This request signal includes a user ID that is user identification information, a reserved song title (or song identification information), and musical instrument information of a performance part that is the subject of the minus one performance. Thereby, the server 58 can confirm the reserved music and the minus one performance part.

  B in FIG. 7 indicates a data format of a response signal that the karaoke apparatus 10 downloads from the server 58. This response signal includes the user ID, the song name of the reserved music, the moving image ID and moving image data of the acquired collaboration image, and the musical instrument information of the playing image synthesized with the collaboration image. . Thereby, the karaoke apparatus 10 can appropriately fit and display the playing image of the minus one performance photographed by the reproduction of the collaboration image and the camera 25.

  C of FIG. 7 shows the data format of various moving image data recorded in the mass storage device 58a of the server 58. The recorded moving image data is roughly divided into two types, one is the moving image data of the already-combined collaboration image, and the other is the moving image data of the individual performance images received so far.

  One collaboration image includes the name of the corresponding music piece, the moving image ID and moving image data of the collaboration image, and the musical instrument information of the playing image synthesized with the collaboration image. The response signal B is obtained corresponding to the currently reserved music and the minus one performance part in the moving image data of the collaboration image. Note that the musical instrument information of the playing image included in the moving image data may be replaced with musical instrument information of a performance part capable of performing minus one performance instead.

  The other playing image includes the song name of the corresponding music piece, moving image data of the playing image recorded by the other user in the minus one performance, and instrument information of the performance part at that time. By recording the performance images individually in this way, it is possible to synthesize collaboration images in various combinations for the same music.

FIG. 7D shows a data format of recorded data transmitted from the karaoke apparatus 10 to the server 58 after the collaboration performance (minus one performance). This recorded data includes at least the name of the music piece that has been collaborated, the video ID of the collaborative image used, the video data shot by the camera 25 during the minus one performance, and the instrument information of the performance part at that time. It is.
The user ID included in the recorded data is used to specify the poster of the captured video data (for example, when a member is displayed as a musical instrument player in the synthesized image). This is not necessary for the synthesis of the collaboration image determined in accordance with the type).

  The server 58 that has received the recording data acquires the moving image data of the original collaboration image from the large-capacity storage device 58a based on the moving image ID included therein, and the musical instrument of the playing image synthesized with the collaboration image. Information can be identified. In addition, a performance image recorded in the large-capacity storage device 58a corresponding to the same music can be acquired. The server 58 corresponds to the same musical composition based on the specified musical instrument information, the acquired in-playing image, the song name included in the recording data, the current recording data, and the musical instrument information of minus one performance. A new collaboration image can be re-synthesized according to the basic form of the arrangement rule of 5. A separate moving image ID is assigned to the collaboration image to form new moving image data, which is recorded in the mass storage device 58a.

  As described above, in the present embodiment, the playing image extracted from the large-capacity storage device 58a in accordance with the procedure of step SS30 is arranged in accordance with a customary arrangement rule according to the reserved music genre, organization form, and the like. Synthesize collaborative images arranged in a grid. The collaboration image synthesized in this way and displayed on the monitor 13 or 14 does not feel uncomfortable not only to the performer of minus one performance but also to other users who are present. As a result, a natural arrangement on the musical instrument performance in the collaboration image can be realized.

  In addition, when the processing capability of the karaoke apparatus 10 is high, the collaboration image synthesis process may be performed on the karaoke apparatus 10 side. In this case, an individual playing image may be downloaded from the server 58, and the collaboration image may be synthesized and displayed in real time before or during the minus one performance. Alternatively, a collaboration image may be downloaded from the server 58 and a new collaboration image may be re-synthesized based on the collaboration image and the recorded data after the minus one performance.

  Moreover, although the case where only one performance part (only a guitar) is performed in the karaoke apparatus 10 is shown in the above embodiment, a plurality of performance parts may be collaborated (minus one performance) at the same time. In this case, positions corresponding to the plurality of performance parts are blank in the collaboration image, and the images during performance photographed by the plurality of cameras 25 are respectively fitted and displayed.

  In this embodiment, the part type of the minus one performance part is automatically designated by the timbre recognition from the sound of the musical instrument in the procedure of step SK17. Accordingly, it is possible to easily designate the part type of the minus one performance part only by the simple musical instrument performance by the user (minus one player himself). In the present embodiment, the above-described timbre recognition process is performed on the karaoke apparatus 10 side. However, the timbre recognition process may be performed on the server 58 side. In this case, the request signal (A in FIG. 7) transmitted from the karaoke apparatus 10 to the server 58 in the procedure of step SK20 includes the sound signal data instead of the instrument information data.

  In addition, during the minus one performance, it is possible to synthesize a collaborative image and display the image during the performance of the performer of the minus one performance part while simultaneously reflecting the photographed performance image in real time. Alternatively, a large processing load is imposed on the communication network 15. On the other hand, in the present embodiment, in particular, the performance image captured during the current minus one performance in the procedure of step SK70 is displayed in the existing collaboration image, and after the completion, the next time for the same music in the procedure of step SS30. By re-synthesizing the collaboration image so as to be reflected at the subsequent minus one performance, the processing load on the karaoke apparatus 10 and the communication network 15 can be greatly reduced. Thereby, the synthesis | combination and display of a collaboration image are realizable using the karaoke apparatus 10 and the communication network 15 of simple structure.

  In the present embodiment, in particular, in the procedure of step SS30, the extracted in-performance image is arranged and synthesized at each specified position in the display area of the collaboration image. Thereby, the arrangement | positioning relationship of each part of the extracted image during performance can be visually recognized clearly. Further, since the collaborative images can be synthesized simply by arranging the extracted images during performance at respective specified positions according to a predetermined arrangement rule, the processing load on the server 58 in the procedure of step SS30 can be reduced. As for the playing image of the part that has not been recorded in the large-capacity storage device 58a of the server 58 and could not be extracted, the name of the part type and the image of the corresponding instrument may be arranged at the specified position. .

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and technical idea of the present invention. Hereinafter, such modifications will be described.

<When changing the position and size according to the number and type of performance parts>
In the above embodiment, the arrangement and size of each performance image in the collaboration image are fixed, but the present invention is not limited to this. For example, as shown in FIG. 8, the position and size thereof may be changed as appropriate according to the performance part type and number of the performance part images to be synthesized.

  As an example shown in the figure, when only one performance part (only vocal) is displayed, as shown in FIG. 8A, only the playing image of that one performance part is enlarged to the same size as the entire display area. And display. On the other hand, when displaying with two performance parts (adding drums), as shown in FIG. 8B, the display area of the two performance parts is enlarged to a size that divides the entire display area into left and right parts. indicate. At this time, in the case of the vocal and drum in the example shown in the figure, the vocal is arranged on the left and the drum is arranged on the right. Further, although not particularly illustrated, the arrangement may be reversed left and right, or may be arranged in two vertically divided parts with the vocals above and the drums below.

  What is important here is that the drum is not always placed in front of the vocal by convention, and such an uncomfortable arrangement, that is, the upper and lower divided arrangement with the drum on and the vocal below is not performed. Although not specifically shown, when only two performance parts other than vocals, for example, a bass and a drum are displayed, the bass is not conventionally positioned on the right and bottom of the drum. For this reason, the arrangement is either left and right divided into two with the base left and the drum right, or two vertically divided with the base up and the drum down.

  If three performance parts, vocal, drum, and bass are displayed, the bass is located to the left of the vocal and the drum is located below the vocal, as shown in FIG. With this arrangement, the display area is divided into four parts, top, bottom, left, and right, so that the performance image of each performance part can be displayed larger than the basic form shown in FIG. In this way, when multiple performance parts are displayed, the display area is displayed in an arrangement and size that maximizes the display area while maintaining only the positional relationship between the two performance parts in any combination. To do.

  When displaying four performance parts: vocal, drum, bass, and guitar, as shown in Fig. 8 (d), the bass is located under the vocal, the guitar is located to the right of the vocal, and the right of the vocal The drum is located below. This is the basic arrangement for a four-piece collaboration of a combination of vocals, drums, bass, and guitar. The fact that the guitar is located beside the vocal is a result of considering that it is a prominent performance part compared to the bass.

  As described above, in the procedure of step SS30, the extracted performance images are arranged and synthesized so as to change the position and size in the display area of the collaboration image according to the number and part type information. At the time of synthesis, the size and arrangement pattern may be determined at that time so as to consider the arrangement relationship between the performance parts in accordance with the number and type of the extracted images during performance, or in advance in a table or the like. Arrangement patterns corresponding to all combinations of performance parts may be stored and determined in accordance therewith.

  This makes it possible to synthesize a collaboration image that maximizes the entire display area of the collaboration image using only the extracted part-playing images while maintaining the positional relationship between the individual performance parts. The visibility of the part is improved.

  In addition to those already described above, the methods according to the above-described embodiments and modifications may be used in appropriate combination.

  In addition, although not illustrated one by one, the present invention is implemented with various modifications within a range not departing from the gist thereof.

DESCRIPTION OF SYMBOLS 1 Karaoke system 4 Electric guitar 10 Karaoke apparatus 13, 14 Monitor 25 Camera (part image photographing means)
58 server 58a mass storage device (part image recording means)

Claims (5)

A part type timbre designating means for automatically designating a part type of a minus one performance part to be subjected to a minus one performance for a predetermined music by timbre recognition from the pronunciation of an instrument;
Part image photographing means for photographing an image during performance of the performer of the minus one performance part;
A karaoke device having
A server having a part image recording unit that records the performance image captured by the part image capturing unit of the karaoke apparatus in association with song identification information of the predetermined music piece and part type information of the minus one performance part; ,
A karaoke system comprising:
In any one of the karaoke apparatus and the server,
When performing a minus one performance by specifying a song and a minus one performance part, it has already been recorded corresponding to the song identification information of the designated song and the part type information of parts other than the designated minus one performance part. A performance image synthesizing means for synthesizing the co-starring images obtained by extracting the performance images being performed from the part image recording means and arranging the extracted performance images in a grid pattern according to a predetermined arrangement rule;
The co-starring image synthesizing means is
The extracted performance images are combined and arranged so as to change the position and size in the display area of the co-starring image according to the number and part type information, and designated by the part type tone color designating means. The display position of the playing image representing the performance by the musical instrument is determined based on the playing image representing the performance by the vocal ,
The karaoke device
A karaoke system comprising co-star image display means for displaying the co-star image synthesized by the co-star image synthesis means. The co-star image synthesizing means is arranged such that, when the performance image includes the performance by the vocal and the performance by the guitar, the performance image representing the performance by the guitar is arranged to the right of the performance image representing the performance by the vocal. The karaoke system according to claim 1, wherein the co-star images are synthesized. The co-star image synthesizing means may be configured such that when the performance image includes the performance by the vocal and the performance by the bass, the performance image representing the performance by the bass is left or below the performance image representing the performance by the vocal. The karaoke system according to claim 1 or 2, wherein the co-star images are synthesized so as to be arranged. The co-star image synthesizing means is configured such that when the performance image includes the performance by the vocal and the performance by the drum, the performance image representing the performance by the drum is on the right or below the performance image representing the performance by the vocal. The karaoke system according to any one of claims 1 to 3, wherein the co-star images are synthesized so as to be arranged. A part type timbre designating means for automatically designating a part type of a minus one performance part to be subjected to a minus one performance for a predetermined music by timbre recognition from the pronunciation of an instrument;
Part image photographing means for photographing an image during performance of the performer of the minus one performance part;
During the performance of transmitting to the external server so as to record the performance image captured by the part image capturing means in association with the song identification information of the predetermined music and the part type information of the minus one performance part A karaoke apparatus having image transmission means,
When the minus one performance piece and the minus one performance part are specified, it corresponds to the song identification information of the designated song, and already corresponds to the part type information of the parts other than the designated minus one performance part. is extracted play in the image being recorded, the performance in an image obtained by these extraction and depending on their number and part type information arranged to change the position and size in the display area of the played image When the vocal image and the guitar performance are included in the playing image, the display area is arranged such that the playing image representing the guitar performance is arranged to the right of the playing image representing the vocal performance. the played images arranged and by combining to change the position and size in the middle, and requests to the server, and played image request means,
Co- star image display means for receiving and displaying the co-star image from the server ;
A karaoke apparatus comprising:

Images