JP4257300B2 - Karaoke terminal device - Google Patents

Description

  The present invention is not limited to the background, and relates to a karaoke terminal device that outputs a synthesized image as a moving image on the spot.

  2. Description of the Related Art Conventionally, a karaoke terminal apparatus is known that repeatedly displays a predetermined image taken together with lyrics during a user's singing in a certain cycle.

Also, there is a karaoke terminal device that combines and displays a singer along with an arbitrary background using a technique called chroma key. If the chroma key method is used, a specific color (blue is often used but green may be used) is extracted from the video screen, a mask image is created, and the mask image is combined with an arbitrary background. be able to.
JP 2004-334049 A

  However, in the above-described karaoke terminal device that displays the photographed background at a constant cycle, any model is not distinguishable by function or the like, and becomes a one pattern. In addition, when used for a long time or repeatedly, the same image is repeatedly displayed, which makes the user bored.

  In addition, there is also known a method of synthesizing and displaying a user's singing image and background as in the above-described chroma key method, but since the chroma key is used for image extraction, the background color is limited to chroma key blue or the like. The background is limited to the selected color. As a result, the singer, for example, sings against a background of chroma key blue, and distracts the user's interest. Furthermore, even when a frame image having a predetermined transparent area is synthesized on a photographed image, the background of the photographed image is directly copied to the synthesized image, so that the quality of the synthesized image is low. .

  The present invention has been made in view of the above-described problems, and shoots a singer without being caught by the background, and synthesizes a composite image with an arbitrary background image as a moving image and outputs it on the spot. An object of the present invention is to provide a karaoke terminal device that can be used.

  Therefore, in order to output a composite image regardless of the background to be photographed, it is necessary to recognize a moving object that is a target singer, create a mask image having a transparent region, and combine it with an arbitrary background image. Here, as a method for specifying an extraction target portion in an arbitrary moving image, a method using a motion vector, a method using a dynamic contour model, a method for tracking changes in feature points of a contour, and the like can be given. .

  In the method using a motion vector, an input image is divided into blocks having a predetermined size, and a difference from the position of a similar block in the current image is calculated as a “motion vector”. In this method, the amount of calculation becomes very large and cannot be used for a karaoke apparatus that requires immediacy. Even if the resolution of the block is increased, it is only the resolution in block units, and the contour data in pixel units cannot be obtained accurately.

  In the method using the active contour model, the contour is set so that the sum of the internal energy given according to the contour shape, the image energy given according to the nature of the image, and the external energy given from the outside is minimized. The contour of the object on the image can be extracted when the energy is minimum. However, since the contour that can be detected by the dynamic contour model changes depending on the parameter that adjusts the energy balance, it is very difficult to determine the parameter for detecting the target contour, and the object is concave. In some cases, there is a problem that the contour cannot be extracted well. Therefore, this method is also difficult to perform initial setting, and is difficult to use for a karaoke terminal device that is required to be immediate and not limited to a singer.

  In order to solve the above problems, the present invention provides a karaoke terminal device as follows.

  (1) A karaoke terminal device having accompaniment sound generating means for generating accompaniment information including at least accompaniment sound and karaoke information control means for controlling information related to karaoke, and storing the voice of the singer and the accompaniment information The singer can select the audio information storage area to be stored, the reference image storage area for storing the reference image, the pre-combination image storage area for storing the pre-combination image, the extraction target part storage area for storing the extraction target part, and the singer A background image storage area for storing a photographed background image, and the karaoke information control means collects the voice of the singer and the accompaniment information and stores the voice information in the voice information storage area. A reference image collecting means for collecting a reference image and storing it in the reference image storage region; and a pre-combination image collection for collecting the pre-combination image and storing it in the pre-combination image storage region The reference image stored in the reference image storage area and the pre-combination image stored in the pre-combination image storage area are digitized and compared for each pixel, and pixels with differences are recognized as extraction target portions. Extraction target part recognizing means, target part extracting means for extracting the extraction target part recognized by the extraction target part recognizing means and storing it in the target part storage area, and a background image stored in the background image storage area And a combining unit that combines the extracted images stored in the target part storage area, and an output unit that outputs the combined image combined by the combining unit together with the voice information stored in the voice information storage unit. Karaoke terminal device.

  According to the invention described in (1), the reference image is stored in the reference image storage area, and the extraction part recognition unit compares the collected image before synthesis with the reference image for each frame of the collected pre-combination image, and selects a portion having a color difference. Extraction is performed to create a mask image, and the synthesizing unit can synthesize a background image arbitrarily selected by the user, and can output a synthesized image independent of the background.

  (2) The karaoke terminal device according to (1), wherein the reference image collection unit and the pre-combination image collection unit are configured by a compound eye camera.

  (3) When collecting a series of moving images, the reference image collecting means adjusts the direction of each compound eye camera based on a reference point written in advance in one of the backgrounds, One pre-combine image included in the collection time of the series of moving images is collected and stored in the pre-combine image storage area, and the other collects the reference image of the certain point in time and stores the reference image The karaoke terminal device according to (2), which is stored in an area.

  According to the invention described in (2) or (3), when the collected compound eye images are collated by collecting images using a compound eye camera in which the orientation of each compound eye camera is matched with the reference point in advance. The part where a color does not correspond arises. That part can be recognized as a new part not in the reference image. It is possible to extract a singer without initial setting and without performing complicated calculations.

  (4) The reference image is an image collected in advance before collecting a series of moving images, and the pre-combination image is an image collected at a certain time included in the collection time of the series of moving images. The karaoke terminal device according to (1).

  According to the invention of (4), by recognizing a pixel having a color tone difference between the pre-combination image collected in advance and the reference image as an extraction target part, the difference from the reference image is changed to the extraction target part. Become. As a result, a mask image can be created by a simple comparison program without requiring complicated calculations. Moreover, a composite image can be output without limitation of a background like a chroma key, and does not distract the interest of a karaoke user. Further, since the pre-combination image and the reference image collected in advance are compared, the extraction target portion can be specified even when the extraction target portion hardly fluctuates.

  (5) The karaoke information control means outputs control signal information for controlling illumination in a predetermined period included in the moving image collection time, and the reference image collecting means before collecting the series of moving images. A reference image is collected in advance and stored in the reference image storage area together with the control signal information. The pre-combination image collection means collects a pre-combination image at a certain time included in the predetermined period, and the control The karaoke terminal device according to (4), wherein the extracted image recognition unit stores the reference image and the pre-combination image in synchronization with each other based on the control signal information. .

  According to the invention of (5), when the reference image is collected, the composite image is stored in the reference image storage area as a series of images included in a certain time together with illumination control information. It is possible to provide a composite image corresponding to a change in illumination by matching a time axis in a frame unit with a series of reference images when collecting and recognizing a moving object by comparing frames on the same time axis. it can.

  (6) The reference image is an image collected one frame before a certain time point included in the collection time of the series of moving images, and the pre-combination image is an image collected at the certain time point. The karaoke terminal device according to (1).

  According to the invention of (6), since the reference image is one frame before the pre-combination image at a certain time, the extraction target portion can be specified even when the environment of the karaoke room changes greatly.

  (7) In the recognition of the extraction target part, the extraction target part recognition unit captures an outline of the extraction target part and tracks its movement, and the target part extraction unit extracts pixels surrounded by the contour The karaoke terminal device according to any one of (1) to (6), which is extracted as a target portion.

  According to the invention of (7), by recognizing the extraction target portion based only on the contour data, the amount of calculation can be further reduced and the burden on the apparatus can be reduced. As a result, a composite image can be easily output.

(8) A karaoke terminal device having accompaniment sound generating means for generating accompaniment information including at least accompaniment sound and karaoke information control means for controlling information related to karaoke, and storing the voice of the singer and the accompaniment information And a background image storage area for storing a photographed background image selectable by the singer, and the karaoke information control means collects the singer's voice and the accompaniment information. Audio information collecting means for saving in the audio information saving area, image collecting means for collecting a series of moving images and saving them in the pre-combination image saving area, and in the background image saving area on the series of moving images Synthesis means for superimposing the stored background images, and output means for outputting the synthesized image synthesized by the synthesis means together with the audio information stored in the audio information storage portion; It includes,
The karaoke terminal apparatus, wherein the background image is set in advance with a transparent portion to be superimposed on the extracted image, and the transparency of the edge of the transparent portion gradually changes.

  According to the invention described in (8), by setting a transparent portion in the background image in advance, the edge portion is blurred and combined while reducing the amount of calculation of the combining process, and the quality is higher easily. A synthesis process can be performed.

  (9) The pre-combination image includes pixels corresponding to a blur compensation region around an output target pixel region necessary for image output, and the extraction portion recognition unit is configured to output the pre-combination image storage region at the certain time. For the pre-combine image stored in the above, after digitizing the color for each pixel, the reference image stored in the reference image storage area is compared by shifting the pixel image vertically, horizontally, and pixel by pixel. The pre-combination image is shifted by the shift amount and stored in the pre-combination image storage area when the total sum of the color numerical differences for each pixel is the smallest (1) to (7) The karaoke terminal device according to any one of the above.

  According to the invention described in (9), if the pre-combination image is captured by including pixels corresponding to the blur compensation area around the output target area, an image corresponding to the blur is generated. Can be shifted and stored as a pre-combine image. As a result, the composition process can be performed without being affected by blurring, and a higher quality image can be provided.

  (10) The karaoke terminal device can communicate with a server on the network, and the output unit transmits the composite image data to the server based on a singer's operation. The karaoke terminal device described in 1.

  According to the invention described in (10), by transmitting the composite image data to the server on the network, the singing information that is the result of actually singing is transmitted to the outside together with the composite image without being limited to the frame of the karaoke room. It can be opened to the public, providing an opportunity for many people to listen to, and inspiring the singers. Thereby, the interest improvement of a singer can be aimed at.

According to the present invention, in a karaoke terminal device, an image that is combined with an arbitrary background can be provided immediately without being limited to a specific background, and the interest of the singer can be improved.

[Configuration of karaoke terminal device]
Preferred embodiments of the present invention will be described below with reference to the drawings.

  As shown in the block diagram of FIG. 1, the karaoke terminal device includes accompaniment sound generating means 10 and karaoke information control means 20.

  The accompaniment sound generation means 10 has a function of displaying an image included in the accompaniment information on the display unit and generating an accompaniment sound included in the accompaniment information according to the result of the existing accompaniment information being selected by the singer. The existing accompaniment information includes an accompaniment sound. The accompaniment information only needs to include at least an accompaniment sound.

  As shown in FIG. 1, the karaoke information control means 20 is a pre-combination image collection means 21, a reference image collection means 22, an extraction target part recognition means 25, a target part extraction means 26, a synthesis means 40, an output means 50, audio information. At least the collecting means 32 is included. The karaoke information control means 20 extracts a part of the collected series of moving images, synthesizes them with a previously selected background, and outputs them.

  The pre-combination image collection means 21 shown in FIG. 1 is for collecting pre-combination images. The pre-combine image collection means 21 is configured by a video camera 232A, but the number of cameras is not limited to one. The collected composite image is stored in the pre-composition image storage area.

  The reference image collection means 22 shown in FIG. 1 collects a reference image. Although the reference image collecting unit 22 is configured by the video camera 232A, the video camera 232A configuring the pre-combination image collecting unit 21 may also be used. In the case of sharing, it is necessary to collect the reference image before collecting the pre-combination image. Then, the collected reference image is stored in the reference image storage area.

  The extraction target part recognizing means 25 shown in FIG. 1 recognizes an extraction target part to be output from the pre-combination image when performing the synthesis process. The recognition of the extraction target portion is not limited to one method as will be described later with reference to FIGS.

  The target portion extraction unit 26 shown in FIG. 1 extracts the extraction target portion recognized by the extraction target portion recognition unit 25 from the pre-combination image, creates a mask image, and combines it with the background image. Save to the extraction area.

  The synthesizing means 40 shown in FIG. 1 synthesizes the mask image stored in the target portion storage area 27 with a background image selected in advance.

  The output means 50 shown in FIG. 1 outputs the voice information and the synthesized image created by the synthesizing means 40. The output location may be output to a display or mixing amplifier 218 constituting the display unit 210, or may be transmitted to a server on the network.

[Electrical configuration of karaoke terminal]
The electrical configuration of the karaoke terminal apparatus will be described with reference to FIG.

  As shown in FIG. 2, in the karaoke terminal device 20, the data BUS includes a CPU 202 as a control unit, a memory 204, a communication interface (hereinafter referred to as a communication I / F 206), RAID (Redundant Arrays of Inexpensive Disks), and the like. A storage unit 208 configured with a liquid crystal display panel or a CRT, an operation unit 212 including a remote controller or a keyboard, a synthesizer 216 for generating audio from audio data, and editing and amplifying audio Are connected to a mixing amplifier 218 and an AV data processing unit 220 for editing images and sounds.

  In addition, a mixing amplifier 218 serving as a control unit is connected to a microphone that collects sound produced by the singer, a speaker 214 that generates sound, and the like. This singer shows the person who sings with accompaniment information of karaoke. Moreover, a user shows not only the singer with respect to the accompaniment information of karaoke but the person who uses a karaoke terminal apparatus in a karaoke room, even if he is not singing.

  The mixing amplifier 218 has a function of controlling the connected microphone and speaker 214 in accordance with a command supplied from the CPU 202. The mixing amplifier 218 has a function of collecting the singer's voice input from the microphone in response to a command supplied from the CPU 202. The mixing amplifier 218 has a function of collecting the user's voice input from the sound collection microphone in response to a command supplied from the CPU 202. The mixing amplifier 218 has a function of generating collected sound from the speaker 214 in accordance with a command supplied from the CPU 202. The mixing amplifier 218 has a function of supplying the collected audio to the AV data processing unit 220 in accordance with a command supplied from the CPU 202. For this reason, the CPU 202 executes various functions by transmitting various commands to the mixing amplifier 218.

  Further, video cameras 232A, 232B,... Are connected to the AV data processing unit 220 as control means. The number of video cameras 232A that capture a moving image of a singer is not limited to one.

  The AV data processing unit 220 has a function of collecting user images and singer images supplied from the video cameras 232A, 232B,... According to commands supplied from the CPU 202. The AV data processing unit 220 has a function of storing audio supplied from the mixing amplifier 218 in response to a command supplied from the CPU 202. The AV data processing unit 220 has a function of editing a collected image and supplied sound in accordance with a command supplied from the CPU 202, and a function of editing an image using a special composite image stored in advance. The AV data processing unit 220 corresponds to a part of the collecting unit. Furthermore, the AV data processing unit 220 corresponds to a part of editing means for editing collected image information. For this reason, the CPU 202 executes various functions by transmitting various commands to the AV data processing unit 220.

  The CPU 202 has a function of executing various processes in accordance with programs stored in the storage unit 208. Further, the CPU 202 implements various means described later in cooperation with the hardware by reading and executing a program stored in the storage unit 208.

  The storage unit 208 stores accompaniment information including accompaniment sounds and images, a program for the CPU 202 to control the operation of the karaoke information control server, and the like. Accompaniment information includes an accompaniment sound of karaoke and an image displayed on the display unit 210 in synchronization with the accompaniment sound, and is associated with music selection information including a music selection number. A specific program will be described later.

  In this embodiment, the storage unit 208 is used as a medium for storing a program or the like. However, the present invention is not limited to this, and any other storage medium that can be read by a computer may be used. For example, it may be recorded on a storage medium such as a ROM, a CD-ROM, and a DVD. These programs may not be recorded in advance, but may be recorded in the memory 204 or the like after the power is turned on. Furthermore, each program may be recorded on a separate storage medium.

  The memory 204 has a function of storing various flags and variable values as a temporary storage area of the CPU 202. Specific examples of data stored in the memory 204 include the following.

  Accompaniment flag for determining whether or not accompaniment is occurring, song selection information indicating the selected song information, song memory flag for determining whether or not to collect singer's image and sound, karaoke terminal device An operation flag or the like for determining whether or not there is an abnormality in the operation state is stored in the memory 204. The music selection information includes music selection information that is being executed and music selection information that is being executed.

  In this embodiment, the memory 204 is used as a temporary storage area of the CPU 202. However, the present invention is not limited to this, and any readable / writable storage medium may be used.

  A display unit 210 is provided in front of the karaoke terminal device 20 shown in FIG. The display unit 210 has a plurality of display areas 210a, 210b, 210c, and 210d. As will be described later, the selected image is displayed in the display area 210a which is the main display area. Images included in the accompaniment information are displayed in the display area 210b and the display area 210c. In the display area 210d, an image of the singer is displayed. This singer's image is a singer's image with respect to the accompaniment information of karaoke, and is included in the singing information described later. Moreover, this singing information includes a singer's image and sound, and accompaniment information.

  In the present embodiment, the display unit 210 is configured as a single display device, but the present invention is not limited to this, and may be configured as a plurality of display devices.

  Below the display unit 210, a karaoke performance device 102 and an image composition device 104 are provided.

  The image synthesizing device 104 has a function of generating a special synthesized image and a function of collecting peripheral sounds from a sound collecting microphone. The image composition device 104 corresponds to a part of the collection unit. The image composition device 104 corresponds to a part of editing means for editing the collected singing information.

  A main screen changeover switch 106 is provided on the side of the image composition device 104. The main screen changeover switch 106 has a function of switching a screen to be displayed in the display area 210a, which is a main display area, according to an operation.

  The karaoke terminal device 20 includes a remote control 108 and a wireless keyboard 110. Various functions are executed in accordance with operations of the remote control 108 and the wireless keyboard 110. The remote control 108 and the wireless keyboard 110 correspond to a part of an operation unit 212 (see FIG. 3) described later.

  In this embodiment, the karaoke performance device 102 and the image composition device 104 are provided. However, the present invention is not limited to this, and various devices such as the karaoke performance device 102 and the image composition device 104 are provided. In this case, it is preferable to adopt a configuration in which the functions possessed by other devices are provided. Of course, the karaoke performance device 102 may be integrated or separate.

[Operation of karaoke information control means]
The operation of the karaoke information control means 20 configured as described above will be described with reference to FIG.

  As shown in FIG. 4, in step S11, a music selection process is executed. In this process, the CPU 202 performs music selection information processing for generating an accompaniment in order to generate an accompaniment corresponding to the music selection in accordance with various input operations. Details will be described later with reference to FIG. If this process ends, the process moves to a step S12.

  In step S12, the CPU 202 executes a synthesis mode selection process. In this process, the CPU 202 determines whether or not to perform the synthesis process. If it is determined that the synthesis process is to be performed, the CPU 202 collects the reference image and stores it in the reference image storage area. Further, the background image data selected by the user is transferred to the background image storage area 28. Details will be described later with reference to FIG. If this process ends, the process moves to a step S13.

  In step S13, the CPU 202 executes accompaniment generation processing. In this process, the CPU 202 passes the accompaniment data to the AV data processing unit 220. The AV data processing unit generates an accompaniment through the speaker 214 based on the received accompaniment data. If this process ends, the process moves to a step S14.

  In step S14, the CPU 202 executes shooting start processing. In this process, if the reference image has been collected in step S12, the CPU 202 collects the pre-combination image and stores it in the pre-combination image storage area 23. In step S13, when the reference image is not collected, the reference image and the pre-combination image are collected. Details will be described later with reference to FIGS. If this process ends, the process moves to a step S15.

  In step S15, the CPU 202 performs extraction target part recognition processing. In this process, the CPU 202 reads out the pre-combine image from the pre-combine image storage area 23 and extracts a portion to be combined with the background stored in the background image storage area 28. This extraction target portion recognition process has several variations, and details will be described later with reference to FIGS. If this process ends, the process moves to a step S16.

  In step S16, the CPU 202 performs target portion extraction processing. In this process, the CPU 202 performs a process of extracting the extraction target part extracted in the extraction target part recognition process in step S15 from the pre-combination image. The CPU 202 creates a mask image for the extraction target part, superimposes it with the pre-combination image, and extracts only the extraction target part from the pre-combination image. Then, the CPU 202 stores the extraction target portion extracted from the pre-combination image in the target portion storage area 27. If this process ends, the process moves to a step S17.

  In step S17, the CPU 202 performs a composition process. In this process, the CPU 202 reads the extracted image from the target part storage area 27 and creates a composite image by superimposing the previously selected background image from the background image storage area 28. If this process ends, the process moves to a step S18.

  In step S18, the CPU 202 performs output processing. In this process, the CPU 202 synchronizes the audio information stored in the audio information storage area 34 and the synthesized image, and passes the synthesized image to the display unit 210 and the audio information to the AV data processing device. That is, data for displaying a composite image is supplied to the display unit 210 and audio information is supplied to the mixing amplifier 218. If it is selected in step S62 in FIG. 6 that the synthesis process is not performed, the CPU 202 supplies accompaniment demonstration screen data for which the synthesis process is not performed. As a result, the display unit 210 displays a composite image or an accompaniment demonstration screen, and the speaker 214 reproduces audio information. If this process ends, the process moves to a step S19.

  In step S19, the CPU 202 determines whether or not the accompaniment ends. In this process, if the CPU 202 determines that the accompaniment has ended, it moves the process to step S20. If it is determined that the accompaniment has not ended, the process proceeds to step S14. That is, the CPU 202 repeats the image composition process starting from the shooting start process.

  In step S20, the CPU 202 performs accompaniment end processing. In this process, the CPU 202 finishes passing the accompaniment data to the AV data processing unit. Thereby, the speaker 214 ends the reproduction of the audio information. In addition, the CPU 202 ends the collection of audio information. If this process ends, the process moves to a step S21.

  In step S21, the CPU 202 performs a photographing end process. In this process, the CPU 202 ends the collection of the pre-combination image and the reference image. In addition, the CPU 202 ends supplying data for displaying the composite image. If this process ends, the process moves to a step S11. That is, returning to the accompaniment generation preparation, the processing so far is repeated.

[Music selection process]
The subroutine executed in step S11 in FIG. 4 will be described with reference to FIG.

  In step S31, it is determined whether or not a music number input operation has been performed. In this process, the CPU 202 determines whether or not a song number input operation has been performed in accordance with the operation of the operation unit 212 such as the remote control 108 or the wireless keyboard 110. If the CPU 202 determines that a song number input operation has been performed, the CPU 202 moves the process to step S32. On the other hand, if the CPU 202 does not determine that the music number input operation has been performed, the process proceeds to step S33.

  If it is determined in the process of step S31 that a music number input operation has been performed, a music selection information storage process is executed (step S32). In this process, the CPU 202 stores the music selection information corresponding to the input operation as the music selection information on standby. If this process ends, the process moves to a step S33.

  In step S33, it is determined whether or not accompaniment is being performed. In this process, the CPU 202 determines whether or not the accompaniment is generated by the accompaniment generating means. If the CPU 202 determines that accompaniment is in progress, the subroutine is terminated. On the other hand, if the CPU 202 does not determine that accompaniment is being performed, the CPU 202 moves the process to step S34.

  If it is determined in step S33 that the accompaniment has ended, it is determined whether there is music selection information (step S34). In this process, the CPU 202 reads the music selection information that is waiting for execution, and determines whether or not there is music selection information by determining whether or not all the music selection information that is waiting for execution is empty information. It becomes. If the CPU 202 determines that there is music selection information, the CPU 202 ends this subroutine. On the other hand, if the CPU 202 does not determine that there is music selection information, the process proceeds to step S35.

  If it is determined in the process of step S34 that there is no music selection information, a music selection process based on the music selection information is executed (step S35). In this process, the CPU 202 stores one piece of music selection information that is waiting to be executed as music selection information that is being executed according to the execution order. When this process is finished, this subroutine is finished.

[Composite mode selection process]
The subroutine executed in step S12 in FIG. 4 will be described with reference to FIG.

  In step S61, the CPU 202 performs a synthesis mode selection screen display process. In this process, the CPU 202 supplies screen data to the display unit 210 that allows the user to select a synthesis mode. If this process ends, the process moves to a step S62.

  In step S62, the CPU 202 determines whether or not to perform a synthesis process. In this process, the CPU 202 determines whether or not to perform the synthesis process based on the data input from the remote control 108 or the wireless keyboard 110 in step S61. If the CPU 202 determines that the composition process is to be performed, the process proceeds to step S63. If it is determined that the synthesis process is not performed, this subroutine is terminated.

  In step S63, the CPU 202 determines whether to capture a reference image in advance. In this process, the CPU 202 determines whether or not to capture a reference image in advance depending on the configuration information of the video cameras 232A and 232B and what variation is selected in the extraction target partial recognition process to be performed later. If it is determined that the reference image is to be taken in advance, the process proceeds to step S64. If it is not determined that the reference image is captured in advance, this subroutine is terminated.

  In step S64, the CPU 202 performs a reference image collection process. In this processing, the CPU 202 collects the reference image by the reference image collection unit 22. If this process ends, the process moves to a step S65.

  In step S65, the CPU 202 performs a reference image storage process. In this process, the CPU 202 stores the reference image collected in step S64 in the reference image storage area 24. If this process ends, the process moves to a step S66.

  In step S66, the CPU 202 performs background image designation processing. In this process, the CPU 202 reads an arbitrary background image from the storage unit 208. This background image may be selected by the user.

  In step S67, the CPU 202 sets a reference image. In this process, the CPU 202 reads the reference image stored in the reference image storage area 24 into the memory 204. If this process ends, the process moves to a step S68.

  In step S68, the CPU 202 stores the reference image set flag as a valid value. When this process ends, this subroutine ends.

[Shooting start processing]
The subroutine executed in step S14 in FIG. 4 will be described with reference to FIGS. There are several variations for the shooting start process, and a process suitable for the configuration of the karaoke terminal apparatus can be selected according to the situation.

[Shooting start processing (background difference)]
In step S71 in FIG. 7, the CPU 202 determines whether or not the reference image set flag is valid. In this process, the CPU 202 reads the reference image set flag and determines whether it is valid. If the CPU 202 determines that the reference image set flag is valid, it moves the process to step S72. On the other hand, if the CPU 202 does not determine that the reference image flag is valid, it moves the process to step S12 in FIG.

  In step S72, the CPU 202 performs pre-combination image collection processing. In this process, the CPU 202 collects the pre-combine image by the pre-combine image collection means 21. If this process ends, the process moves to a step S73.

  In step S73, the CPU 202 performs pre-combination image storage processing. In this process, the CPU 202 stores the pre-combination image collected in step S72 in the pre-combination image storage area 23. When this process is finished, this subroutine is finished. Moreover, although it is time to use the background difference, this may be used for other methods in which the reference image needs to be collected and stored in advance.

[Shooting start processing (difference between frames)]
In step S81 in FIG. 8, the CPU 202 performs a reference image collection process. In this processing, the CPU 202 collects the reference image by the reference image collection unit 22. If this process ends, the process moves to a step S82.

  In step S82, the CPU 202 performs a reference image storage process. In this process, the CPU 202 stores the reference image collected in step S81 in the reference image storage area 24. If this process ends, the process moves to a step S83.

  In step S83, the CPU 202 performs pre-combination image collection processing. In this process, the CPU 202 collects the pre-combine image by the pre-combine image collection means 21. If this process ends, the process moves to a step S84.

  In step S84, the CPU 202 performs pre-combination image storage processing. In this process, the CPU 202 stores the pre-combination image collected in step S83 in the pre-combination image storage area 23. When this process is finished, this subroutine is finished. In addition, although the difference between frames is used, it is not necessary to collect the reference image in advance, and it may be used for another method of collecting and storing the reference image at the same time.

[Extraction target partial recognition processing]
The subroutine executed in step S15 in FIG. 4 will be described with reference to FIGS. There are several variations for the extraction target partial recognition process, and a process suitable for the configuration of the karaoke terminal apparatus can be selected according to the situation.

[Extraction target partial recognition process 1]
The first variation of the extraction target partial recognition process will be described with reference to FIG.

  In step S111, the CPU 202 reads a reference image from the reference image storage area 24 and digitizes each pixel. Then, the CPU 202 stores the reference image numerical value. If this process ends, the process moves to a step S112.

  In step S112, the CPU 202 reads the pre-combine image from the pre-combine image storage area 23, and digitizes each pixel. Then, the CPU 202 stores the pre-combination image numerical value. If this process ends, the process moves to a step S113.

  In step S113, the CPU 202 performs a difference process. In this process, the CPU 202 calculates the difference between the reference image values obtained in step S111 from the pre-combination image values obtained in step S112. If this process ends, the process moves to a step S114.

  In step S114, the CPU 202 performs a difference value storage process. In this process, the CPU 202 stores the difference value obtained in step S113. When this process is finished, this subroutine is finished.

  By the processing from step S111 to step S114 described above, a part having a difference between the reference image and the pre-combination image is recognized as an extraction target part.

[Extraction target partial recognition process 2]
A second variation of the extraction target partial recognition process will be described with reference to FIG. The CPU 202 selects the second extraction target part recognition process particularly when a moving image is collected by a compound eye camera. When this process is selected, the image collected from one of the compound-eye cameras is stored as a reference image, so that it is not necessary to collect and store the reference image in advance.

  When a compound eye camera is used, one of the compound eye cameras 241a is set in focus on the subject 243 (FIG. 11). At that time, it is necessary to make sure that the mark 245 is photographed on any of the screens (image 257). The other camera 241b is installed on either the left or right side of the camera 241a, and the focus is adjusted so that the position of the mark 245 is at the same position as when the camera 241a is photographed (image 259). As a result, the backgrounds of the image 257 photographed by the camera 241a and the image 259 photographed by the camera 241b coincide with each other when the image 257 and the image 259 are superimposed. On the other hand, only the photographing target 243 is displayed with being shifted when the images 257 and 259 are superimposed.

  In step S121, the CPU 202 performs reference image numerical information storage processing. In this process, the CPU 202 digitizes each pixel of the reference image acquired in step S121. Then, the CPU 202 stores the reference image numerical value. If this process ends, the process moves to a step S122.

  In step S122, the CPU 202 performs numerical information storage processing for the pre-combine image. In this process, the CPU 202 digitizes each pixel of the pre-combination image. Then, the CPU 202 stores the pre-combination image numerical value. If this process ends, the process moves to a step S123.

  In step S123, the CPU 202 performs a difference process. In this process, the CPU 202 calculates the difference between the reference image values obtained in step S123 from the pre-combine image values obtained in step S122. If this process ends, the process moves to a step S124.

  In step S124, the CPU 202 performs shift value correction processing. In this process, the CPU 202 stores in advance the parallax of the compound-eye cameras 241a and 241b as a shift value, and corrects the parallax with respect to the pixel difference value obtained in step S123 using the shift value. By performing this process, it is possible to recognize an extraction handling area without a parallax shift. If this process ends, the process moves to a step S125.

  In step S125, the CPU 202 performs a corrected difference value storage process. In this process, the CPU 202 stores the corrected difference value obtained in step S123. When this process is finished, this subroutine is finished.

[Extraction target partial recognition process 3]
A third variation of the extraction target partial recognition process will be described with reference to FIG.

  In step S131, the CPU 202 reads the reference image from the reference image storage area 24 and digitizes each pixel. Then, the CPU 202 stores the reference image numerical value. If this process ends, the process moves to a step S132.

  In step S132, the CPU 202 performs numerical information storage processing for the pre-combine image. In this process, the CPU 202 digitizes each pixel of the pre-combination image. Then, the CPU 202 stores the pre-combination image numerical value. If this process ends, the process moves to a step S133.

  In step S133, the CPU 202 performs difference processing. In this process, the CPU 202 calculates the difference between the reference image values obtained in step S131 from the pre-combination image values obtained in step S132. If this process ends, the process moves to a step S134.

  In step S134, the CPU 202 performs a difference value storage process. In this process, the CPU 202 stores the difference value obtained in step S133. If this process ends, the process moves to a step S135.

  In step S135, the CPU 202 performs binarization processing. In this processing, the CPU 202 performs binarization processing using a preset threshold value for the difference value stored in step S134. If this process ends, the process moves to a step S136.

  In step S136, the CPU 202 performs a thinning process. In this process, the CPU 202 recognizes the boundary line between the pixel values 0 and 1 of the image subjected to the binarization process in step S135 and performs the thinning process. Then, the CPU 202 extracts the contour of the extraction target portion as a result of the thinning process. When this process is finished, this subroutine is finished.

[Extraction target recognition process 4]
A fourth variation of the extraction target partial recognition process will be described with reference to FIG.

  In step S <b> 141, the CPU 202 reads control signal information from the reference image storage area 24. Then, the CPU 202 synchronizes the control signal information and the reference image.

  In step S142, the CPU 202 performs numerical information storage processing for the reference image. In this process, the CPU 202 reads the reference image from the reference image image storage area 23 and digitizes each pixel of the reference image. Then, the CPU 202 stores the reference image numerical value. If this process ends, the process moves to a step S143.

  In step S143, the CPU 202 performs numerical information storage processing for the pre-combine image. In this process, the CPU 202 reads out the pre-combination image from the pre-combination image storage area 23 and digitizes each pixel of the pre-combination image. Then, the CPU 202 stores the pre-combination image numerical value. If this process ends, the process moves to a step S144.

  In step S144, the CPU 202 performs a difference process. In this process, the CPU 202 calculates the difference between the reference image values obtained in step S142 from the pre-combination image values obtained in step S143. If this process ends, the process moves to a step S145.

  In step S145, the CPU 202 performs a difference value storage process. In this process, the CPU 202 stores the difference value obtained in step S144. When this process is finished, this subroutine is finished.

[Extraction target partial recognition processing 5]
A fourth variation of the extraction target partial recognition process will be described with reference to FIG.

  In step S151, the CPU 202 performs numerical information storage processing for the pre-combine image. In this process, the CPU 202 reads out the pre-combination image from the pre-combination image storage area 23 and digitizes each pixel of the pre-combination image. Then, the CPU 202 stores the pre-combination image numerical value. If this process ends, the process moves to a step S152.

  In step S152, the CPU 202 performs a reference image shift comparison process. In this process, the CPU 202 reads the reference image from the reference image storage area 24, and shifts the reference image vertically, horizontally, and pixel by pixel with respect to the shake compensation area. At that time, each pixel of the reference image is digitized, and a place where the total sum of pixel differences between the reference image and the entire output target area is minimized is calculated. Then, the CPU 202 stores the distance and direction between the place where the total sum of pixel differences is minimized and the original reference image as the shift value. If this process ends, the process moves to a step S153.

  In step S153, the CPU 202 performs a shift process for the pre-combine image. In this process, the CPU 202 corrects the position of the pre-combine image using the shift value obtained in step S152. If this process ends, the process moves to a step S154.

  In step S154, the CPU 203 stores the pre-combination image whose position is corrected by the shift value in the pre-combination image storage area. When this process is finished, this subroutine is finished.

  In the fifth extraction target part recognition process, after this process is performed, another extraction target part recognition process may be performed.

  As mentioned above, although embodiment of this invention was described, it only showed the specific example and does not specifically limit this invention. Further, the effects described in the embodiments of the present invention only list the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to the effects described in the embodiments of the present invention.

It is a block diagram which shows the structure of the karaoke terminal device which is an Example of this invention. It is a figure which shows the hardware constitutions of the karaoke terminal device which is an Example of this invention. It is a screen image of the display part 210 of the karaoke terminal device which is an Example of this invention. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs. It is a conceptual diagram when the karaoke terminal device which is an Example of this invention collects a moving image with a compound eye camera. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs. It is a flowchart figure which shows the operation | movement which the karaoke terminal device which is an Example of this invention performs.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Karaoke terminal apparatus 10 Accompaniment sound generation means 20 Karaoke information control means 21 Pre-synthesis image collection means 22 Reference image collection means 25 Extraction target part recognition means 26 Target part extraction means 32 Audio information collection means 40 Synthesis means 50 Output means

Claims (4)

Accompaniment sound generating means for generating accompaniment information including at least an accompaniment sound;
Karaoke information control means for processing images taken by a video camera and controlling information related to karaoke;
A karaoke terminal device having output means for outputting a voice and an image of a singer,
A voice information storage area for storing the voice of the singer and the accompaniment information;
A reference image storage area for storing first control signal information that is control information of illumination output during a collection time of the series of reference images together with a series of reference images captured in advance;
A pre-combination image storage area for storing a pre-combination image collected at a certain point in time during which the output means outputs the sound and the image together with second control signal information that is control information of illumination output at the certain point of time ; ,
An extraction target part storage area for storing the extraction target part extracted from the pre-synthesis image;
A background image storage area for storing a background image selected by the singer to synthesize with the extraction target part, and
The karaoke information control means includes:
Voice information collecting means for storing the voice and the accompaniment information in the voice information storage area;
The pre-combination image stored in the pre-combination image storage area and the series of reference images stored in the reference image storage area, the time of the first control signal information , the time of the second control signal information, , Synchronize the frames of the same time axis, calculate the difference of pixel values, extraction target portion recognition means,
Target part extraction means for storing the difference portion as the extraction target part in the extraction target part storage area;
Synthesizing means for synthesizing the extraction target portion and the background image,
A karaoke apparatus in which a synthesized image synthesized by the synthesizing unit is output by the output unit.   2. The karaoke apparatus according to claim 1, wherein the karaoke information control unit synchronizes the accompaniment information, the sound, and the image synthesized by the synthesis unit and displays the image on the output unit. 3.   3. The karaoke apparatus according to claim 1, wherein the output unit includes a communication interface, and connects to a server to transmit the accompaniment information, the voice, and an image synthesized by the synthesis unit. The karaoke apparatus includes an operation unit operable by the singer,
The karaoke apparatus according to any one of claims 1 to 3, wherein the karaoke information control unit determines whether or not to perform a combining process by the combining unit based on an input from the operation unit .

Images