JP2004328788A - Method for compounding person video image separately photographed and background video image recorded and outputting to indicator, and karaoke apparatus adopting the method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To determine automatically synthetic conditions such as an appropriate position and dimension according to the contents of background video image, and hereby provide a person video image composite method for compounding a person video image for background without uncomfortable feeling according to the determination and a karaoke system which adopts the method. <P>SOLUTION: While a video data used as a background video image is accumulated in a database, a person composite prescription data which specifies an appropriate person composite layout is matched with the above video data. While taking out the appointed video data from the above database and reproducing a background video image, the person video image portion is started from the video signal photographed apart from this and the background video image is equipped with a video image composite means which gives a synthetic indicator. The video image composite means is the person video image composite method which performs variable setting of the synthetic qualification in what capacity the started person video image is compounded in which position of screen of the background video image, based on the person composite prescription data matched with video data outputted as the background video image. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a method of synthesizing and displaying a person image separately shot on a recorded existing background image, and particularly to a method of extracting only a person portion from an image shot by a video camera and synthesizing the same with a background image. . Also, the present invention relates to a karaoke apparatus employing the method.

  Anyone would have seen a video commentary by a weather forecaster on a weather map in a television broadcast weather forecast program. The technique of synthesizing a person image that combines a person being photographed with the video camera with the background image and outputs the synthesized image is not limited to television broadcasting. If there is a video camera that captures the foreground of a person or the like, a video device that plays back a recorded (recorded) background image, and an editing device that has a chroma key combining function, similar combined images can be displayed on a display. Therefore, this video synthesis technology is being widely used for various purposes for entertainment and hobbies. For example, an application of this technology can be seen in a recent karaoke apparatus.

  Normally, a karaoke apparatus outputs a mood image for a karaoke song on a display in synchronization with the audio output of the karaoke accompaniment music. A karaoke apparatus to which the above-described video synthesizing technique is applied reproduces an appropriate background video using a device for reproducing the mood video (such as a video CD player), and shoots the background video with a video camera. Chromakey composition of the singing person image. This makes it possible to create an image as if the singer is singing in a tourist attraction around the world or in a world of science fiction and fairy tales. In this way, by adding the person image synthesizing function to the karaoke apparatus, the entertainment of karaoke is improved, and the chance of using the karaoke apparatus is increased.

  In the conventional person image synthesizing method, a person to be synthesized is displayed with the same size at the end even if the content (scenery, structure, etc.) included in the background image changes. Certainly, the background video in television news and the like is a video of a news material, so it is more convenient that the display size of a person video such as an announcer does not change regardless of the video content.

  On the other hand, when compositing as if the person is actually present in a place where no person actually exists, it is more realistic to change the compositing state of the person (the display position and display size of the person on the screen) according to the background content. There is a feeling. If a person is synthesized in the background of a still and distant landscape, it may seem as if the person is present at the place without any discomfort. However, when the background image is zoomed in / out, the display size of the person must be changed according to the magnification of the background, and the background, which should not move, approaches or moves away from the person. Occurs.

  Furthermore, in the case where a person stands on the street corner of a building with forests, if the person is not synthesized with an appropriate display size according to the size of the building, the person looks like a large monster. I will. When a person is synthesized with a background image such as a sandy beach where the sea can be seen in the background, if the person is "floating" on the sea, this is also very unnatural in terms of image expression. To avoid this, it is only necessary to variably control the composition while watching the composite video of the background video being reproduced and the video of the person being shot, but this requires advanced editing techniques and complicated editing operations. Become. Of course, since it is a manual operation, it takes time to obtain an appropriate combination state.

  In the case of using a person image synthesizing function supplementarily and additionally, such as in a karaoke apparatus, there is no need for any operation other than the original operation (such as inputting music and adjusting the volume). I don't feel like trying to use that function until I do a complicated operation. For this reason, an increase in utilization opportunities due to the addition of functions cannot be expected, and only equipment costs increase.

  Therefore, the present invention provides a person image combining method for automatically determining a combination condition such as an appropriate position and size according to the content of a background image, and combining a person image with the background without discomfort according to the determination. The purpose of the present invention is to provide a karaoke apparatus that employs a karaoke device.

The first invention is specified by the following items (1) to (4).
(1) This is a method in which a separately shot human image is combined with a recorded background image and displayed and output.
(2) The video data serving as the background video is stored in a database. In this database, person composition prescription data that directly or indirectly specifies an appropriate person composition layout according to the content of the background image is associated with the image data.
(3) The specified video data is taken out from the database and a background video is reproduced, and a person video portion is cut out from a video signal separately shot and synthesized with the background video, and the synthesized video is displayed on a display. Video synthesizing means for outputting to the user.
(4) The image synthesizing unit, based on the person synthesizing prescription data associated with the image data to be output as a background image, maps the cut-out person image portion to what position on the screen of the background image and how large. Then, variably set the synthesis conditions such as whether to synthesize.

  A second aspect of the present invention is a person image synthesizing method in which the image data is digital image data and the person prescription data is multiplexed with the digital image data. The video data is recorded separately from the video data as data associated with the recording position of the data, and the video synthesizing means sequentially acquires the recording position of the video data being reproduced, and the person composition prescription data corresponding to the recording position. A third aspect of the present invention is a person image synthesizing method in which a synthetic layout is appropriately variably controlled on the basis of the above and synthesized and displayed.

The fourth invention is specified by the following items (41) to (44).
(41) A karaoke apparatus employing the person image synthesizing method according to any one of claims 1 to 3, wherein the background image is reproduced by extracting specified image data from the database, the image synthesizing means, and the image database. And a video camera.
(42) The music generation data that is the origin of the karaoke accompaniment music is stored for each karaoke song in the karaoke database. Karaoke performance processing means processes the music generation data of the designated music piece and outputs the accompaniment music as sound.
(43) A background video designation sequence is defined for each karaoke song. The video reproducing means sequentially reproduces the specified video data from the video database in accordance with the specified background video sequence of the specified music, thereby displaying the background video on the display in synchronization with the audio output of the karaoke accompaniment music. .
(44) According to the person composition prescription data, the video composition means appropriately variably controls the composition condition of the person image during the performance of the designated music piece and performs composition display.

Further, in the fourth invention, the lyrics rendering data that is the origin of the lyrics image is stored in the karaoke database for each karaoke song, and the lyrics rendering means processes the lyrics rendering data of the designated music and synchronizes with the accompaniment music. A fifth aspect of the present invention is a karaoke apparatus in which, when displaying the corresponding lyrics image on the display, the video combining means combines and displays the person image and the lyrics image on the corresponding background video.

  According to the person image synthesis method of the present invention, person synthesis prescription data expressing information on a layout when a person image is synthesized with a background image is associated with video data to be reproduced, and a synthesis display process is performed based on the prescription data. I'm going to do it. For this reason, a person is automatically synthesized at an appropriate position, size, or the like in a background video without requiring complicated operations or advanced editing techniques. In addition, it is possible to follow a change in the background image. Therefore, a sense of reality is given to the composite image as if the person were present at the shooting site of the background image.

  The data format can be appropriately selected according to the recording format (digital, analog, etc.) of the background video, such as multiplexing the human prescription data with the digital video data or using separate data according to the reproduction time series of the video data. . For this reason, an advanced person image combining function can be realized without depending on the data format of the background image.

  If this person image synthesizing method is applied to a karaoke apparatus, a singer can be synthesized with a mood image displayed on a display at the time of playing a karaoke song as a background image without discomfort. In addition, if a lyric image is also inserted, the singer or companion can view the composite video simultaneously with the lyrics.

=== Basic Configuration and Operation of Karaoke Device ===
FIG. 1 shows the configuration of a karaoke apparatus according to an embodiment of the present invention. The central control unit 11 is a microcomputer including a CPU, a RAM, and a ROM therein, and performs data communication with peripheral components via a data bus 100 and a control bus 110 to control and control the karaoke apparatus 1. .

<Data processed by the karaoke device and its storage unit>
The hard disk device 12 stores music IDs such as music generation data as the origin of the karaoke accompaniment music, lyrics rendering data as the origin of the lyrics image to be displayed on the display in synchronization with the music performance, and a background video designation sequence described later. Are stored as karaoke data.

  The DVD-ROM changer 23 stores a large number of DVD ROMs (hereinafter referred to as DVDs), and a built-in drive device drives an arbitrary DVD to randomly access an appropriate data recording position and read data recorded there. The data recorded on the DVD is multiplexed data mainly composed of digital video data of the MPEG2-VIDEO standard for a long time.

  The music generation data is sound source control data described by the MIDI standard, and describes at what point in the performance time sequence the synthesizer 13 should generate what accompaniment sound. The lyrics rendering data describes a formatted lyrics character, the timing of displaying / erasing the lyrics character string on the display, and a color changing pattern for the displayed lyrics character string. The background video designation sequence is data that specifies how to connect short-time videos displayed on the display 22 as the background video of the lyrics image, and which data of which DVD stored in the DVD changer 23 is recorded. Data that specifies how long playback from the position is described in chronological order.

<Karaoke performance processing>
When the central control unit 11 receives the ID of the requested music input from the operation input unit such as the remote control transmitter 17 or the operation panel 18 via the operation control unit 19, the central control unit 11 associates the input order with the ID and stores the ID in the RAM. , And make a music performance reservation registration. Then, karaoke data of the corresponding music is taken out according to the reservation order. The MIDI data is transferred to the synthesizer 13 to generate karaoke accompaniment music. The audio signal of the karaoke accompaniment music output from the synthesizer 13 is mixed with the singing audio signal from the microphone 15 and output as sound to the speaker 16.

  On the other hand, the lyric rendering data is processed in synchronization with the karaoke accompaniment music generation processing, lyric images that change color are sequentially generated as bitmap image data, and are developed in the video RAM 20. Further, the DVD changer 23 is controlled in accordance with the background video designation sequence, and the DVD changer 23 outputs multiplexed data including predetermined video data. The decoder 40 demultiplexes the read multiplexed data to extract digital video data, decodes the video data, and reproduces the video data as an NTSC signal corresponding to a background video. The video controller 21 performs a process of superimposing the lyrics image in the video RAM 20 on the background video and outputs the video to the display 22.

  The karaoke apparatus 1 of this embodiment has, in addition to the basic functions described above, a person image synthesizing function of photographing a singer who sings along with the karaoke music with the video camera 25 and synthesizing only the person part with the background image. .

=== Overview of the function for synthesizing human images ===
The person image synthesizing method according to the present invention is basically based on an image synthesizing technique such as chroma key synthesizing. In this embodiment, a chroma key combining technique is applied. In other words, the singer sings while standing at a predetermined position on the stage where the background wall and the stage are painted in a predetermined color (blue in this embodiment). The singer is photographed by the video camera 25. A blue portion (blue back portion) is cut out from the video captured by the video camera 25, and a video (person video) of only the singer is extracted. Then, the image control unit 30 performs a process of synthesizing the person image with the background image and displays the result on the display 22.

  However, the point that the present invention is decisively different from the conventional video synthesizing method is that the content included in the background video, for example, landscapes and structures, dynamic changes thereof (such as zooming and panning), A composite layout such as a display size and a display position of a person image can be appropriately automatically controlled and combined in accordance with the movement of a component (object: person or thing). In this embodiment, control data (person synthesis prescription data) expressing information on a synthesis layout is multiplex-recorded on a DVD together with digital video data in order to achieve this advanced person image synthesis function.

  In addition to the above-described superimposition processing of the lyric image, the video control unit 30 also performs A / D and D / A conversion of the video signal, chroma key synthesis, enlargement / reduction or movement of the video, and the like for the input video signal. A function for executing various image effect processing is provided. The image control unit 30 performs image effect processing on the image signals of the background image and the photographed image input based on the person synthesis prescription data, thereby achieving the above-described person image synthesis function. Has been achieved.

=== Person synthesis prescription data ===
The person prescription data is recorded on the DVD as multiplexed data according to the MPEG2-systems standard in the same time series as the digital video data. Then, in accordance with the content of the background video reproduced at the same time as the processing of the composite prescription data, the user specifies the size and display coordinates of the person video to be synthesized on the display 22 screen as a display size and a display coordinate. ing.

  The display size is a relative size when the height of the screen of the display 22 is set to 1, and when this value is 1, the composition is performed without changing the scale of the person image extracted from the captured image. If it is ２, a portrait image is reduced to 横 in height and width. If it is 2, the height and width are doubled. The display coordinates indicate the horizontal and vertical positions of the screen of the display 22 by x and y coordinates. The coordinates of the lower left corner are designated as (0, 0), and the coordinates of the upper right corner are designated as (640, 480). ing.

=== User interface and preliminary setting ===
When the user specifies a karaoke song, an appropriate input operation to use the person image synthesis function (for example, separately prepare a song number for using the person image synthesis function for the same song and specify the performance with that number) Is performed, the central control unit 11 reserves the performance by activating the person image synthesizing function when performing the music.

  Prior to the performance of this song, the singer stands on a special stage with a blue background. The position where the singer should stand is marked on this stage. The zoom magnification and the visual angle direction of the video camera 25 pointed at the stage are determined by a person having a normal physique standing on this mark (in this embodiment, a person of average adult height of Japanese) in a predetermined composition. Pre-fixed to fit. In the present embodiment, it is fixed in the center of the field of view in the horizontal direction so as to be fully displayed vertically from the feet of the person to the top of the head. The start / stop of shooting of the video camera 25 is controlled by an infrared signal from the video camera remote controller IF 26. Of course, the transmission path of the control signal may be wired.

  When starting playing music, the central control unit 11 instructs the video control unit 30 to activate a person video synthesis function. Further, it instructs the decoder 40 to transmit the multiplex-separated person prescription data. Further, it causes the video camera 25 to transmit an infrared signal to the video camera remote controller IF 26 to start shooting.

=== Operation of the person image synthesis function ===
When the music performance process is started, multiplexed data including digital video data of a background video is output from the DVD changer 23 in synchronization therewith. The decoder 40 separates the multiplexed data into digital video data and the above-mentioned person video prescription data. The video data is decoded in the decoder 40 and input to the video control unit 30 as an analog video signal. The person image prescription data is input as digital data to the image control unit 30 via the data bus 100.

  On the other hand, a video signal (analog signal) captured by the video camera is also input to the video controller 30. The video control unit 30 converts the captured video into digital data and executes a process for the above-described human video synthesis function. First, a human video portion obtained by removing a blue background portion from a captured video is extracted. Then, the person image data is processed into image data in which display dimensions and display positions are appropriately converted based on the person prescription data.

  Regarding the control of the display size, when the human image data is developed in correspondence with each pixel on the screen of the display 22, the pixels constituting the human image are reduced or enlarged by appropriately thinning out or interpolating. . In this way, the scale-converted person image data is generated.

Further, the display position of the scaled converted person image is moved according to the display coordinates. The starting point of the person image is the mark position of the stage where the singer stands. This base point is moved to the specified display coordinates (x, y). In addition, at the time of enlargement, since the upper part of the person may protrude from the screen, the y direction is shifted downward by a new pixel generated by interpolation. For example, if the designation of the display size is 1.5, the base point is shifted downward by 480 × 0.5 = 240 pixels. In this way, the face of the person always enters the screen. The human video data whose scale and coordinates have been converted are converted into analog video signals, and inserted into the background video by conventional chromakey synthesis. Further, it also synthesizes the lyrics image developed in the video RAM 20 and outputs the image to the display 22.

  FIG. 2 shows an outline of the above-described human image synthesis function as a display state on a display screen. Here, an example is described in which a singer is combined with a background image indicating that a car is parked in front of a building. In the conventional chroma key synthesis or when the display size is specified as 1 and the display position is specified as (320, 0), that is, in the case of performing the human image synthesis without performing the scale conversion or the moving process, the singer on the stage which is a blue background is identified. A person video (A1) is extracted from the captured video. This is chroma-key composited with the background video (B) to display the composite video (C1).

  On the other hand, according to the person image synthesis method of the present invention, the image (A1) is digitally processed to generate a person image (A2) whose scale and position have been changed. The video (A2) is chromakey-combined with the background video (B) to obtain a composite video (C2) that does not give a sense of incongruity in video expression.

=== Supplement, other embodiments, etc. ===
In the above embodiment, the scale of the person image is converted in accordance with the display size of the person prescription data regardless of the singer's physique. However, if the singer is tall, it may be out of the field of view of the video camera. In that case, the zoom magnification of the video camera may be changed in advance so that the singer just fits in the field of view of the captured video. Alternatively, the zoom magnification may be variably controlled while monitoring a person image extracted from a video image captured by a video camera, and the zoom magnification may be fixed when a blue-back color appears at the upper end of the screen. Conversely, when the singer is photographed small in the field of view of a video camera by a child or the like, the human image can be adjusted to the height of the display screen by changing the zoom magnification in the same manner. If the singer is within the field of view, the height of the person can be obtained from the extracted person video, so that the person's height may be adjusted to the screen height by image processing without changing the zoom magnification. .

  Furthermore, instead of specifying the size of the person to be synthesized with the background image with a relative value based on the height of the screen, the size (height and width, etc.) of a predetermined part such as the face of the person is specified. Is also good. In this case, a predetermined part may be specified from the cut-out person video by image recognition, and the whole person may be similarly deformed and combined so that the part has a specified size.

  When synthesizing a person image, in addition to the dimensions and positions described above, image effects processing that can be performed by the image control unit 30, such as rotating a person, making a mirror image, or copying and synthesizing a plurality of persons with a background image. If so, it can be specified as person synthesis prescription data and synthesized with a layout according to it. In addition, a part of a person image such as an upper half or a lower half of a person image can be combined and displayed.

  Further, it is easy to generate a background video by CG (computer graphics) technology. In this case, a display position of a human video can be designated by a three-dimensional position in a virtual space by CG. As a result, for example, a singer is placed on the stage of a concert hall, and the singer can be seen or hidden between large crowds in front of the stage, for example, the depth direction between the person synthesized in the background image and the object in the background image Can also be expressed.

  In the above embodiment, the predetermined position on the stage is processed as the base point of the person image. However, the singer may be able to track the specified position even when moving on the stage. The horizontal width of the person can be obtained from the extracted person video. The intersection between the center of the width and the stage surface (under the feet of the person) may be used as the base point. Alternatively, a video camera may be mounted on a camera platform that can freely rotate in the horizontal direction and control the rotation angle, and may track the singer so that the singer always appears in the center of the screen.

Further, the person prescription data need not be directly defined as numerical values as in the above-described embodiment. Any information on the layout at the time of synthesis may be used. For example, the contents of the background image may be expressed by keywords (sea, town, etc.), and dimensions and positions corresponding to the keywords may be prepared in a table or the like.

  It is apparent that the above-described method of synthesizing a person image can be applied not only to a karaoke apparatus but also to a case where images from a plurality of sources are synthesized by a personal computer, a video editing device, or the like. Of course, the background image and the person image need not be moving images, and may be still images. The same video compositing function can be realized if the human composite prescription data corresponding to the background video is prepared and the compositing process can be performed based on this data.

  For example, when the background video is recorded on a medium such as an analog optical disk, the person prescription data cannot be prepared as multiplexed video data. As described above, when the person prescription data cannot be recorded in parallel with the video data, the video recording position output at the time of reproduction, such as a frame number, a track number, or a reproduction elapsed time (linear time) from the recording start point of the video on the medium. (Frame data) and person-combined prescription data may be prepared in association with each other. Of course, when video data such as a disc changer is recorded over a plurality of media, the media ID is associated with the frame data. If this method is applied to a karaoke apparatus, a predetermined composite layout can be specified based on the frame data specified by the background video specifying sequence. Of course, the person prescription data may be separately prepared for each music as data associated with the performance time series.

FIG. 1 is a configuration diagram of a karaoke apparatus according to an embodiment of the present invention. FIG. 3 is a schematic explanatory diagram of a person image combining function in the embodiment.

Explanation of reference numerals

DESCRIPTION OF SYMBOLS 1 Karaoke device 11 Central control unit 12 Hard disk device 23 DVD changer 30 Video control unit 40 Decoder

Claims (5)

Inventions specified by the following items (1) to (4).
(1) This is a method in which a separately shot human image is combined with a recorded background image and displayed and output.
(2) The video data serving as the background video is stored in a database. In this database, person composition prescription data that directly or indirectly specifies an appropriate person composition layout according to the content of the background image is associated with the image data.
(3) The specified video data is taken out from the database and a background video is reproduced, and a person video portion is cut out from a video signal separately shot and synthesized with the background video, and the synthesized video is displayed on a display. Video synthesizing means for outputting to the user.
(4) The image synthesizing unit, based on the person synthesis prescription data associated with the image data to be output as a background image, displays the cut-out person image portion at what position on the screen of the background image and at what size. Then, variably set the synthesis conditions such as whether to synthesize.   2. The method according to claim 1, wherein the image data is digital image data, and the person composition prescription data is multiplexed with the digital image data.   2. The video recording apparatus according to claim 1, wherein the person-synthesizing prescription data is recorded separately from the video data as data associated with a recording position of the video data in the database, and the video synthesizing unit records the video data being reproduced. A person image synthesizing method, wherein a position is sequentially acquired, and a synthetic layout is appropriately variably controlled based on person synthetic prescription data corresponding to the recording position and synthesized and displayed. Inventions specified by the following items (41) to (44).
(41) A karaoke apparatus that employs the person image synthesizing method according to any one of claims 1 to 3, wherein the background image is reproduced by extracting specified image data from the database, the image synthesizing unit, and the image database. And a video camera.
(42) The music generation data that is the origin of the karaoke accompaniment music is stored for each karaoke song in the karaoke database. Karaoke performance processing means processes the music generation data of the designated music piece and outputs the accompaniment music as sound.
(43) A background video designation sequence is defined for each karaoke song. The video playback means sequentially outputs the specified video data from the video database in accordance with the background video specification sequence of the specified music, thereby displaying the background video on the display in synchronization with the audio output of the karaoke accompaniment music. .
(44) According to the person composition prescription data, the video composition means appropriately variably controls the composition condition of the person image during the performance of the designated music piece and performs composition display. 5. The karaoke apparatus according to claim 4, wherein lyric rendering data, which is the origin of the lyric image, is stored in the karaoke database for each karaoke song, and a lyric rendering unit processes the lyric rendering data of the designated tune into accompaniment music. When the corresponding lyric image is displayed and output on the display in synchronization, the video synthesizing unit synthesizes and displays the person image and the lyric image on the corresponding background image.

Images