JP2004005721A - Method and device for composing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent a residual background color from being generated when a subject is cut out to provide a natural composition result. <P>SOLUTION: This image composition method wherein a subject image with the background light portion is removed is cut from the subject image photographed under the condition of a background color of a specified color, and wherein the cut-out subject image is composed with a preliminarily selected background image to output a composed image therein is provided with the first step for generating a mask image of which the outline is expanded by a prescribed picture element number after the outline of the subject image is extracted when the subject image is cut out, the second step for cutting out the background light portion of the subject image by the generated mask image, and the third step for contracting, by the prescribed picture element number, the outline of the subject image with the background light portion cut out, to remove the outline. <P>COPYRIGHT: (C)2004,JPO

Description

The present invention relates to an image synthesizing method and apparatus for obtaining a high-resolution natural synthesis result with a simple configuration.

Conventionally, image synthesis systems are often used in broadcasting stations and movie shooting studios. In particular, image synthesis using processing by chroma key synthesis is widely used for movies such as weather forecasts, news, and special effects. Chroma-key synthesis is a method of determining a hue difference between a subject in an image for synthesis and a background color and obtaining a signal for image clipping. The object to be synthesized has a hue that differs from the color of the object as much as possible. An image is taken as a synthesis screen using a (distant) color background. Then, a key signal indicating contour information of the subject is created by performing waveform processing by determining the color of the background, and using the key signal, clipping processing of the inset portion of the base screen and clipping of the background of the synthesis screen are performed. Image synthesis is performed by superimposing a person to be synthesized with the background after performing the processing.

FIG. 32 is a diagram illustrating the relationship between illumination, a person, and a background when performing normal chroma key processing. In a case where image synthesis is performed using the chroma key synthesizing processing, the background of a person 3201 to be photographed by the camera 3200 is displayed. A blue cloth, which is a complementary color of human flesh color, or a large polyester plate 3202 colored blue, is erected, and a plurality of illumination light sources 3203 are used to illuminate the person to be photographed from multiple directions. .

As an image printing apparatus that can be seen on the street, for example, there is an apparatus described in Utility Model Registration No. 3034012 or an apparatus configured as shown in the block diagram of FIG. 33. This is almost the same system as printing a photo on photographic paper with a speed photographing machine, etc., and combining a portrait image of the photographed user with a separately prepared frame image such as a background screen, frame image, character, etc. To the user, or a photo of only the upper body, such as an ID photo. In this case, the appearance of the apparatus is configured such that an “eave-shaped” curtain for blocking external light during photographing is installed on the front of the photographing unit in the image printing apparatus main body.

The conventional image print creating apparatus shown in FIG. 33 includes a CPU 3301, an operation unit 3302 including a touch panel, various operation buttons, a joystick device, and the like, an image input device 3303 including a CCD camera and the like, and a portrait (face image) of a user. , A voice operation unit 3305 having a speaker and a voice ROM for generating a synthesized voice, a printing unit 3306 for printing out a print sheet or a sticker, and a coin detecting unit for adjusting a usage fee. A depositing device 3307 provided, a ROM 3308 storing background pattern data, combined pattern data, and a program for image combining processing, and an AD converter for converting an image signal of a portrait of a user photographed by the image input device 3303 into digital image data. 3310, image data after conversion I / O interface 3311 is stored in memory 3309, for performing editing of the image data RAM3312, it is composed of a hard disk 3313 or the auxiliary storage medium 3314 for storing various data.
Japanese Utility Model Registration No. 3034012

In the above-described conventional technology, when performing chroma key processing at a broadcasting station or a shooting studio, the background of a subject is set to a blue background, and a plurality of illuminations from multiple directions are set so that a shadow of a person is not reflected on the blue background. To create a shadowless state. However, image synthesis using chroma key processing using such a blue background requires a large number of lighting devices to be installed in order to create a shadowless state. Lighting equipment needed to be installed. Therefore, there is a problem that sufficient facilities required for processing image synthesis using chroma key processing cannot be prepared in a broadcasting room of a school or a local station such as a CATV.

Similarly, in the conventional image printing apparatus, when it is desired to perform synthesis so that the synthesized result of the image looks natural, it is possible to use the image synthesis using the above-described chroma key processing. It is conceivable, however, that the space of the housing is limited, so that there is a problem that it is not possible to provide lighting devices from multiple directions for creating a shadowless state in the housing. Further, in the shooting of a studio or the like, there is a problem that the lighting condition changes every time the shooting target moves, the brightness of the background changes, and the composition is not successful.

To solve this problem, increase the distance between the background and the subject, illuminate the background itself directly, avoid clothing with colors similar to the background color, and adjust the saturation and brightness of the key colors. There was a problem that it was necessary to take troublesome measures such as increasing the width.

In addition, in the conventional image printing apparatus, since the composition is such that the background and the photographed subject are simply superimposed, the composition result is such that the subject emerges with respect to the background, and the composition result is unnatural. On the other hand, there is a problem that a result that naturally dissolves cannot be obtained.
Further, even if the saturation or brightness of the background color is adjusted, there is a problem that when the subject is cut out, the background color remains unremoved, and a natural synthesis result cannot be obtained.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide an image synthesis method that can prevent a background color from being left behind when a subject is cut out, and can obtain a natural synthesis result. And to provide a device.

In order to achieve the above object, the image synthesizing method of the present invention cuts out a subject image excluding a background light portion from a subject image taken in a lighting state of a specific color of background light, and selects the cut-out subject image and a preselected subject image. In an image combining method of combining a background image and outputting the combined image,
When extracting the subject image, a first step of extracting a contour line of the subject image and generating a mask image obtained by expanding the contour line by a predetermined number of pixels, and a background of the subject image based on the generated mask image The method is characterized by including a second step of cutting out the light part and a third step of contracting the outline of the subject image from which the background light part has been cut out by a predetermined number of pixels to remove the outline.
The image processing apparatus further includes a fourth step of performing a smooth filtering process on a peripheral pixel of the subject image from which the contour is removed.

The image synthesizing apparatus according to the present invention cuts out a subject image excluding a background light portion from a subject image captured in an illumination state of a background light of a specific color, and combines the cut-out subject image with a previously selected background image, In the image synthesizing apparatus for outputting the synthesized image, when cutting out the subject image, after extracting a contour line of the subject image, a first means for generating a mask image in which the contour line is expanded by a predetermined number of pixels Second means for cutting out the background light portion of the subject image by the generated mask image, and third means for removing the contour line by shrinking the contour line of the subject image from which the background light portion has been cut out by a predetermined number of pixels. It is characterized by having.
The image processing apparatus may further include a fourth unit that performs a smooth filtering process on a peripheral pixel of the subject image from which the contour is removed.

ADVANTAGE OF THE INVENTION According to this invention, when clipping processing of a subject is performed, generation | occurrence | production of the missing of a background color can be prevented from occurring, and the synthetic | combination image which looks as if the subject melt | dissolves in the background naturally can be obtained. .

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
1 and 2 are diagrams showing the principle structure of the present invention. FIG. 1 is a plan view of the inside of a casing when the device of the present invention is viewed from above, and FIG. It is the housing | casing inside side view.
In these figures, the camera 101 is set inside the housing at an upper position such that the whole body of the subject (person) enters the field of view. Then, lighting is performed on the person (subject) 104 from the lighting devices 102a to 102d on the front side in the shooting direction of the camera 101 and the lighting device 103 on the ceiling, and further on the back surface and bottom surface (floor surface) of the person 104. By illuminating the installed surface-emitting display panels 105 and 106, shadows generated on the back and feet of the person 104 are offset by the emission of the surface-emitting display panels 105 and 106. The photographing of the person 104 is performed in a state where the shadows generated on the back and feet of the person 104 are canceled. Each numerical value described in the drawings is a numerical value suitable for canceling a shadow, but is appropriately changed depending on the size of a person, the size of a housing surrounding an outer frame, and the illuminance of illumination.

FIG. 3 is an internal side view showing an embodiment of the image synthesizing apparatus 300 to which the above-described principle structure is applied. A camera 303 is installed at the position, and a lighting fixture 304 that illuminates the person 302 from the head direction is installed at the ceiling. Further, surface light emitting panels 305 and 306 corresponding to the surface light emitting panels 105 and 106 of FIG. 1 are provided on the back surface and bottom surface (floor surface) of the person 104.

In this case, in particular, it is desirable that the surface light emitting display panel 305 on the rear side be large enough to cover at least the person 302 as the subject from the head to the feet. Further, it is preferable that the surface light emitting display panel 306 at the bottom portion occupies at least 30 cm to 50 cm from the rear surface of the housing where the surface light emitting display plate 305 is installed in order to cancel the shadow generated at the foot.

Further, below the camera 303, a display device 307 that has a touch panel on the display screen surface and displays a composite image is provided, and further below the camera 303, a printer 308 that prints out the composite image is provided. On the back side of the display device 307, a control device 309 for controlling the entire device including image synthesis processing is provided. Note that, in FIG. 3, a lighting device that illuminates the person 302 from the front is omitted.

FIG. 4 is a block diagram showing a functional configuration. The image synthesizing apparatus according to this embodiment includes a CPU (Central Processing Unit) 1 that performs overall control. The CPU 1 is connected to a memory 2, an I / O control unit 3, a data storage unit 4, an image output unit 5, an image processing unit 6, a pattern memory 7, and an image input unit 8. The I / O control unit 3 is provided on a communication control I / F (interface) 9, a digital camera I / F (interface) 10, an inverter 11, a sensor 12, a touch panel 13, a coin control unit 14, a back surface and a bottom surface. It is connected to the light emitting unit / light adjusting unit 15 that adjusts the light emission of the surface illuminator and the lighting fixture that illuminates the person 302 from the front, and controls the whole.

(4) The data storage unit 4 sends and receives various data to and from the auxiliary storage medium 20 such as a floppy disk. The image output unit 5 is connected to a video printer 18 and a display device 19, respectively.

The I / O control unit 3 controls the inverter 11 to apply a voltage to the surface-emitting display panels 305 and 306 connected to the light-emitting unit / light-adjusting unit 15 to cause the entire light-emitting surface to emit light uniformly. The image input unit 8 converts image data captured by the CCD camera 17 from an analog signal to a digital signal using an A / D converter (not shown). Since the CCD camera 17 is equipped with a wide-angle lens, it is possible to capture images in a wide range of images both vertically and horizontally even at a short distance. Further, the saturation and lightness of light emitted from the surface light emitting display panels 305 and 306 installed on the back and the bottom are measured.

The CPU 1 receives compressed digital signals from the image input device 200 and the digital camera I / F (interface) 10, extracts a contour of the whole body image of the subject, and performs distortion correction of the taken whole body image. Further, the background color by the surface-emitting display panels 305 and 306 was compared with the brightness and saturation of the captured image, and the saturation and brightness of the whole-body image were corrected so as to match the saturation and brightness of the background color. Thereafter, the program is combined with an arbitrary background image selected in advance by the user according to the program stored in the memory 2 and the print output processing is performed on the video printer 18. Further, this data is stored in an auxiliary storage medium 20, such as a floppy disk, PD, or MO. When stored, it is encrypted by an encryption program such as PGP. The data to be encrypted here is privacy-related data such as an image and a name and a telephone number input from an input device (touch panel).

The memory 2 stores the image data taken in from the image input unit 8, and also stores, as a program, a command issued to the CPU 1 and a command group for controlling the image synthesizing apparatus. The processing data of the CPU 1 is also temporarily stored as work data. Further, data of the background illuminated at the time of shooting and the saturation and brightness of light from the surface light emitting display panels 305 and 306 installed on the bottom surface are stored.

The pattern memory 7 stores background image data to be superimposed on the image of the subject, and also stores a guide for an operation menu to be displayed on the display device 19. The contents of the pattern memory 7 can be rewritten by the data storage unit 4. The communication control I / F 9 is used for connection with an external network. This device is connected to an external device management server by a communication control I / F 9 via a dedicated line. The purpose of this device management server (not shown) is to manage the presence or absence of a failure.

{Circle around (2)} Also, a new background image or character image is transmitted from the device management server and stored in the data storage unit 4 via the I / O control unit 3. The data storage unit 4 checks the date of the data written in the pattern memory 7 and, if the data stored in the data storage unit 4 is new, rewrites the data in the pattern memory 7. As will be described later, by providing the optical communication control I / F, a portable information terminal such as a PDA or a personal computer can be directly connected, and binary data such as a program can be exchanged.

The touch panel 13 is provided on the surface of the display screen of the display device 19, and is used, for example, to display an operation menu on the display device screen and to input various selection conditions by a pressing operation. The CPU 1 detects, via the A / D converter, a resistance value when the operation menu is pressed and pressed at the position of the displayed operation menu, which is set in close contact with the screen, and calculates the coordinates on the screen. Which button is selected is determined. The sensor 12 detects the presence or absence of a user at the operation site by, for example, an infrared sensor, and confirms whether or not the user is at a predetermined position after inserting a coin, and whether or not there is one or more users. belongs to. The coin control unit 14 detects whether or not a coin has been inserted and whether or not an illegal coin has been inserted, under the control of the CPU 1, measures the shape and weight of the inserted coin, and determines whether the coin is not a genuine coin. Then, the pattern of the front and back of the coin is determined, and it is determined whether the inserted coin is a genuine coin, a fraudulent coin, or a counterfeit coin.

(4) The inverter 11 performs flicker prevention control on the surface-emitting display panels 305 and 306. The flicker prevention period can be manually changed according to the external light control. The light emitting unit / light adjusting unit 15 adds a light emitting function / light adjusting function in order to control a firing angle for applying a voltage to the CCD camera 17 as an imaging unit. Specifically, the surface-emitting display plates 305 and 306 use an EL plate, a PIS, or the like. Further, a blue film or a green film is pressed on the entire surface of the surface light-emitting display panels 305 and 306 to emit a color other than white or flesh color such as blue or green. The surface is processed with acrylic or the like to prevent surface reflection. Of course, bead processing or the like may be used.

The image processing unit 6 extracts the contour of the face from the image data of the subject stored in the memory 2 and the color data of the face stored in the pattern memory 7, and arranges the face at an arbitrary position in the space on the memory 2. Is performed in cooperation with the CPU 1 or independently, and processing such as sending the synthesized image data to the video printer 18 is performed. The video printer 18 prints and outputs a color image on a postcard or a sticker.

{Circle around (2)} A process of extracting a face contour from the image data stored in the memory 2 and the face color data stored in the pattern memory 7 and arranging the contour at an arbitrary position in the space on the memory 2 is also performed. The timer 21 is provided in the apparatus main body when there is no environment in which the timer 21 can be connected to an external device management server by a dedicated line, and operates according to an instruction from the CPU 1. The optical communication control I / F 22 controls IrDA for performing infrared communication and controls when communication using an optical fiber is performed. By using this interface, binary data such as a program can be exchanged, and it can be directly connected to a personal digital assistant such as a PDA or a notebook personal computer. Programs and data. Further, data or a program created by a personal computer can be sent to the image synthesizing apparatus of the present embodiment, and can be arbitrarily changed to a background pattern or a character.

(4) The photographing of the user is performed by the CCD camera 17 when the photographing instruction to the user is instructed on the display screen of the display device 18 after the coin is inserted. As the imaging device, a camera equipped with a C-MOS, a normal video camera, or the like may be used in addition to the CCD camera 16. Image data captured by the image input device 200 such as the CCD camera 16 is subjected to exposure adjustment by an AE (auto iris) function built in the CCD camera 17 and the like, and is stored in the memory 2. In addition, for example, a digital still camera having an Ir-DA optical communication function may be used as the imaging device. In this case, an Ir-DA optical communication function that can receive and transfer previously captured image data and a buffer memory that can store a plurality of image data allow the user to select an icon serving as a selection condition of a menu image, and perform JPEG A plurality of compressed still images can be easily synthesized.

When the sensor 12 detects that the user is standing on the operation unit, the detection signal causes the CPU 1 to generate an interrupt so that the display on the display device 19 is switched from the demonstration screen to the operation guide screen. Further, by inserting coins, the back and bottom surface light-emitting display panels 305 and 306 are controlled to be ready for light emission, and the shutter of the CCD camera 17 is released, that is, the back and bottom surfaces are synchronized in synchronization with the shooting timing. The light emitting display panels 305 and 306 are controlled to emit light. Note that the surface light-emitting display plates 305 and 306 may start emitting light at the stage when the user inserts a coin.

The photographing fee is received by the coin control unit 14, but the fee receiving means is not limited to the coin detector, and a bill reader, a prepaid card reader, a credit card, or the like can be used. The photographed image data is converted into RGB data after photographing, and displayed on the display device 19 via the image output unit 5. A suitable screen resolution of the display device 19 is, for example, 320 × 240 pixels per screen.

(4) The display device 18 provided in the present image synthesizing device needs to display a menu screen required at the stage of operation by the user and confirm whether the synthesized image or the like created by the user is appropriate. These screen displays are controlled and displayed by the CPU 1. Further, since the operation states of display on the screen, image processing, and the like are managed by the CPU 1, during the print processing, the display can be switched to an image or video stored in the pattern memory 7 in advance. When the operation screen is unnecessary, the menu is switched to broadcast commercials, product advertisements, and the like. The display device 18 is not limited to a CRT display device, but may be a liquid crystal LCD display device or a character display device using LED light emission. 4 are formed on a semiconductor substrate or a glass epoxy substrate by a known semiconductor integrated circuit application technique, for example. And it is installed in the control device 309 of FIG.

FIG. 5 is a flowchart showing the initial setting of the color tone of the surface emitting display panels 305 and 306 in the present invention.
First, the apparatus is first turned on in the morning or the like (step 501). When the power is turned on, a voltage is applied to the illuminating devices provided on the front and the ceiling of the device housing and the surface emitting display panels 305 and 306 on the back and the bottom, and light emission is started (step 502). When the surface-emitting display panels 305 and 306 emit light, the CCD camera 17 provided on the main unit automatically performs photographing once to measure the saturation and brightness of the surface-emitting display panels 305 and 306 (step). 503). The saturation and brightness are stored from the photographed image (step 504), and the base color of the image to be cut out of the image used in the chroma key processing is adjusted. The stored color tone information is registered in the memory 2 and initialized (step 505).

FIG. 6 is a schematic flowchart showing image processing in the present invention. The captured image is taken into the memory 2 (step 601). It is determined whether or not distortion correction is to be performed on the captured image with reference to information on whether the image is a whole-body image or an upper-body image at first by the user (step 602). If the whole-body image has been selected, distortion correction is performed because unnatural distortion has occurred (step 603). If a half-body image has been selected in step 602, the process proceeds to step 604. The color tone of the photographed subject is compared with the color tone of the selected background (step 604), and the color tone of the subject is adjusted to match the color tone of the background side (step 605). Perform synthesis. The combined result is displayed (step 606), and the result is printed and output.

Next, image distortion correction will be described with reference to FIGS. In the case of the image synthesizing apparatus according to the present invention, a wide-angle lens is used as the lens of the CCD camera 17 in order to take a whole-body image from a short distance. Because of the use of a wide-angle lens from a close distance, “tilt” (due to the angle of the optical axis of the camera) causes distortion, such as close clogging, in which the center of the lens looks large, resulting in an unnatural image. turn into. However, the distortion is not caused only by the distortion of the lens, but is caused by various causes. For this reason, a corrected coordinate system for performing distortion correction cannot be defined by a single and simple function.

In such a geometric transformation of the image, a method is used in which each pixel of the original image is made to correspond to the transformed pixel by using a transformation formula. Since two or more pixels correspond to the same pixel, the value of each pixel on the converted image is determined using density interpolation. In the density interpolation, a set of coordinates is given, and the density value is obtained by using linear interpolation using four pixel values around the coordinates. The linear interpolation means that a density value is obtained from a ratio of distances of four neighboring points.

In the present invention, a corresponding point of each point is calculated from a set of pairs of a point Pn (Xn, Yn) and a corrected corresponding point P'n (X'n, Y'n) on the actually captured coordinates, An image that looks unnatural is corrected to an image that looks natural by using linear correction and density correction.

FIG. 7 shows the calculated corresponding points of each point from a set of pairs of points Pn (Xn, Yn) on the actually photographed coordinates and corresponding points Pn (Xn, Yn) after correction. FIG. 8 is a diagram showing a result of correcting an actually photographed image. FIG. 9 is a diagram showing how the correction result shown in FIG. 8 is based on the processing result. In the present invention, all correction data of the photographing coordinate system and the corrected coordinate system shown in FIG. 7 are prepared in advance. A method of obtaining a point (X, Y) on the photographing coordinate system corresponding to the point (X ', Y') on the corrected coordinate system using the correction data obtained in FIG. 7 will be described.

First, in order to find (X, Y), a corrected coordinate system is obtained by using the correction data from the shooting coordinates, and four points near the shooting coordinates and the corresponding correction coordinates are obtained. In this case, it is assumed that the corresponding points on the corrected coordinates are points arranged in a grid. Four points on the correction coordinates are obtained, and (Xp, Yp) and (Xq, Yq) on the shooting coordinates are obtained by the following equations. By obtaining (Xp, Yp) and (Xq, Yq), (X, Y) corresponding to (X ', Y') is obtained.

Calculation example Xp = X1 + (X2-X1) (X-Xa) / (Xb-Xa) Yp = Y1 + (Y2-Y1) (Y-Ya) / (Yb-Ya) Xq = X3 + (X4-X3) (X −Xa) / (Xb−Xa) Yq = X3 + (Y4-Y3) (Y−Ya) / (Yb−Ya)
How to calculate (X, Y) X = Xp + (Xq−Xp) / (Y−Ya) / (Yb−Ya) Y = Yp + (Yq−Yp) / (X−Xa) / (Xb−Xa) FIG. 10 shows the calculation result by the linear correction. The point (X, Y) on the photographing coordinates is obtained from FIG.

{Next, FIG. 10 shows (68.84, 116.13) using density interpolation from the obtained (X, Y). Therefore, the density of (X ′, Y ′) (100, 100) corresponding to (X, Y) ((68.84, 116.13)) is the density of the point (68.84, 1163.13) on the shooting coordinates. It is calculated from the density values on four neighboring grid points, that is, (68,116), (68,117), (69,116), and (69,117). The density d ′ (100, 100) of the point (100, 100) on the coordinate system after correction is d ′ (100, 100) = d (68.84, 116.13) = 4 points ((68, 116) , (69,116), (68,117), (69,117)) are obtained by linear interpolation.

(A) of FIG. 11 is a diagram showing the coordinates of the image actually taken before the correction, and (b) is a diagram showing the coordinates after the correction. The data shown in FIG. 11 is only a part of the area in the coordinate space, points arranged in a grid, and therefore, points not arranged in the grid exist in the coordinate space. In this case, the correction in the area not on the shooting coordinates is obtained by extrapolating the corresponding data of the grid points. In the image distortion correction according to the present invention, it may be desirable to use a linear interpolation method for obtaining corrected coordinates from four neighboring points.

Next, details of image composition will be described with reference to FIG. The center coordinates are provided for the image of the photographed subject (person), and the position is aligned with the background (step 1201). This alignment is performed on the light source. The contour of the aligned subject image is cut out (step 1202). A region where the skin color of the subject is distributed most is determined from the extracted contour and the center coordinates, and the color tone of the image is determined (step 1203).

(4) The determined color tone is compared with the color tone of the background image selected by the user in the first operation (step 1204). Based on the comparison result, the color tone of the image of the subject is corrected so as to correspond to the color tone of the background (step 1205). When the correction is completed, the subject image and the background image are combined.

Next, each step of the image synthesis in FIG. 12 will be described in detail for each function. For the purpose of the present invention, the following four items are indispensable factors in how to combine a person and a background so as not to give a sense of discomfort.

(1) Match the color tone, saturation, and brightness. (2) The light source position is appropriate. (3) The position and size of the person are appropriate. (4) The outline of the person is blurred.

手法 With respect to the above items, the method of the present invention particularly for (1) and (4) will be described. Incidentally, the description of (2) is omitted because the light source position at the optimum position has already been determined for the ordinary image synthesizing apparatus. In the case of (3), since the physical characteristics of the user who uses the image synthesizing apparatus are individual, and it is impossible to determine the automatic position and size from the background image for each user, the selection is made. The position where the image of the person photographed according to the background image is embedded in the background image is set as a fixed value.

First, chroma key synthesis will be described. For clipping the subject, a process called soft key chroma key process for performing color synthesis in consideration of a difference from a designated color is used. As a method of specifying color data, a specification method using RGB and HSV is used.

FIG. 13 shows parameters for specifying a range of colors in the RGB system. In a normal RGB system, the ranges of R, G, and B colors are manually determined. When the pixel is determined, it is determined within the range from which the pixels of the maximum value and the minimum value of R (Red), the maximum value and the minimum value of G (Green), and the minimum value of B (Blue) are removed.

In FIG. 13, the color range of Red designates a darker gradation than other Green and Blue, the color range of Green designates a wide range from a lighter gradation, and the designation of a color range of Blue designates a darker gradation as a narrower range. To be specified. The specification of these color ranges can be varied depending on the color of the subject to be extracted. At the time of chroma key synthesis, the outline of the image to be extracted is most clearly determined using the color “blue” which is the most distant from the other colors. Therefore, when the background is blue, the ideal numerical value is RGB = (0, 0, 255), and the color is extracted with a certain range around this numerical value. In the case of the RGB system, since ordinary image data is in the RGB format, there is no need for conversion and no need for conversion. As a disadvantage, if the distribution of the background color selected is large, the color width of RGB must be increased. Therefore, there is a possibility that the color and skin of the clothes of the unexpected person may come off.

FIG. 14 is a conceptual diagram of the HSV format. In the HSV method, a color perception is represented by three variables such as H (hue), S (saturation), and V (luminance). H and S are independent variables. The value of each variable decreases as the distance from the center increases. That is, H (hue) decreases, S (saturation) decreases, and V (luminance) decreases. In the case of the HSV format, since the color changes continuously, when the width of the blue color is broadened with respect to the blue color which is the base color of the blue background in the background of the subject, a bluish color is lost but yellow without discontinuity There is a characteristic that can not be extinguished and red. As a disadvantage, image data is usually in RGB format, so there is a need for conversion, and the conversion takes a long time.

ブ ロ ッ ク Also, when the image data to be used is JPEG, the block at the time of JPEG compression appears in color. In the present invention, a subject image is extracted by specifying a color using one of the RGB and HSV formats, and it is most preferable to use the two systems by switching them according to the color of the subject. Is used to extract a contour image.

Next, (1) adjusting the color tone, saturation, and brightness, that is, adjusting the color tone, saturation, and brightness of the background image and the subject image will be described. FIG. 15A is a histogram showing the distribution of the luminance of the background image, and FIG. 15B is a histogram showing the distribution of the luminance of the human image as the subject. Comparison between FIG. 15A and FIG. 15B shows that the distribution does not match the distribution of the luminance of the background. It can be seen that the brightness of the background image in FIG. 15A is dispersed on average, while the person image in FIG. 15B is bright at a certain specific place and the other parts are dark. In this case, since it is necessary to reflect the dispersed brightness of the background image in the subject image, the center of gravity and the bias of the luminance are obtained from the histogram of the luminance of the background. This can be determined by matching the luminance parameters.

変 換 A conversion function f (x) (FIG. 16B) of the histogram of the subject is created from the obtained center of gravity and bias data. Next, the brightness value of the subject is converted by f (x).

FIG. 16A is a diagram showing a subject image before brightness conversion, and FIG. 16C is a diagram showing a subject image after brightness conversion by a conversion function f (x). It can be seen that the luminance distribution of the background image was reflected on the person image as the subject by this conversion function.

FIG. 17 shows a histogram function of luminance conversion, and this conversion function is automatically generated based on the luminance distribution of the background image. Here, a curve for distributing the luminance on average over the entire gradation and a conversion function that does not change the luminance level for a certain gradation or higher are shown.

Next, the adjustment of the saturation will be described. If the background image has vivid colors, the background image is converted to the above-described HSV format, the average of S (saturation) is calculated, and when the average value of S of the subject image is lower than that of the background, After increasing the value of S, it is converted to RGB.

彩 Saturation conversion using HSV data will be described with reference to FIG. In the case of the color conversion based on the HSV data, first, the saturation histogram of the subject image and the background image is obtained, and the position of the center of gravity of each histogram is obtained. In FIG. 18A, the center of gravity of the background image is placed in a color close to Green. Next, the respective centers of gravity of Blue, Red, and Green in FIG. 18B are obtained. As can be seen from FIG. 18A, since the colors of Blue and Green are prominent here, the center of gravity of Blue and Green is obtained in FIG. 18B. An intermediate is determined from the determined centroids, and a complementary color to the intermediate color is determined. The subject histogram is converted at the position of the obtained intermediate color complementary color. Here, since the background image has a strong saturation of Blue and Green, the saturation is reflected on a person image as a subject with a boundary between complementary colors of intermediate colors of these saturations.

で は In step 1402 of FIG. 12, the subject is cut out. However, after the cut-out, the background color may be left behind and the background color may remain. Background color missing often occurs in (1) the vicinity of the outline of a person, and (2) where the shadow is dark. This may be because a shadow is likely to fall near the contour of the subject, and a different color appears on the boundary between the subject and the background color. If you try to specify a color range that removes only the shaded areas, the color range becomes wider and the colors of the clothes that are very similar to the background color (dark blue, etc. when the background is blue) tend to fall out. Would. Also, the phenomenon that different colors appear on the border, for example, the orange color appears on the blue and yellow borders, is due to the optical characteristics of the camera, so adjusting the lighting conditions alone will not solve the problem. I don't care. For this purpose, correction is performed using the following method.

Next, a method of shaving the contour by image processing will be described. When performing chroma key synthesis, first, a binary mask image is generated. At the time of this generation, expansion and contraction, which are techniques for removing noise from the binary image, are performed, and the outline is expanded by several pixels before the outline is cut. First, an algorithm for dilation / contraction, which is a technique for removing noise, will be described with reference to FIG. If even one pixel is white in the adjacent pixels (8 pixels), it is expanded to be rewritten to white at its own position. Conversely, if even one pixel is black, its position is also replaced by black. is there.

FIG. 20 is a diagram showing noise removal by an adjacent area as another method. Count the number of pixels with the same attribute (white, black) as the own in eight neighborhoods of the pixel of interest, and invert the pixel if the pixel with the same attribute is a certain number or less (a threshold is set in advance). This expands the contour. By performing this operation on the entire screen, noise is removed. In the noise elimination using the dilation / shrinkage algorithm in FIG. 19 and the noise elimination in the adjacent area in FIG. 20, the processing time is short using the noise elimination in the adjacent area. Is a method using an expansion / contraction algorithm. In addition, if the number of thresholds is increased in noise removal based on adjacent areas, a phenomenon in which pixels disappear in a chain may occur, resulting in an unnatural image. On the other hand, in the noise removal using the expansion / contraction algorithm, the outline tends to be rounded, and the outline looks more natural.

FIG. 21 is a diagram showing that the background near the contour is image-processed, the contour is expanded, and then the contour is scraped off so that the contour line is eliminated.

When performing chroma key synthesis, the boundary between the background image and the human image, that is, the outline may be cut off in the middle or may not be clear due to optical unevenness, and the outline may not be extracted properly. In the present invention, as shown in FIG. 19.20, the outline of a person image to be extracted is traced, a binary mask image of the outline is generated, and the mask image is enlarged by 1 to 4 pixels to enlarge the mask image. I do. Thereafter, the mask image on the outline is cut off, thereby eliminating the omission of the background when the chroma key is synthesized.

FIG. 22 is a diagram showing contour blurring. When performing image synthesis, a phenomenon in which a difference in color or luminance optically causes a different color to occur on a boundary line between a subject and a background color due to aberration of a lens, that is, so-called jagging may occur. In the present invention, as shown in FIG. 21, in order to extract a person image from a background image, the outline is expanded by masking the outline. At this time, as shown in FIG. 21, a mask image is first generated in the outline portion, and the generated mask image is scanned in the horizontal and vertical directions, and the position where the signal changes, that is, the outline is generated. Edge information is written at a position where a line image is read. An edge means that a noticeable difference occurs in the density distribution between adjacent areas. A 5 × 5 smoothing filter is applied to the contour to add a blurring effect to the contour. The 5 × 5 smoothing filter is a filter that, as shown in FIG. 20, sets a certain pixel as a center value and replaces the pixel value of the center value with the average of eight neighboring pixels. The detected edge is 1 pixel wide, and the effect is low even if the “blurring effect” is performed only on the edge. Therefore, the edge is expanded to 3 pixels using the expansion processing described above. Apply a smoothing filter. By applying the smoothing filter, the outline is blurred, and jaggies can be eliminated.

Next, a description will be given of a color tone adjustment that affects and reflects the color tone of the background image on the color tone of the subject. When the color of the background image has an extreme color such as green, red, or yellow, it is more natural to see the effect of the color on the subject. So how do you match the effect of the background color to the color of the subject? In this method, the effect of the hue of the background can be adjusted to the hue of the subject by automatically extracting the parameters of the hue from the RGB histogram of the background image and correcting the hue.

First, an RGB histogram (256 levels from 0 to 255) of a certain color extracted from the background image in FIG. 23 is created. The vertical axis indicates the color gradation of each of RGB, and the horizontal axis indicates the degree of brightness. As the value increases, the brightness increases. Next, the total frequency of each histogram of level 128 or more is counted, and the center value is obtained. Next, a conversion function of each RGB value is determined.

で は In FIG. 24, a primary conversion of Y = aX is used. In FIG. 24, assuming that the coefficient a of the primary conversion has a variable B as a reference value in the conversion and the center position of the histogram at level 128 or higher is p, the slope a of the straight line is expressed by the following equation. a = p / B For the colors (R, G, B) in which the center value of the histogram is higher than B, the histogram moves to the right as a whole. Also, an upper limit and a lower limit are determined for the slope a, and the range of the coefficient a is limited. This is to prevent extreme color correction. This is because if the color is corrected excessively, the influence from the background greatly affects the color of the subject, so that the synthesis result becomes very unnatural.

Next, the operation procedure of the image synthesizing apparatus of the present invention will be described with reference to FIGS. FIG. 25 is a diagram showing a procedure for determining settings such as the number of shots and the background frame by the user. The demo screen is in a state of waiting for a user to insert a coin (step 2501). At this time, a message indicating that the music or the device itself is not currently used may be played. When coins are inserted by the user, the screen message changes, and when the number of inserted coins is small, the amount of the shortage is displayed, and the replenishment amount is displayed (step 2502). In addition, the shortage may be indicated by voice. When a predetermined coin is inserted, a screen for selecting a shooting method is displayed (step 2503). In the present invention, the user who wants to take a picture is asked to select any one of a plurality of persons, one body, one body and one and a half. Next, a background screen as a background is displayed. Although not specifically shown here, the background is, for example, the sea, a famous tourist spot in or outside the country, a scene of a movie, etc., and scrolls by pressing the left and right buttons to select from a plurality of screens. (Step 2504). In the apparatus according to the present invention, a digital camera I / F may be provided on the front side of the apparatus, a digital camera owned by a user may be connected, and an image photographed by the user may be used as a background.

FIG. 26 is a diagram showing the transition of the shooting screen. When the background screen to be photographed is determined according to the procedure shown in FIG. 25, the photographing mode is set (step 2601). Next, the user who is currently taking a picture is displayed on the display device screen by the CCD camera 17 (step 2601). Here, the background selected by the user in step 2504 of FIG. 25 has already been displayed on the screen, and here the user adjusts his or her standing position according to the background displayed on the screen. The photographing is automatically performed three times, and the user selects the most preferable one of the photographed results (step 2603). Next, the user switches to the color change or graffiti mode (retouch function selection screen) for the selected screen (steps 2604 and 2605). The screen may be shifted to the next input screen without any modification to the screen according to the user's intention. The color change can be selected according to the user's preference such as color, sepia, black and white. In the graffiti mode, characters can be input as well as graffiti.

FIG. 27 is a diagram showing a screen for selecting an output size. Several different sizes can be selected by the user (steps 2701, 2702). When a postcard, a postcard, or the like is output, a selection screen that allows the user to specify the vertical and horizontal directions is displayed.

FIG. 28 is a flowchart showing the input of personal information regarding the user. First, a name and a nickname are input (step 2801). All input items in this flowchart are arbitrary, and can be canceled if the user does not want to input. Next, an age is input (2802), and a telephone number is input. (Step 2803) When inputting a telephone number, the type of mobile phone such as a mobile phone or PHS is also selected (Step 2804). In order to confirm whether or not the input content is correct, the input content is displayed again (step 2805). The age to be input here is analyzed from the accumulated age data, based on the age-based usage according to the installed location, and becomes development data such as a background and a frame pattern later.

FIG. 29 is a flowchart showing approval for publication in a magazine. First, the user confirms whether or not the photographed image is to be published in a magazine (step 2901). If the user selects not to publish, the screen shifts to the printing confirmation screen. If the user wants to be published in a magazine, he or she is required to input personal information in the same manner as in FIG. 27 (step 2902). On this input screen, the real name needs to be contacted because it is a post to a magazine or the like, and an input is required in any of alphanumeric characters, kanji, hiragana, and kana characters (step 2903). When the input of the name is completed, the nickname is input (steps 2904 and 2905).

FIG. 30 is a flowchart showing the input of a questionnaire and the selection of a magazine desired to be published. First, the publication approval to the magazine input in FIG. 29 is confirmed again (step 3001). This suggests that minors such as children and students may be users, and that publication of magazines and the like requires permission from parents and school. Next, it is confirmed whether or not the questionnaire number described in the magazine to be published is known (step 3002). If the questionnaire number is known, the questionnaire result is answered by the number (step 3003). Next, the user inputs the desired magazine to which magazine the user wants to publish (step 3004) and shifts to the print screen.

FIG. 31 is a flowchart showing screen transition during printing. While printing is being performed, "printing" is displayed (step 3101). Here, it is not only displayed that printing is being performed, but a character, a CM, or the like may be flowing. When printing of the combined image is completed on a post card or a sticker, "print completed" is displayed (step 3102). The user may confirm by voice. Next, it is confirmed whether to continue using the device (step 3103). Here, a certain period is displayed. For example, a countdown is displayed on the display screen, and when ending, the user is asked to confirm the end. If you continue, you will be charged additional money during the countdown. Here, it is confirmed whether to print the same image which has been shot and synthesized up to now, or to take a new photo again (step 3104). If the printing is to be ended, the user is informed of the end, and the user is made to confirm that the print is output, and the processing is ended (step 3105).

In the above embodiment, an example has been described in which the composite image is printed and output in the print gist, but the present invention is not limited to this. For example, the present invention can be applied to a case where a composite image is created in an environment where a space for creating a shadowless state cannot be secured, such as a broadcast room in a school or a small-scale shooting studio.

FIG. 2 is a plan view of the inside of the housing showing the principle configuration of the present invention. FIG. 3 is a side view of the inside of a housing showing a principle configuration according to the present invention. It is an inside side view of a case showing an embodiment of an image composition device by the present invention. FIG. 4 is a block diagram illustrating a functional configuration of the image configuration device of FIG. 3. 6 is a flowchart illustrating an initial setting according to the present invention. 5 is a flowchart illustrating an outline of an image processing procedure according to the present invention. FIG. 4 is an explanatory diagram showing actually photographed coordinate points and corresponding points after correction. FIG. 9 is an explanatory diagram showing a result of correcting an image captured in the present invention. FIG. 9 is an explanatory diagram showing a result of coordinate conversion using correction in the present invention. It is an explanatory view showing an example of actual coordinate conversion. FIG. 9 is an explanatory diagram illustrating an example of a correction result. 5 is a flowchart illustrating details of image processing according to the present invention. FIG. 9 is an explanatory diagram showing color designation by RGB. FIG. 3 is a conceptual diagram illustrating color designation by HSV. It is a figure showing a histogram of the brightness of a background and a person. FIG. 9 is an explanatory diagram showing transition of luminance conversion. FIG. 4 is an explanatory diagram showing a histogram function. It is explanatory drawing which shows adjustment of a color tone. It is explanatory drawing which shows an expansion algorithm. It is explanatory drawing which shows adjacent noise removal. It is explanatory drawing which shows contour cutting. It is explanatory drawing which shows a contour blur. It is a figure showing a histogram of RGB. FIG. 3 is an explanatory diagram showing a color tone conversion function. 5 is a flowchart illustrating an operation procedure of the image composition device according to the present invention. 5 is a flowchart illustrating an operation procedure of the image composition device according to the present invention. 5 is a flowchart illustrating an operation procedure of the image composition device according to the present invention. 5 is a flowchart illustrating an operation procedure of the image composition device according to the present invention. 5 is a flowchart illustrating an operation procedure of the image composition device according to the present invention. 5 is a flowchart illustrating an operation procedure of the image composition device according to the present invention. 5 is a flowchart illustrating an operation procedure of the image composition device according to the present invention. It is explanatory drawing which shows the lighting method in the conventional studio photography. FIG. 10 is a diagram illustrating a block configuration of a conventional image synthesizing device.

Explanation of reference numerals

1 CPU
2 memory,
3 I / O control unit 4 Data storage unit 5 Image output unit 6 Image processing unit 7 Pattern memory 8 Image input unit 9 Communication control I / F
10 Digital camera I / F
DESCRIPTION OF SYMBOLS 11 Inverter 12 Sensor 13 Touch panel 14 Coin control part 15 Light emission part / dimming part 17 CCD camera 18 Video printer 19 Display device 20 Auxiliary storage medium 21 Modem 22 Timer 305,306 Surface-emitting display board

Claims (4)

An image synthesizing method for cutting out a subject image excluding a background light portion from a subject image taken in a lighting state of a specific color background light, combining the cut out subject image with a previously selected background image, and outputting the synthesized image At
A first step of extracting a contour line of the subject image when extracting the subject image, and then generating a mask image obtained by expanding the contour line by a predetermined number of pixels;
A second step of cutting out a background light portion of the subject image by the generated mask image;
A third step of contracting the contour of the subject image from which the background light portion has been cut out by a predetermined number of pixels to remove the contour. 2. The image synthesizing method according to claim 1, further comprising a fourth step of performing a smooth filtering process on a peripheral pixel of the subject image from which the contour is removed. An image synthesizing apparatus that cuts out a subject image obtained by removing a background light portion from a subject image captured in a lighting state of a specific color background light, combines the cut subject image with a previously selected background image, and outputs the combined image. At
First means for extracting a contour of the subject image when extracting the subject image, and then generating a mask image obtained by expanding the contour by a predetermined number of pixels;
Second means for cutting out the background light portion of the subject image by the generated mask image;
An image synthesizing apparatus comprising: a third unit configured to shrink the outline of the subject image obtained by cutting out the background light portion by a predetermined number of pixels and remove the outline. 4. The image synthesizing apparatus according to claim 3, further comprising: fourth means for performing a smooth filtering process on a peripheral pixel of the subject image from which the contour line has been removed.