JP4399462B2 - Image processing apparatus, method, and program - Google Patents

Description

  The present invention relates to a technique for extracting a specific image from a photographed image and synthesizing it with another image. In particular, the present invention reduces noise of the synthesized image and is used even under conditions that require real-time performance such as a game. The present invention relates to an image processing apparatus, method, and program.

  Conventionally, there is a so-called blue back technique for extracting a specific image such as a person from a photographed image and combining it with an arbitrary background image. This means that the background of the image to be photographed is blue, the portion corresponding to this blue is regarded as the background by the color recognition means, and this portion is replaced with another background image. Since it is sufficient to recognize this specific color and replace it with the background, colors other than blue are also possible, but when extracting a person image, if the color to be replaced with the background is blue, the color of the person's skin and the complementary color It is possible to reduce the possibility that the relationship is close and a part of the face or the like becomes the background, that is, noise is added.

  This blue back technique has a problem that, for example, if a person to be extracted wears clothes having the same color pattern as that of the background, the pattern is replaced by the background.

On the other hand, a technique has been proposed in which a background only image and an image in which an extraction target such as a person is copied together with the background are photographed, and only the extraction target is extracted from the difference between the two images to perform noise processing. . (See Patent Document 1). In this technique, a mask image (silhouette image) is created from a difference between pixel information (for example, RGB data) of both images, and noise is deleted by editing.
JP 2000-36032 A

  By the way, the above-described conventional technique is not intended to automatically detect and remove noise by a computer. For example, as a video game, a background image recorded in advance according to a game scenario is used. When extracting only the image of a person who is a game player who is always shooting with a camera and combining it with the background image of the game, the CPU is overloaded and the real-time performance is impaired. There is. Also, a device that executes image processing by a computer such as a game device has a memory capacity limitation.

  The present invention has been made in view of the above-described circumstances, and automatically eliminates image noise without imposing a load on a computer when a background portion is deleted from a captured image and combined with another image. An object is to provide an image processing apparatus, method, and program that can be detected and removed or alleviated.

  In order to achieve the above object, an image processing apparatus according to the present invention includes a background image, an image input means for inputting an extraction target image including the extraction target in the same background as the background image, and the background image and the extraction target image. An image separation means for creating a silhouette image having a non-background pixel and a background pixel based on the difference between the pixel values of the image, and inputting the silhouette image, and marking an area surrounded by the non-background pixels for each line of the input image If this mark area touches a background pixel in an adjacent row or column, the mark area is sequentially changed to a background area, and the remaining mark area is converted to a non-background area to edit the silhouette image. Pixels from the edge of the image area of the silhouette image edited by the noise removal means and the background noise removal means based on the digitized non-background degree In addition, a noise mitigation means that sequentially performs non-background degree averaging processing on the neighboring area of the pixel and then corrects the image area range, and for alpha synthesis based on the non-background degree that has been averaged and the extraction target image Image output means for generating an image and outputting the image together with a separately input background image is provided.

The basic idea of the present invention is as follows.
Noise that occurs when the difference between the background image and the extraction target image is taken as background noise that considers part of the extraction target as the background, non-background noise that considers part of the background as the extraction target, and extraction target And there is edge noise that the background boundary part bites into other areas.

  Among these, regarding the background noise, the closed background area in the non-background area is regarded as noise, and the closed background area is scanned and converted into the non-background, thereby removing the noise. On the other hand, non-background noise and edge noise are made inconspicuous by applying a semi-transparent gradation at the edge portion in consideration of the tendency that these are small regions and thin regions.

  According to the present invention, background noise can be removed, non-background noise and edge noise can be reduced, and image processing quality can be improved without imposing a load on the CPU. In addition, since gradation is applied to the boundary between the background image and the non-background image, a natural composite image can be created with the non-background image familiar with the background image.

  Further, in the present invention, it is not necessary to store a copy of the silhouette image, and noise can be removed or reduced while directly editing the image. Therefore, the memory capacity can be reduced, and hardware resources can be saved. Can be planned.

The image processing apparatus according to the present invention further creates an image (edge extracted image) obtained by extracting the edges of the silhouette image created by the image separating means, and combines the edge extracted image and the silhouette image to create a non-background. The image processing apparatus includes preprocessing means for creating a silhouette image with an enlarged region, and the background noise removal means performs background noise removal processing using the silhouette image created by the preprocessing means as an input image.
By creating a silhouette image in which the non-background area is enlarged, the so-called mouth part of the boundary between the non-background area of the edge noise and the background area is closed and converted to background noise, and this background noise is removed by means of background noise removal. To do. As a result, the edge noise can be removed, thereby improving the image quality.

  Note that it is preferable to enlarge the non-background area, convert the edge noise into background noise, perform the background noise removal process, and then reduce the non-background area toward the original size.

  The image processing apparatus according to the present invention further takes a vertical or horizontal histogram of the silhouette image created by the image separation means, and edits the background noise removal means or noise mitigation means based on the width of the histogram. Alternatively, a silhouette region extracting means for determining a correction range is provided.

  By limiting the silhouette region, it is possible to reduce the load on the computer required for filtering processing such as noise removal and mitigation.

  In the image processing apparatus according to the present invention, preferably, the image separation means determines whether the pixel is a non-background pixel or a background pixel by comparing a difference between pixel values of the background image and the extraction target image with a threshold value. It is preferable to include a threshold adjustment unit that determines and displays an image that can be determined based on the threshold and can determine the degree of noise, and that changes the threshold based on a threshold change request input from the input unit.

  Thereby, noise processing can be performed in a state in which noise is suppressed to some extent, and the quality after synthesis can be predicted.

  More preferably, the range of the extraction target is specified with the noise level displayed. In this way, the outside of the range is unconditionally set as the background and only noise processing within the range needs to be performed, so that the burden on the computer is reduced and the speed of the image processing can be increased. Furthermore, by enabling the operator (user) to perform noise processing of background noise, non-background noise, or edge noise, the computer's load is reduced and the quality meets the operator's requirements. Image processing becomes possible.

  Instead of inputting the extraction target range by the operator, range designation information (for example, a frame line) indicating the extraction target area is displayed in advance on the display unit, and the image input means displays the range designation information. Based on the above, an extraction target image in the range may be input and processed.

  Here, the “image processing device” means a device having an image processing function, and includes a general-purpose computer device, a game device having the function, a video phone, an imaging device such as a camera, and the like. .

  The “non-background degree” is set for each pixel or pixel group of the extraction target image, and indicates the degree to which the pixel or pixel group is non-background (extraction target). When the non-background level is represented by 8 bits, it takes a value between 0 and 255. In addition, it is sufficient if it has the significance of non-background level. Therefore, the background level is also included.

  An image processing method according to the present invention is an image processing method for extracting a specific image from an input image and synthesizing it with another image, and includes a background image and an extraction target including the same background as the background image. A target image, a step of creating a silhouette image having a non-background pixel and a background pixel based on a difference between pixel values of the background image and the extraction target image, and a non-background pixel for each line of the silhouette image. When the enclosed area is marked, and the area touches the background pixel of the adjacent row or column, it is changed to the background area sequentially, and the remaining mark area is converted to the non-background area and the silhouette image is edited. Then, based on the digitized non-background degree, the non-background degree averaging process of the neighboring area of the pixel for each pixel from the edge of the image area of the silhouette image Performing sequentially, then correcting the image area range, and generating an α composition image based on the non-background degree averaged and the extraction target image, and outputting together with the separately input background image. It is characterized by including.

  According to the present invention, a series of processes for noise removal and mitigation can be performed while directly editing a silhouette image, thus saving memory.

  A program according to the present invention is a computer game program for displaying a game player image taken by a camera in the background of a game and playing the game, and a process of inputting and storing the background image; An image that creates a silhouette image having non-background pixels and background pixels based on the process of inputting an extraction target image including a player with the same background as the background image and the difference between the pixel values of the stored background image and the extraction target image For each line of this silhouette image, mark an area surrounded by non-background pixels, and if that area touches a background pixel in an adjacent row or column, it will be changed to the background area in order and remain Based on the background noise removal process that edits the silhouette image by converting the mark area to the non-background area, and the non-background degree that is digitized, A noise reduction process that averages the non-background degree of the neighboring area of the pixel for each pixel from the edge of the image area of the silhouette image edited by the background noise removing means, and then corrects the image area range, and the average Output based on the non-background level and the extraction target image that have been processed, and output together with the background image that is created by the process of executing the game or input from the external storage device And causing the computer to execute the processing.

  A series of noise processing can be performed while directly editing the silhouette image, and the memory of the game apparatus can be saved. In addition, the noise mitigation process averages the non-background degree in the vicinity of the pixels, so that the hit rate is increased and the processing speed is improved.

  Preferably, the α composition image creation processing and the game execution processing can be operated independently. Since the α composition image stored when the game player needs to be displayed is used, the responsiveness during the game can be improved.

  According to the present invention, it is possible to automatically detect image noise, remove background noise, and mitigate non-background noise and edge noise. Further, since a series of noise processing can be performed while directly editing the silhouette image, it is possible to save the memory area.

  In addition, in the game program, when a foreshadow image (non-background image) such as a game player shot with a camera is combined with a background image of the game, a gradation is applied to the boundary portion, so the game player blends into the game background. It is possible to provide a more realistic game.

Embodiments of the present invention will be described below.
<First Embodiment>
FIG. 1 is a block diagram of an image processing apparatus 1 according to the first embodiment of the present invention. Here, the image processing apparatus 1 includes an A / D conversion unit 11 that inputs a video signal captured by the camera 51 and converts it into a digital signal, a processing unit 12 that executes image processing, a memory 13 that stores image data, Display memory 14 for storing display data to be sent to the TV monitor 52, D / A conversion means 15 for converting a digital signal into a signal for a TV monitor, and a drive device for a storage medium such as a CD-ROM or DVD-ROM (external Storage device) 16 and an input unit 17 such as a mouse or a keyboard.

  As shown in FIG. 2, the processing unit 12 includes an image input means (process) 21, an image separation means (process) 22, a threshold adjustment means (process) 23, a background noise removal means (process) 24, and a noise mitigation means (process). 25 and image output means (processing) 26. In addition, it is assumed that a background image is stored in the memory 13 in advance.

  Next, the operation of each means will be described. In the following description, the image data is described in the RGB system for the sake of convenience. However, it is assumed that it is appropriately converted to another unit system such as a YCrCb system if necessary according to the specifications of the camera 51 and the television monitor 52.

[Image input processing]
First, in response to a request from the input unit 17, the image input unit 21 of the processing unit 12 is activated, and image data captured by the camera 51 is input and stored in the memory 13. At this time, an image of only the background (background image) not including an extraction target such as a person and an image including the extraction target (extraction target image) are respectively photographed and stored in the memory 13. 3 is a background image, and FIG. 4 is an example of an extraction target image. In FIG. 4, a person to be extracted (for example, a game player) is photographed with the same background as in FIG.

[Image separation processing]
Next, a difference for each pixel between the background image and the extraction target image is obtained by the image separation unit 22. In other words, the total value of the differences between the background image and the extraction target image for each of RGB of 8 bits for each of RGB (24 bits in total) is calculated. When the total value is equal to or less than a predetermined threshold value, the value (non-background degree) corresponding to the pixel is represented by 8-bit data A, and A = 0. On the other hand, when the total value is larger than the predetermined threshold, the pixel is set as an extraction target (non-background), and A = 255.

  In this way, a binarized silhouette image is created based on the background image and the extraction target image. The method of comparison with the threshold value is not limited to the above, and binarization is performed based on the difference between the two images, such as the average value of the differences between the RGB values as a threshold value. It ’s fine.

  FIG. 5 is an explanatory diagram of a silhouette image. Here, the blacked out part is the background (A = 0) and the white part is the extraction target (A = 255).

  However, since the actual silhouette image has noise as shown in FIG. 6, this noise processing is required. As types of noise, background noise that considers part of the extraction target shown in FIG. 6A as the background, non-background noise that considers part of the background shown in FIG. 6B as the extraction target, There is edge noise in which the boundary between the extraction target and the background shown in 6 (c) bites into another region.

  Background noise appears when the operator's clothes are close to the background color. Non-background noise appears due to background pixel errors due to flickering of fluorescent lamps, etc., and appears as an arbitrary spot in the background. Edge noise appears as a combination of background noise and non-background noise, and the edges become jagged.

  These noises can be reduced to some extent by adjusting the threshold value by the threshold value adjusting means 23, but may not be completely removed when the background and the color to be extracted are approximated. In addition, there is a certain limit such that non-background noise increases when background noise is reduced. Therefore, it is necessary to remove or mitigate these remaining noises.

Hereinafter, the noise processing procedure will be described with reference to FIGS.
[Background noise removal]
The background noise removal process includes the processes in steps S301 to 304 in FIG.

  First, in background noise candidate region mark processing, a background pixel (A = 0) sandwiched between non-background pixels (A = 255) is marked for all horizontal lines in the silhouette image ( S401, S402). For example, gray (for example, A = 128) is assigned as the mark. FIG. 7 is an explanatory diagram of the background noise removal process. In this figure, pixels filled in black (unit squares in the figure) are background pixels, and white pixels are non-background pixels. FIG. 7B shows a state after the processing of steps S401 and S402 is performed for each line and this is performed for all lines. Here, the gray pixel is a marked pixel.

  Then, in the candidate area upper confirmation process, the line segment marked from the top to the bottom is scanned for all lines, and for at least one pixel of the marked line segments, the pixel one above is scanned. In the case of the background (A = 0), the entire mark portion in contact with the pixel is repainted with the background (S501, S502). That is, the A value of the marked pixel is changed from 128 to 0. FIG. 7C shows a state where the process has been completed up to an intermediate line, and FIG. 7D shows a state after the process has been completed for all lines.

  When the above processing is completed for the lowermost line, next, in the candidate area lower confirmation processing, all lines are scanned in the reverse direction, that is, the line segment marked from the bottom to the top. If at least one pixel of the marked line segment is the background, the entire marked line segment is painted with the background (S601, S602). FIGS. 7E and 7F show a state where the process has been completed up to the middle line, and FIG. 7G shows a state after the process has been completed for all the lines.

When all of the above are completed, the pixels remaining as marks are background noise and are repainted as non-background (S701, S702). FIG. 7H shows the state of the silhouette image after the background noise removal process.
Background noise is removed by the above processing.

[Reduction of non-background noise and edge noise]
Next, the procedure for mitigating non-background noise and edge noise will be described with reference to FIGS.

  For all the pixels on the image, a 4 × 4 pixel area is scanned in the lower right direction and replaced with the average of the A values in the area (S801, S802). For example, the circled pixels in FIG. 8B are background pixels (A = 0), but when averaged, A = 255 * 3/16 = 48. Here, the symbol “*” means multiplication, and “/” means division.

  Further, although the pixel in FIG. 8C is a non-background pixel (A = 255), the average is A = 255 * 3/16 = 48 as in FIG. 8B.

  This averaging process is first performed for the pixels from the left to the right of the uppermost line, and when this is completed, the lower line is sequentially performed. This process is repeated until the rightmost pixel of the bottom line. FIG. 8D shows a state after completion of a series of averaging processes.

  If the accuracy of the boundary portion of the calculation area becomes a problem, an image may be acquired in advance so as to sufficiently include the calculation area. If the accuracy of the boundary does not matter so much, after averaging from the bottom row (bottom line) to the top four rows, the bottom line is filled with the same value as the pixel above it Anyway.

  As a result of the above averaging process, the silhouette image is shifted in the upper left direction. For example, the right pixel is shifted by 2 pixels in the lower right direction, that is, by shifting the whole of the average calculation unit area 4 × 4. (S803). (State of FIG. 8 (e))

  It is not necessary to actually rewrite the pixel value on the memory, and the address information of the area in which the silhouette image is stored need only be shifted to the upper left by 2 × 2.

  In the above description, for the sake of convenience, the terms “horizontal” and “vertical” are used. However, when the image data is expressed by the coordinates of the X axis and the Y axis, either of the above processing may be performed vertically or horizontally, but pixel information is stored. If the direction in which the addresses to be processed are arranged in the horizontal direction, a processing program can be easily created.

[Image output processing]
Image data for α synthesis (alpha blending) is created using the extraction target image data of FIG. 4 using the silhouette image after the series of noise processing described above as α channel data.

  Here, the image data for α synthesis is configured as an array of data of 8 bits each of R: G: B: A (32 bits in total) for each pixel.

  Then, the composition background data read from the CD-ROM of the external storage device 16 is written in the display memory 14, and the image data for α composition is similarly written in the display memory 14, and then a display output command is issued. As a result, both image data are combined and displayed on the television monitor 52. FIG. 9 is an explanatory diagram of this. Image data for α synthesis is created from the image data to be extracted based on RGB values and silhouette image data based on A values (FIG. 9A), and this is combined with background data based on RGB values (FIG. 9B). Specifically, the background data in FIG. 9B is drawn and the image data for α composition in FIG. 9A is overwritten. As a result, a composite image is displayed (FIG. 9C). At this time, the extraction target image displayed on the background is completely transparent (A = 0), completely opaque (A = 255), and in an intermediate state (0 <A <255) depending on the A value corresponding to the pixel. .

  As a result of the above noise processing, background noise is removed, non-background noise and edge noise are alleviated, and the A value becomes an intermediate value and gradation is applied to the boundary portion between the background and the target image. It becomes familiar with the background and becomes a natural composite image.

  Note that each unit of the processing unit 12 may be activated by a request from the input unit every time, but may be automatically activated upon completion of the preprocessing.

  According to the present embodiment, it is possible to automatically perform noise processing without applying a load to the CPU to perform image synthesis.

  For background noise removal processing in particular, first mark the area surrounded by non-background pixels for each line, and if the area touches the background pixel of the adjacent row or column, then change to the background area sequentially, Since the remaining mark area is removed as background noise, processing can be performed while editing the silhouette image itself, and it is not necessary to secure a temporary memory for processing, so that memory can be saved. In addition, since the memory area to be scanned and the memory area to be written are close to each other, the cache hit rate is increased and the speed can be improved.

  In addition, with regard to noise reduction processing, averaging processing is performed sequentially from the edge of the region, and finally the region range is corrected. Therefore, the output image of the background noise removal processing can be edited as it is, for processing. There is no need to reserve temporary memory. Similarly to the background noise removal process, since the memory area to be scanned and the memory area to be written are close to each other, the cache hit rate is increased and the speed can be improved.

<Second Embodiment>
Next, a second embodiment will be described.
The configuration is the same as in FIGS. 1 and 2, but the image processing device 1 in FIG. 1 functions as a game device. As the input unit 17, a game controller is used.

  A storage medium such as a CD-ROM or a DVD-ROM stores game data and image data to be displayed as the background of the game video. Further, information on the insertion position of the image of the game player (extraction target) is stored in association with each frame of the game video.

  The storage medium includes image input means (process) 21, image separation means (process) 22, threshold adjustment means (process) 23, background noise removal means (process) 24, noise mitigation means (process) 25, image processing means ( In addition to (processing) 26, a program for causing a computer to execute game execution means (processing) 27 for progressing the game is stored.

  When the storage medium is inserted into the external storage device 16 and activated, necessary programs stored in the CD-ROM are loaded into the program area of the memory 13 and activated by the processing unit 12.

[Preparation stage]
First, as a pre-preparation before starting the game, a background image and an extraction target image (player image) are input, and a silhouette image is created by taking a difference between the two images by an image separation unit.
The background image is stored in the memory 13 and used for image separation during the subsequent game execution. Then, the threshold value of the A value (0 or 255) of the silhouette image is adjusted.

  When inputting a player image, in order to limit the position at which the player (operator) wants to stand, as shown in FIG. 11, a silhouette is displayed on the image including the player to guide the position of the player. Is preferred. In FIG. 11, a guidance instruction “Please stand and play in silhouette” is displayed on the screen. By limiting the position where the player exists in the image, the range of image analysis can be limited and the processing load on the CPU can be reduced.

[Threshold adjustment processing]
Although the threshold for background / non-background separation can be set to a certain degree based on experience, it is not possible to limit the environment for generating an image including the player. A rough threshold is a value of about 40 in the range of 0-255.

  As one example of the user screen for threshold adjustment, there is a method of enabling fine adjustment by displaying a slider as illustrated in FIG. The player changes the threshold by moving the slider on the screen to the left and right, and directly confirms the composite image of the change result on the screen. In FIG. 12A, since the threshold value is low, that is, even when the difference between the pixel values of the background image and the extraction target image is small, it is determined as non-background (extraction target), and thus non-background noise is greatly generated. FIG. 12B is an example in which the threshold is appropriate and almost no noise is generated. In FIG. 12C, when the threshold value is high, it is not regarded as non-background unless the difference in pixel values is large, and therefore, the extraction target is bitten into the background by edge noise.

  The other is a method of sampling a plurality of representative threshold values as shown in FIG. 13, preparing respective setting buttons, and roughly setting the player by selecting the button. For example, assign a low (about 20), medium (about 40), and high (about 60) threshold to each of the three buttons, and select the appropriate button by displaying the composite image corresponding to each setting button. . As a result, the threshold value of the selected button is set.

When this advance preparation is completed, the game is started.
As image processing during the game, player image extraction processing and image output processing that is combined with the game background image and output are performed independently.

[Player image extraction processing]
The player's image is periodically fetched, and image input processing, image separation processing, background noise removal processing, and noise reduction processing are executed (S105 to S108).

In the image input process, the player image is input through the camera 51, and the process after the image separation process is executed using this image and the background image stored in the memory 13 by the pre-processing.
Since these processing procedures are the same as those in the first embodiment, a description thereof will be omitted.
As the output of the noise reduction processing, image data for α synthesis is created and stored in the memory 13.

[Image output processing]
Next, image output processing during game execution will be described. Note that the data passed from the game module (program for executing the game) for image output is a frame image, output position information, and deformation information, as shown in FIG. Here, the output position information is information for designating the output position of the player image, and is represented by coordinates (address range) on the screen. Further, the deformation information includes not only the case where the image is deformed vertically and horizontally, but also a part of the image that is cut out and enlarged, reduced, rotated, and color change as a clip image. In addition to these pieces of information, a flag indicating whether or not to output a player image may be provided.

  In the image output process to the television monitor 52, first, image data of a game background frame is output (S201). Next, it is determined whether or not output of the player image is necessary based on the presence of the flag or the output position information of the player image. If not necessary (“N” in S202), a display output command is sent (S206). Then, the process proceeds to the next frame output process.

  On the other hand, when it is necessary to output the player image (“Y” in S202), output position information and deformation information are acquired (S203). Then, the α composition image data stored in the memory 13 is acquired and edited based on the deformation information (S204). Then, it outputs to the coordinate position of output position information (S205), and sends a display output command (S206). The above processing is repeated for each display frame.

(Supplementary explanation regarding α composition etc.)
The above is an explanation for the RGB system, but for the YCrCb system, it is as follows.
In the YCrCb system, Y: Cr: Cb = 8: 8: 8 bit, this is converted to R: G: B = 8: 8: 8 bit, and the generated A = 8 bit is synthesized, and R : G: B: A = 8: 8: 8: 8 bit pixel information.

  At this time, the formula of α composition (alpha blend) is as follows. Basically, the hardware is implemented in the graphic chip.

(Cs-Cd) * As + Cd (1)
here,
Cs: Source RGB value
Cd: RGB value to write to
As: Source alpha value (theoretically takes a value between 0 and 1)
Cs and Cd are each a value between 0 and 255 for each RGB. The composition processing is performed based on the above equation (1).

  According to this embodiment, the player image extraction process and the image output process during the game are executed independently, and when the image output process needs to be combined with the player image, it is stored in the memory at that time. Since the player's α composition image data is used, the waiting time due to the player image extraction process can be reduced, so that the real-time property can be improved.

  In addition, the background noise removal processing and noise reduction processing do not require copying of the captured original image data, and the processing can be performed while directly changing the original image data itself, so that the memory capacity can be saved. it can.

  Note that when the player's clip image is combined with the CG background, the player's clip image is rotated and enlarged / reduced to generate a more realistic composite image.

<Third Embodiment>
Next, a third embodiment will be described.
In the present embodiment, by performing preprocessing with respect to edge processing with respect to the first or second embodiment, thin edges are removed, and edge noise is converted into background noise. The background noise can be removed.

  FIG. 23 is a block diagram of the image processing apparatus according to the present embodiment. 2, a pre-processing unit 29 and a post-processing unit 30 are added before and after the silhouette region extracting unit 28 for extracting the range of the non-background region of the silhouette image and the background noise removing unit 24, respectively.

  FIG. 24 is a flowchart illustrating a processing procedure of the image processing apparatus 1 according to the present embodiment. In this figure, each process of step S901, step S906, step S908, and step S909 is the same as that of the first embodiment.

  Hereinafter, the operation of the image processing apparatus 1 according to the present embodiment will be described with reference to FIGS. 24 to 34, focusing on differences from the first embodiment.

[Image Separation Processing] (Step S902 in FIG. 24)
The image separation unit 22 creates a silhouette image by the procedure described in the first embodiment using the background image and the extraction target image that are the outputs of the image input unit 21, and then activates the silhouette region extraction unit 28.

[Silhouette Area Extraction Processing] (Step S903)
The silhouette area extraction means 28 takes a histogram for this silhouette image, compares it with a predetermined threshold value, and determines that the area narrower than that is noise and removes it.

  FIG. 25 is a flowchart showing the processing procedure of the silhouette region extraction means 28. In this figure, when activated, the silhouette area extraction means 28 first generates a vertical histogram of a silhouette image (S1001). FIG. 29 is an explanatory diagram of silhouette region extraction processing. In the silhouette image of FIG. 29 (a), the white part at the center is a non-background area, and a plurality of white circles at the upper left indicate non-background noise that appears in the background area. The vertical histogram of this silhouette image is as shown in FIG. The white portion in this figure means the number of non-background pixels in the corresponding vertical (column) direction in FIG.

  Then, as a non-background extraction process, a vertical non-background pixel histogram value is divided into an area where the value of the histogram is 0 and an area where the histogram is not, and an area having a width equal to or greater than a predetermined value among the non-zero areas The area A in FIG. 29B is selected, and this area is divided into silhouette areas (S1002). FIG. 29C is a diagram in which both sides of the silhouette image of FIG. 29A are divided (cut) in the vertical direction by this processing. In this figure, the area where the histogram value is not 0 but is equal to or smaller than a predetermined value (area B in FIG. 29B) is excluded as a non-background noise area, and the area where the histogram value is 0 ( The area C in FIG. 29B is excluded as a background area.

  Next, a horizontal histogram is generated for the silhouette image of FIG. 29C (S1003). FIG. 29D shows a horizontal histogram. Based on this histogram, a non-background area is extracted and divided as a silhouette area in the same manner as described above (S1004). The processes from step S1001 to step S1004 are repeated a predetermined number of times, and the final silhouette region is extracted while sequentially dividing the silhouette region. Instead of repeating a predetermined number of times, if no new non-background noise region is found, the extraction process may be terminated. FIG. 29E shows the extracted silhouette region. Then, an area other than the silhouette area is filled with a background pixel value as a background area (S1005).

[Preprocessing] (Steps S904 and S905)
The pre-processing unit 29 performs thin edge removal and edge noise backgrounding processing on the region extracted by the silhouette region extraction unit 28 in the silhouette image.

1. Thin edge noise (background noise) removal process (step S904)
FIG. 30A shows an example of thin edge noise that occurs inside the operator silhouette when there is a window frame such as a sash in the background and the operator's hair overlaps. The A part in FIG. 30A is the edge noise.

In the following, a procedure for filling such fine noise by edge extraction and synthesis of silhouettes (outlined portions in FIG. 30A) will be described.
First, an outline extraction filter is applied to the silhouette image to extract silhouette edges (S1101). For the edge extraction, for example, a weighted Laplacian filter shown in FIG. 33 is used. FIG. 30B shows an image extracted by this process.

Next, the edge extracted image (FIG. 30B) and the original silhouette image (FIG. 30A) are combined (S1102).
Since both the silhouette part and the edge part have a value of 255, when they are overlapped, the silhouette edge is expanded by one pixel.
Since the thin edge noise is usually only about 1 or 2 pixels wide, by applying the above processing, it can be filled as non-background as shown in FIG.

2. Edge Noise Background Noise Processing FIG. 31A shows an example of edge noise that has become a large hole. This edge noise is a noise that is likely to be generated when the background is a single color region having a wide background and the color of the clothes is similar to that of the operator. This noise can be removed by the background noise removal algorithm described in the first embodiment by closing the noise mouth (B portion in the figure) to make background noise.

Hereinafter, the procedure for closing the mouth portion of the noise will be described with reference to FIG.
First, an edge image (FIG. 31B) is generated by applying a contour extraction filter to a silhouette image (FIG. 31A) (S1201), and an image (FIG. 31) that combines the edge image and the original silhouette image. 31 (c)) is generated (S1202). Further, the processing from step S1201 to step S1202 is repeated a plurality of times for this synthesized silhouette image. (FIGS. 31D and 31E). Since the mouth is a small noise, if it is repeated several times, the edge swells and the mouth closes.

The edge noise can be removed by performing background noise removal processing on the silhouette image when the mouth is closed. (Fig. 31 (f))
Note that since the size of the mouth of the noise varies depending on the situation, it is preferable that the operator can specify how many times the edge extension is repeated.

  For example, by using the threshold setting unit 23 described in the second embodiment, a threshold setting screen corresponding to the screen illustrated in FIG. 12 is displayed for each repetition count, and the repetition count set by the operator is set and registered. By doing so, it is possible to perform image processing using an appropriate number of repetitions.

  By the above pre-processing, a thin edge is removed, and for edge noise that has become a large hole, a noise image is closed to create a silhouette image with background noise.

[Background Noise Removal Processing] (Step S906)
Next, background noise is removed from the silhouette image created by the preprocessing means 29 by the background noise removing means 24. Since this process is the same as in the first embodiment, a description thereof will be omitted. The background noise of the silhouette image is removed by the background noise removal process.

[Post-processing] (Step S907)
The silhouette image created by the above processing is expanded with respect to the original silhouette image although noise is removed. For this reason, the post-processing means 30 is activated and this expanded silhouette image is brought close to the original silhouette image by the procedure shown in FIG.

  First, an outline extraction filter is applied to extract the inner edge of the silhouette image (S1301). At this time, a Laplacian filter illustrated in FIG. 34 may be applied. Next, the pixel value of the edge image created by applying this filter is subtracted from the pixel value of the original silhouette image (extended image) (S1302). If the pixel value is smaller than 0, it is set to 0 (S1303, S1304). By this processing, the edge portion is reduced by one pixel. The processes in steps S1301 to S1304 are repeated as many times as the number of applied expansion processes.

[Noise mitigation processing, image output processing] (steps S908 and S909)
Moreover, the remaining non-background noise and edge noise are reduced by the noise reduction means 25 (S908), and the composite image is output by the image output means 26 (S909). Since the processing procedures of these means 25 and 26 are the same as those in the first embodiment, description thereof will be omitted.

  The operation of the image processing apparatus according to the present embodiment has been described above with a focus on differences from the first embodiment. However, the pre-processing unit 29, the post-processing unit 30 and the like of the present embodiment are added to the second embodiment. It is also possible to provide it.

  In the present embodiment, the filtering (noise removal, mitigation) process is applied only to the region that cannot be determined as noise by the silhouette region extraction means, so that the filtering speed is improved.

  Also, thin edge noise is removed in advance by the pre-processing means, and edge noise that has a narrow mouth portion and a large spread in the non-background area is closed to become background noise, and subsequent background noise removing means In order to remove the noise, it is possible to obtain a composite image with better quality than the noise mitigation effect of the first embodiment.

  The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention. For example, when both the background noise removing means and the noise mitigating means are executed, it is necessary to execute them in the above order, but either the background noise removing means or the noise mitigating means is executed depending on the CPU performance or the like. Only has a noise reduction effect. Further, the post-processing means 30 can obtain a certain effect even if it is activated at any timing after the background noise removal processing. For example, you may make it start after a noise mitigation process. If the number of loops of the edge enlargement process by the pre-processing means is small, the post-processing means can be omitted.

  The image processing apparatus and the image processing method according to the present invention can be used for all purposes such as extracting a specific image such as a person and replacing other background portions with other images. In addition to the game machine described in the above embodiment, for example, in a videophone, only the caller is extracted and the background is replaced with another image, or only a specific subject is extracted in camera shooting and other backgrounds are extracted. It becomes possible to replace with. The image processing apparatus is intended to include game machines, videophones, cameras, and the like having such functions.

1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. It is a functional block diagram of the process part of FIG. It is an example of a background image input by the image input means of FIG. It is an example of an extraction object image containing the extraction object (person) image | photographed against the background of FIG. It is explanatory drawing of the silhouette image produced with the image separation means of FIG. FIG. 6A is an explanatory diagram of background noise, FIG. 6B is an explanatory diagram of non-background noise, and FIG. 6C is an explanatory diagram of edge noise. FIG. It is explanatory drawing of the process sequence of the background noise removal means of FIG. It is explanatory drawing of the process sequence of the noise mitigation means of FIG. It is explanatory drawing of the image composition by the image output means of FIG. It is a functional block diagram of the image processing apparatus process part by the 2nd Embodiment of this invention. It is explanatory drawing of how to limit a player's standing position in the preliminary preparation stage by the 2nd Embodiment of this invention. It is explanatory drawing of the threshold value setting screen by the 2nd Embodiment of this invention. FIG. 12A shows an example in which a lot of noise is generated when the threshold is low. FIG. 12B is an example in which the threshold is appropriate and almost no noise is generated. FIG. 12C is an example in which the extraction target is bitten by the background due to edge noise when the threshold is high. It is explanatory drawing of the other threshold value setting screen by the 2nd Embodiment of this invention. 13A is a diagram showing a low threshold value, FIG. 13B is a diagram showing a threshold value, and FIG. 13C is a diagram showing a high threshold value. It is a flowchart which shows the procedure of the image processing of the game player by the 2nd Embodiment of this invention. It is an example of the data which link | relates the player image and game background frame by the 2nd Embodiment of this invention. It is a flowchart which shows the process sequence of the image output means of FIG. It is a flowchart which shows the process sequence of the background noise removal means of FIG. 2 and FIG. It is a flowchart of the background noise candidate area | region mark processing routine of FIG. It is a flowchart of the candidate area | region upper confirmation process routine of FIG. It is a flowchart of the candidate area | region downward confirmation process routine of FIG. It is a flowchart of the candidate area | region conversion process routine of FIG. It is a flowchart which shows the process sequence of the noise mitigation means of FIG. 2 and FIG. It is a functional block diagram of the image processing apparatus process part by the 3rd Embodiment of this invention. It is a flowchart of the process sequence performed by each means of the process part of FIG. It is a flowchart which shows the process sequence of the silhouette area | region extraction means of FIG. It is a flowchart which shows the procedure of the thin edge noise removal process performed by the pre-processing means of FIG. It is a flowchart which shows the procedure of the edge noise background noise conversion process performed by the pre-processing means of FIG. 24 is a flowchart showing a procedure of extended edge reduction processing executed by the preprocessing means of FIG. It is explanatory drawing of the silhouette area | region extraction process of FIG. It is explanatory drawing of the removal process of the thin edge noise of FIG. It is explanatory drawing of the edge noise background noise conversion process of FIG. FIG. 29 is an explanatory diagram of the extended edge reduction processing of FIG. 28. FIG. 28 is an example of a weighted Laplacian filter used in the thin edge noise removal process of FIG. 26 or the edge noise background noise conversion process of FIG. 27; FIG. 29 is an example of a weighted Laplacian filter used in the extended edge reduction process of FIG. 28.

Explanation of symbols

1 Image processing device (game device)
DESCRIPTION OF SYMBOLS 11 A / D conversion means 12 Processing part 13 Memory 14 Display memory 15 D / A conversion means 16 External storage device 17 Input part 21 Image input means 22 Image separation means 23 Threshold adjustment means 24 Background noise removal means 25 Noise reduction means 26 Image output means 27 Game execution means 28 Silhouette region extraction means 29 Pre-processing means 30 Post-processing means 51 Camera 52 Television monitor

Claims (8)

An image input means for inputting a background image and an extraction target image including the extraction target in the same background as the background image;
Image separation means for creating a silhouette image having a non-background pixel and a background pixel based on a difference between pixel values of the background image and the extraction target image;
When the silhouette image is input, an area surrounded by non-background pixels is marked for each line of the input image, and when the mark area touches a background pixel in an adjacent row or column, the mark area is set as a background area. A background noise removing means that sequentially changes, converts the remaining mark area into a non-background area and edits the silhouette image,
Based on the digitized non-background degree, the non-background degree averaging process of the neighboring area of the pixel is sequentially performed for each pixel from the edge of the image area of the silhouette image edited by the background noise removing unit, and then the image Noise mitigation means for correcting the area range;
An image output means for creating an α composition image based on the averaged non-background degree and the extraction target image, and outputting the image together with a separately input background image;
An image processing apparatus comprising: An edge extracted image of the silhouette image created by the image separating means is created, and the edge extracted image and the silhouette image are combined to expand the non-background region of the silhouette image, thereby converting the edge noise into the background noise. Pre-processing means to perform,
The image processing apparatus according to claim 1, wherein the background noise removing unit uses the silhouette image created by the preprocessing unit as an input image.   A silhouette region extraction that takes a vertical or horizontal histogram of a silhouette image created by the image separating means and determines a range to be edited or corrected by the background noise removing means or the noise mitigating means based on the width of the histogram The image processing apparatus according to claim 1, further comprising: means. The image separating means determines whether the pixel is a non-background pixel or a background pixel by comparing a difference between pixel values of the background image and the extraction target image with a threshold value; and
The image forming apparatus includes: a threshold adjustment unit configured to display an image that can be determined based on the threshold value and that can determine a noise degree, and to change the threshold value based on a change request for the threshold value input from an input unit. The image processing apparatus according to any one of 1 to 3. Means for displaying range designation information indicating an extraction target area on the display unit;
The image processing apparatus according to claim 1, wherein the image input unit inputs an extraction target image in the range based on the range designation information. An image processing method for extracting a specific image from an input image and combining it with another image,
Inputting a background image and an extraction target image including an extraction target in the same background as the background image;
Creating a silhouette image having a non-background pixel and a background pixel based on a difference between pixel values of the background image and the extraction target image; and marking an area surrounded by non-background pixels for each line of the silhouette image; If this mark area touches a background pixel in an adjacent row or column, the mark area is sequentially changed to a background area, the remaining mark area is converted to a non-background area, and the silhouette image is edited. A step of sequentially averaging the non-background degree of the neighboring area of the pixel for each pixel from the edge of the image area of the silhouette image based on the converted non-background degree, and then correcting the image area range;
Generating an α synthesis image based on the averaged non-background degree and the extraction target image, and outputting the image together with a separately input background image;
An image processing method comprising: An image processing method for extracting a specific image from an input image and combining it with another image,
Inputting a background image and an extraction target image including an extraction target in the same background as the background image;
Creating a silhouette image having a non-background pixel and a background pixel based on a difference between pixel values of the background image and the extraction target image;
Converting edge noise into background noise by enlarging the non-background region of the silhouette image;
If this silhouette image line is marked with an area surrounded by non-background pixels and this mark area touches a background pixel in an adjacent row or column, the mark area is sequentially changed to a background area, and the remaining area remains. After editing the silhouette image by converting the marked area to a non-background region, based on the digitized non-background degree, the non-background degree of the neighboring area of the pixel for each pixel from the edge of the image area of the silhouette image Sequentially performing the process of averaging, and then correcting the image area range;
Generating an α synthesis image based on the averaged non-background degree and the extraction target image, and outputting the image together with a separately input background image;
An image processing method comprising: A computer game program for displaying a game player image captured by a camera in a game background and playing the game,
Input and save a background image,
A process of inputting an extraction target image including a player with the same background as the background image;
An image separation process for creating a silhouette image having a non-background pixel and a background pixel based on a difference between pixel values of the stored background image and the extraction target image;
If this silhouette image line is marked with an area surrounded by non-background pixels and this mark area touches a background pixel in an adjacent row or column, the mark area is sequentially changed to a background area, and the remaining area remains. Background noise removal processing to edit the silhouette image by converting the marked area into a non-background area,
Based on the digitized non-background degree, a process of sequentially averaging the non-background degree of the neighboring area of the pixel for each pixel from the edge of the image area of the silhouette image edited by the background noise removing unit is performed. Noise reduction processing to correct the image area range,
An α composition image is created based on the averaged non-background level and the extraction target image, and is generated by a process of executing a game, or output together with a background image input from an external storage device Image output processing to
A program that causes a computer to execute.