WO2006033178A1

WO2006033178A1 - Image processing device, method, and program

Info

Publication number: WO2006033178A1
Application number: PCT/JP2005/004469
Authority: WO
Inventors: Fumikazu Matsubara; Ken-Ichi Hoshino
Original assignee: Polygon Magic, Inc.
Priority date: 2004-09-22
Filing date: 2005-03-14
Publication date: 2006-03-30
Also published as: JP4399462B2; JPWO2006033178A1

Abstract

[PROBLEMS] To automatically detect image noise when the background area of an image is erased and the resultant image is combined with another image and to remove or mitigate the image noise. [MEANS FOR SOLVING PROBLEMS] A silhouette image having non-background pixels and background pixels is created according to the differences between the pixel values of the background image and the pixel values of an image to be extracted. The area surrounded by the non-background pixels is marked for each line of the silhouetted image. If the areas are in contact with a background pixel of the adjacent line or column, the areas are changed to the background areas sequentially, and the remaining mark areas are changed to non-background areas. Thereafter, from the end of the image area of the silhouette image, the degrees of non-background near the pixel are averaged, and the range of the image area is corrected. An image for α-combining is created according to the averaged degree of non-background and the image to be extracted. The created image for α-combining is outputted together with a separately inputted background image.

Description

Image processing apparatus, method, and program

Technical field

[0001] The present invention relates to a technique for extracting a specific image from a photographed image and combining it with another image. In particular, the present invention requires real-time performance such as a game by reducing noise in the composite image. The present invention relates to an image processing device, a method, and a program that can be used even under conditions. Background art

Conventionally, there is a so-called blue background technique for extracting a specific image such as a person from a photographed image and combining it with an arbitrary background image. This means that the background of the image to be photographed is blue, the portion corresponding to this blue is regarded as the background by the color recognition means, and this portion is replaced with another background image. Since it is only necessary to recognize this specific color and replace it with the background, when extracting power images that are possible even with colors other than blue, if the color to be replaced with the background is blue, the color of the person's skin and the complementary color It is possible to reduce the possibility that noise will get on, that is, close to the relationship and a part of the face becomes the background.

[0003] This blue-back technique has a problem that, for example, if the person to be extracted wears clothes having the same color pattern as that of the background, the pattern is replaced by the background.

[0004] On the other hand, there is a technique for performing noise processing by taking an image of a background only and an image of an extraction target such as a person together with the background, extracting only the extraction target from the difference between the two images. Proposed. (See Patent Document 1). This technology creates a mask image (silhouette image) from the difference between pixel information (eg, RGB data) of both images, and deletes noise by editing.

Patent Document 1: JP 2000-36032 A

Disclosure of the invention

Problems to be solved by the invention

By the way, the above-described conventional technique automatically detects and removes noise by a computer. For example, as a video game, only the image of a person who is a game player who always shoots with a camera on a background image recorded in advance according to the game scenario. If you try to extract and combine it with the background image of the game to display it, the CPU will be overloaded and the real-time performance will be impaired. Also, a device that performs image processing by a computer such as a game device has a memory capacity limitation.

[0006] The present invention has been made in view of the above-mentioned troublesome circumstances. When a background portion is erased from a photographed image and synthesized with another image, the computer is automatically loaded without imposing a load on the computer. An object of the present invention is to provide an image processing apparatus, method, and program capable of detecting and removing or mitigating image noise. Means for solving the problem

In order to achieve the above object, an image processing apparatus according to the present invention includes an image input means for inputting a background image and an extraction target image including the extraction target in the same background as the background image, and a background image. Based on the difference between the pixel values of the image to be extracted, an image separation means for creating a silhouette image having a non-background pixel and a background pixel and the silhouette image are input, and a non-background pixel is input for each line of the input image. If an area surrounded by is marked and this mark area touches a background pixel in an adjacent row or column, the mark area is sequentially changed to a background area, and the remaining mark area is changed to a non-background area. The background noise removing means for editing the silhouette image by converting the image to the edge of the image area of the silhouette image edited by the background noise removing means based on the digitized non-background degree. For each pixel, the non-background level averaging process of the neighboring area of the pixel is sequentially performed. After that, the noise mitigation means for correcting the image area range, the non-background level that has been averaged, and the image to be extracted are also included. And an image output means for generating an α composition image and outputting it together with a separately input background image.

[0008] The basic idea of the present invention is as follows.

Noise that occurs when the difference between the background image and the extraction target image is taken as background noise that considers a part of the extraction target as the background, non-background noise that considers a part of the background as the extraction target, and extraction The boundary between the target and the background is eaten by other areas! is there.

Among these, for background noise, the closed background area in the non-background area is regarded as noise, and the closed background area is scanned and converted to non-background to remove the noise. On the other hand, non-background noise and edge noise are made inconspicuous by applying a semi-transparent gradation at the edge considering the tendency of these to be small and thin areas.

[0010] According to the present invention, background noise can be removed, non-background noise and edge noise can be reduced, and image processing quality can be improved without imposing a load on the CPU. be able to. In addition, since gradation is applied to the boundary between the background image and the non-background image, a natural composite image can be created by adapting the non-background image to the background image.

[0011] In addition, according to the present invention, it is not necessary to save a copy of the silhouette image, and it is possible to perform noise reduction and noise reduction processing while directly editing the image. Hardware resources can be saved.

[0012] The image processing apparatus according to the present invention further creates an image (edge extracted image) obtained by extracting the edges of the shot image created by the image separating means, and synthesizes the edge extracted image and the silhouette image. Pre-processing means for creating a silhouette image in which the non-background region is enlarged, and the background noise removing means performs background noise removal processing using the silhouette image created by the pre-processing means as an input image. Characterized by

By creating a silhouette image in which the non-background area is enlarged, the border between the non-background area of the edge noise and the background area is closed and converted into background noise, and this background noise is removed by the background noise removal means. Remove noise. As a result, the edge noise can be removed, thereby improving the image quality.

[0013] It is preferable to enlarge the non-background area, convert edge noise into background noise, perform the background noise removal process, and then reduce the non-background area toward the original size.

[0014] Further, in the image processing apparatus according to the present invention, a histogram in the vertical direction or the horizontal direction of the silhouette image created by the image separation means is taken, and the width of the histogram is taken. On the basis of this, there is provided a silhouette region extracting means for determining a range to be edited or corrected by the background noise removing means or the noise mitigating means.

By limiting the silhouette region, it is possible to reduce the load on the computer required for filtering processing such as noise removal and mitigation.

In the image processing apparatus according to the present invention, preferably, the image separation means is a non-background pixel by comparing a difference between pixel values of the background image and the extraction target image with a threshold value, or is a background pixel. Threshold adjustment means for determining whether or not the image is present, displaying an image that can be determined based on the threshold and capable of determining the degree of noise, and changing the threshold based on a threshold change request input from the input unit. Prepare! /.

Accordingly, noise processing can be performed in a state where noise is suppressed to some extent, and quality after synthesis can be predicted.

More preferably, the extraction target range is specified in a state where the noise level is displayed. In this way, the outside of the range is unconditionally set as the background and only noise processing within the range needs to be performed, so that the burden on the computer is reduced and the speed of image processing can be increased. In addition, by enabling the operator (user) to perform noise processing of background noise, non-background noise, or edge noise, the load on the computer is reduced and the quality meets the operator's requirements. Image processing becomes possible.

[0019] It should be noted that instead of the operator inputting the extraction target range, range designation information (for example, a frame line) indicating the extraction target region is displayed in advance on the display unit, and the image input means Based on the range designation information, an extraction target image in the range may be input and processed.

[0020] Here, "image processing device" means a device having an image processing function. In addition to a general-purpose computer device, a game device having the function, a video phone, an imaging device such as a camera, and the like It is also intended to include.

The “non-background degree” is set for each pixel or pixel group of the extraction target image and indicates the degree to which the pixel or pixel group is non-background (extraction target). When the non-background level is expressed in 8 bits, it takes a value between 0 and 255. In addition, it is sufficient if it has a significance of non-background level. Therefore, the background level is also included. [0022] An image processing method according to the present invention is an image processing method for extracting a specific image from an input image image and synthesizing it with another image. The image processing method has the same background as this background image. A step of inputting an extraction target image including an extraction target, a step of creating a silhouette image having non-background pixels and a background pixel based on a difference between pixel values of the background image and the extraction target image, and If you mark an area surrounded by non-background pixels for each line, and that area touches a background pixel in an adjacent row or column, change to the background area sequentially, and then leave the remaining mark area as non-background After editing the silhouette image after converting it to an area, based on the digitized non-background degree, the edge power of the image area of the silhouette image Non-background degree of the neighboring area for each pixel Performs averaging processing in sequence, then corrects the image area range, generates an image for alpha synthesis based on the averaged non-background level and the extraction target image, and outputs it together with the separately input background image And a step of performing.

[0023] According to the present invention, a series of processing powers for noise removal and relaxation can be performed while directly editing silhouette images, thus saving memory.

[0024] A program according to the present invention is a computer game program for displaying a game player image taken by a camera in the background of a game and playing a game.

, Processing to input and save the background image, processing to input the extraction target image including the player with the same background as the background image, and non-existence based on the difference between the pixel value of the background image to be extracted and the extraction target image Create a silhouette image that has a background pixel and a background pixel Image separation processing, and when you mark an area surrounded by non-background pixels for each line of this silhouette image, and that area touches a background pixel in an adjacent row or column The background noise removal process based on the non-background degree that is converted into a background area, converts the remaining mark area into a non-background area, and edits the silhouette image. For each pixel from the edge of the image area of the silhouette image edited by means, the process of averaging the non-background degree of the neighboring area of the pixel is sequentially performed in line 1 行, It is created by the noise reduction processing that corrects the subsequent image area range and the α composition image based on the non-background level and the extraction target image that have been averaged, and the game progresses, or External storage device power Image output processing that is output together with the input background image Data.

[0025] A series of noise processing can be performed while directly editing the silhouette image, and the memory of the game apparatus can be saved. In addition, the noise reduction processing averages the non-background level of the neighboring area of the pixel, increasing the hit rate and improving the processing speed.

[0026] Preferably, the α composition image creation processing and the game execution processing are independently operable. Since the α composition image stored when the game player needs to be displayed is used, the responsiveness during the game can be improved.

The invention's effect

[0027] According to the present invention, it is possible to automatically detect image noise, remove background noise, and mitigate non-background noise and edge noise. In addition, a series of noise processing can be performed while directly editing the silhouette image, so that the memory area can be saved.

[0028] In addition, in a game program, when a foreshadow image (non-background image) such as a game player photographed with a camera is combined with a background image of a game, a gradation is applied to the boundary portion. It can blend into the game background and provide a more realistic game.

BEST MODE FOR CARRYING OUT THE INVENTION

[0029] Hereinafter, embodiments of the present invention will be described.

First Embodiment>

FIG. 1 is a block diagram of an image processing apparatus 1 according to the first embodiment of the present invention. Here, the image processing apparatus 1 includes an AZD conversion unit 11 that inputs a video signal photographed by the camera 51 and converts it into a digital signal, a processing unit 12 that executes image processing, a memory 13 that stores image data, a television monitor Display memory 14 for storing display data to be sent to 52, DZA conversion means 15 for converting digital signals to TV monitor signals, and drive devices for storage media such as CD-ROM and DV D-ROM (external storage devices) ) 16 and an input unit 17 such as a mouse or keyboard.

As shown in FIG. 2, the processing unit 12 includes an image input means (process) 21, an image separation means (process) 22, a threshold adjustment means (process) 23, a background noise removal means (process) 24, and a noise mitigation means. (Process) 2 5 and image output means (processing) 26. Further, it is assumed that a background image is stored in the memory 13 in advance.

[0031] Next, the operation of each means will be described. In the following description, the image data is appropriately converted to another unit system such as a YCrCb system if necessary according to the specifications of the power camera 51 and the television monitor 52 described in the RGB system for convenience.

[0032] [Image input processing]

First, in response to a request from the input unit 17, the image input unit 21 of the processing unit 12 is activated, and the image data captured by the camera 51 is input and stored in the memory 13. At this time, an image including only the background (background image) not including the extraction target such as a person and an image including the extraction target (extraction target image) are respectively photographed and stored in the memory 13. Fig. 3 shows an example of a background image, and Fig. 4 shows an example of an extraction target image. In FIG. 4, a person to be extracted (for example, a game player) is photographed with the same background as in FIG.

[0033] [Image separation processing]

Next, a difference for each pixel of the background image and the extraction target image is obtained by the image separation means 22. In other words, the sum of the differences between the RGB of the background image and the image to be extracted for each RGB of 8 bits (24 bits in total) is calculated. When the total value is less than or equal to a predetermined threshold value, the value (non-background level) corresponding to the pixel is represented by 8-bit data A, and A = 0. On the other hand, when the total value is larger than the predetermined threshold, the pixel is set as an extraction target (non-background), and A = 255.

In this way, a binarized silhouette image is created based on the background image and the extraction target image.

The method of comparison with the threshold value is not limited to the above, and the hue, saturation, and brightness values are binarized based on the difference between the two images, for example, the average value of each RGB value difference is used as the threshold value. Just do it.

FIG. 5 is an explanatory diagram of a silhouette image. Here, the blacked out part is the background (A = 0), and the white part is the extraction target (A = 255).

However, since an actual silhouette image has noise as shown in FIG. 6, this noise processing is necessary. The types of noise include background noise that considers a part of the extraction target shown in Fig. 6 (a) as the background, and a part of the background shown in Fig. 6 (b) as the extraction target. Non-background noise, the boundary between the extraction target and the background shown in Fig. 6 (c) eats into other areas! , There will be edge noise.

[0037] Background noise appears when the operator's clothes and the background color are close. Non-background noise appears as background pixel errors due to flickering of fluorescent lights, etc., and pops up anywhere in the background. Edge noise is a combination of background noise and non-background noise, and the edges become jagged.

[0038] These noises can be reduced to some extent by adjusting the threshold value using the threshold adjusting means 23, but they cannot be completely removed when the background and the color to be extracted are approximated. There is. In addition, there is a certain limit such as increasing non-background noise when trying to reduce background noise. Therefore, it is necessary to remove or mitigate these remaining noises.

In the following, these noise processing procedures will be described with reference to FIGS.

[Background noise removal]

The background noise removal process includes the processes in steps S301 to 304 in FIG.

[0040] First, for background noise candidate area mark processing! For all horizontal lines in the silhouette image, background pixels (A = 255) sandwiched between non-background pixels (A = 255) = 0) (S401, S402). For example, gray (for example, A = 128) is assigned as the mark. FIG. 7 is an explanatory diagram of the background noise removal process. In this figure, pixels filled in black (unit square in the figure) are background pixels, and white pixels are non-background pixels. The processing after steps S401 and S402 is performed for each line and this is performed for all lines is shown in FIG. 7 (b). Here, gray pixels are marked pixels.

[0041] Then, in the candidate region upper confirmation process !, all the lines marked from top to bottom are scanned, and at least one pixel among the marked line segments is scanned. If the pixel above is the background (A = 0), the entire mark portion in contact with the pixel is painted with the background (S501, S502). In other words, the A value of the marked pixel is changed from 12 8 to 0. Fig. 7 (c) shows the state where processing has been completed to the middle line, Fig. 7 (d) Indicates the state after all the lines have been processed.

[0042] When the above process is completed for the bottom line, next, in the candidate area lower confirmation process, scan the line segment in which all lines are marked in the reverse direction, that is, the lower force is also marked upward. To do. Then, if at least one pixel of the marked line segment is the background, the entire marked line segment is replaced with the background (S601, S602). Figures 7 (e) and (f) show the state where the line has been completed, and Figure 7 (g) shows the state after all the lines have been processed.

[0043] At the stage where all of the above are completed, the pixels remaining as marks are background noise, so they are repainted as non-background (S701, S702). Figure 7 (h) shows the state of the silhouette image after the background noise removal process.

Background noise is removed by the above processing.

[0044] [Non-background noise / edge noise mitigation]

Next, the procedure for mitigating non-background noise and edge noise will be described using FIG. 8 and FIG.

[0045] For all the pixels on the image, a 4x4 pixel area is scanned in the lower right direction and replaced with the average of the A values in the area (S801, S802). For example, the pixel in the O part of Fig. 8 (b) is A = 255 * 3/16 = 48 when taking the power average of the background pixels (A = 0). Here, the symbol “*” means multiplication, and “/” means division.

In addition, when the power average of the pixel in FIG. 8C is a non-background pixel (A = 255), A = 255 * 3/16 = 48 as in FIG. 8B.

[0047] This averaging process is first executed for the pixels from the left to the right of the top line, and when it is completed, the lower lines are executed in order. This process is repeated until the rightmost pixel in the bottom line. Figure 8 (d) shows the state after the completion of a series of averaging processes.

[0048] When the accuracy of the boundary portion of the calculation area becomes a problem, an image may be acquired in advance so as to sufficiently include the calculation area. If the accuracy of the border does not matter much, after averaging from the bottom line (bottom line) to the top four lines, the line below is the same as the pixel value above it. You may make it fill. [0049] As a result of the above averaging process, the silhouette image is shifted in the upper left direction. For example, the entire silhouette image is shifted by 2 pixels in the lower right direction, that is, half of the average calculation unit area 4 X 4. Thus, the filter can be set to an appropriate position (S803). (State of Fig. 8 (e))

[0050] It should be noted that the movement of this position need only be shifted to the upper left by 2 X 2 in the area where the shot image is stored without having to actually rewrite the pixel value in the memory.

[0051] In the above description, for the sake of convenience, the terms horizontal and vertical are used. However, when the image data is represented by the coordinates of the X axis and the Y axis, the above processing may be performed with either the vertical or horizontal direction. It is easier to create a processing program if the address where information is stored is in the horizontal direction.

[0052] [Image output processing]

Using the above-mentioned series of noise-processed silhouette images as alpha channel data, the image data for a synthesis (alpha blending) is created using the extraction target image data in FIG.

Here, the image data for α synthesis is configured as an array of 8 bits (32 bits in total) of R: G: B: Α for each pixel.

[0054] Then, after the background data for synthesis read in the CD-ROM force of the external storage device 16 is written in the display memory 14, and the image data for alpha synthesis is similarly written in the display memory 14. By issuing a display output command, both image data are synthesized and displayed on the television monitor 52. FIG. 9 is an explanatory diagram of this. Image data for composition is created from image data to be extracted based on RGB values and silhouette image data based on A values (Figure 9 (a)), and this is combined with background data based on RGB values (Figure 9 (b)). . Specifically, after the background data in Fig. 9 (b) is drawn, the image data for oc composition in Fig. 9 (a) is overwritten. As a result, a composite image is displayed (FIG. 9 (c)). At this time, the extraction target image displayed on the background is completely transparent (A = 0), completely opaque (A = 255), intermediate state (0 <A <255) depending on the A value corresponding to the pixel. Become.

[0055] As a result of the above noise processing, background noise is removed, non-background noise 'edge noise is reduced, and the A value is intermediate for the boundary between the background and the target image. Since the value and gradation are high, the target image becomes familiar with the background and becomes a natural composite image.

[0056] It should be noted that each means of the processing unit 12 may be activated by a request from the input unit each time, or may be automatically activated by completion of the preprocessing.

[0057] According to the present embodiment, it is possible to automatically perform noise processing without applying a load on the CPU to synthesize an image.

[0058] In particular, for background noise removal processing, first, an area surrounded by non-background pixels is marked for each line, and when the area touches a background pixel in an adjacent row or column, the area is sequentially changed to the background area. Since the remaining mark area is removed as background noise, it is possible to perform processing while editing the silhouette image itself, and it is not necessary to secure temporary memory for processing, thus saving memory. Can do. Also, since the memory area to be scanned and the memory area to be written are close, the cache hit rate can be increased and the speed can be improved.

[0059] In addition, since noise reduction processing is performed, averaging processing is performed sequentially from the end of the region, and finally the region range is corrected, so the output image of the background noise removal processing is edited as it is. Therefore, there is no need to secure a temporary memory for processing. Similarly to the background noise removal process, the memory area to be scanned and the memory area to be written are close, so the cache hit rate can be increased and the speed can be improved.

[0060] <Second Embodiment>

Next, a second embodiment will be described.

The configuration is the same as in FIGS. 1 and 2, but the image processing device 1 in FIG. 1 functions as a game device. As the input unit 17, a game controller is used.

[0061] A storage medium such as a CD-ROM or DVD-ROM stores image data to be displayed as a background of a game program and a game video. Further, information on the insertion position of the image of the game player (extraction target) is stored in association with each frame of the game video.

[0062] The storage medium includes image input means (process) 21, image separation means (process) 22, threshold adjustment means (process) 23, background noise removal means (process) 24, noise mitigation means (process) 25, Image processing In addition to means (process) 26, a program for causing a computer to execute game execution means (process) 27 for executing a game is stored.

When the storage medium is inserted into the external storage device 16 and activated, the required program power stored in the CD-ROM is loaded into the program area of the memory 13 and activated by the processing unit 12.

[0064] [Preparation stage]

First, as a pre-preparation before starting the game, a background image and an extraction target image (player image) are input, and a silhouette image is created by taking the difference between the two images by the image separation means.

The background image is stored in the memory 13 and used for image separation during the subsequent game execution. Then adjust the threshold of the A value (0 or 255) of the silhouette image.

[0065] When inputting a player image, in order to limit the player's (operator) standing and position, a silhouette is displayed on the image including the player as shown in FIG. It is preferable to guide the position of In FIG. 11, a guidance instruction “Please stand and play in silhouette” is displayed on the screen. By limiting the position of the player in the image, it is possible to limit the image analysis range and reduce the CPU processing load.

[0066] [Threshold adjustment processing]

Although the threshold for background / non-background separation can be set to a certain degree of experience, it is not possible to limit the environment for generating an image including a player, so it is preferable to have a function that the operator can adjust. A rough guideline for the threshold is a value of about 40 in the range of 0-255.

[0067] As one example of the user screen for threshold adjustment, there is a method that allows fine adjustment by displaying a slider as illustrated in FIG. The player changes the threshold by moving the slider on the screen to the left and right, and directly checks the composite image of the change result on the screen. In Fig. 12 (a), the threshold value is low, that is, the difference between the pixel values of the background image and the extraction target image. . Figure 12 (b) shows an example where the threshold is appropriate and almost no noise is generated. Fig. 12 ( In the case of c), when the threshold value is high, it is not regarded as non-background unless the difference in pixel values is large. Therefore, the extraction target is bitten into the background by edge noise.

[0068] The other is a method in which a plurality of representative threshold values are sampled as shown in FIG. 13, each setting button is prepared, and the player roughly sets it by selecting the button. . For example, each of the three buttons is low (about 20), medium (about 40) and high (about 60)

) Is assigned, and a composite image corresponding to each setting button is displayed, and an appropriate button is selected. Thereby, the threshold value of the selected button is set.

[0069] When the advance preparation is completed, the game is started.

As image processing during the game, player image extraction processing and image output processing that is combined with the game background image and output are performed independently.

[0070] [Player Image Extraction Processing]

The player's images are periodically fetched, and image input processing, image separation processing, background noise removal processing, and noise reduction processing are executed (S105 to S108).

In the image input process, a player image is input through the camera 51, and the processes after the image separation process are executed using this image and the background image stored in the memory 13 by the pre-processing.

Since these processing procedures are the same as those in the first embodiment, a description thereof will be omitted. As the output of the noise reduction processing, image data for α synthesis is created and stored in the memory 13.

[0072] [Image output processing]

Next, image output processing during game execution will be described. As shown in FIG. 15, the data to which the game module (program for executing the game) power is also passed for image output is a frame image, output position information, and deformation information. Here, the output position information is information for designating the output position of the player image, and is represented by coordinates (address range) on the screen. Further, the deformation information includes not only the case where the image is deformed vertically or horizontally, but also a part of the image that is cut out and enlarged, reduced, rotated, or changed in color as a clip image. In addition to this information, a flag indicating whether or not the player image is output may be provided.

[0073] In the image output process to the television monitor 52, first, the image of the background frame of the game Data is output (S201). Next, it is determined whether or not the output of the player image is necessary based on the flag or the presence / absence of the output position information of the player image. If not necessary (“N” in S202), a display output command is sent (S206). Then, the process proceeds to the next frame output process.

On the other hand, when the player image needs to be output (“Y” in S202), output position information and deformation information are acquired (S203). Then, the image data for α composition stored in the memory 13 is acquired and edited based on the deformation information (S204). Thereafter, the output position information is output to the coordinate position (S205), and a display output command is sent (S206). The above processing is repeated for each display frame.

[0075] (Supplementary explanation on alpha synthesis, etc.)

The above is the explanation for the RGB system. For the YCrCb system, it is as follows. In YCrCb system, Y: Cr: Cb = 8: 8: 8 bit, this is converted to R: G: B = 8: 8: 8 bit, and the generated A = 8 bit is synthesized, and R : G: B: A = 8: 8: 8: 8 bit pixel information.

At this time, the formula of α synthesis (alpha blend) is as follows. Basically, it is hardware-implemented on a daraffic chip.

[0077] (Cs-Cd) * As + Cd · · · · (1)

here,

Cs: Source RGB value

Cd: RGB value to write to

As: Alpha value of the source (theoretically takes a value of 0—1)

Cs and Cd are each 0 to 255 for each RGB.

Based on the above equation (1), the synthesis process is performed.

According to the present embodiment, the player image extraction process and the image output process during the game are executed independently, and when the image output process needs to be combined with the player image, it is stored in the memory at that time. Since the player's a-compositing image data is used, the waiting time due to the player image extraction process can be reduced, so that real-time performance can be improved.

[0079] In addition, the background noise removal process and the noise reduction process can be performed while directly changing the original image data itself without having to copy the captured original image data. The memory capacity can be saved.

It should be noted that when the CG background and the player's clip image are combined, the player's clip image is rotated and enlarged / reduced to generate a more realistic composite image.

[0081] <Third Embodiment>

Next, a third embodiment will be described.

This embodiment removes fine edges by performing pre-processing on edge processing as compared to the first or second embodiment, and also removes background noise from edge noise to remove background noise. 24 enables the removal of the background noise.

FIG. 23 is a block diagram of the image processing apparatus according to the present embodiment. For the configuration in Figure 2

A pre-processing means 29 and a post-processing means 30 are added before and after the silhouette area extracting means 28 for extracting the range of the non-background area of the silhouette image and the background noise removing means 24, respectively.

FIG. 24 is a flowchart showing a processing procedure of the image processing apparatus 1 of the present embodiment. In this figure, step S901, step S906, step S908, and step S9

Each process of 09 is the same as that of the first embodiment.

Hereinafter, the operation of the image processing apparatus 1 according to the present embodiment will be described with reference to FIGS. 24 to 34, focusing on differences from the first embodiment.

[0085] [Image separation processing] (Step S902 in FIG. 24)

The image separation means 22 uses the background image and the extraction target image that are the output of the image input means 21.

V, a silhouette image is created by the procedure described in the first embodiment, and then the silhouette area extracting means 28 is activated.

[0086] [Silhouette region extraction processing] (step S903)

The silhouette area extraction means 28 takes a histogram for this silhouette image, compares it with a predetermined threshold value, and determines that the area narrower than that is determined as noise and removes it.

FIG. 25 is a flowchart showing the processing procedure of the silhouette region extraction means 28. In this figure, when activated, the silhouette area extraction means 28 first generates a vertical histogram of the silhouette image (S 1001). FIG. 29 is an explanatory diagram of the silhouette region extraction process. In the silhouette image of Fig. 29 (a), the white outline is the non-background area. The white circles in the upper left indicate non-background noise that appears in the background area. The vertical histogram of this silhouette image is shown in Fig. 29 (b). The white part in this figure means the number of non-background pixels in the corresponding vertical (column) direction in Fig. 29 (a) or the appearance ratio.

[0088] Then, as the non-background extraction processing, the vertical non-background pixel histogram value is divided into an area where the value is 0 and an area where it is not, and the non-zero area has a width equal to or greater than a predetermined value. This area is selected as a silhouette area (S1002). FIG. 29 (c) is a diagram in which both sides of the silhouette image of FIG. 29 (a) are divided (cut) in the vertical direction by this processing. In this figure, the area where the histogram value is not 0 but is less than or equal to the predetermined value (area B in Figure 29 (b)) is excluded as a non-background noise region, and the histogram value is 0. Area (area C in Figure 29 (b)) is excluded as a background area.

Next, a horizontal histogram is generated for the silhouette image of FIG. 29 (c) (S1003). FIG. 29 (d) shows a horizontal histogram. Based on this histogram, a non-background area is extracted and divided as a silhouette area in the same manner as described above (S1004). The above steps S1001 to S1004 are repeated a predetermined number of times, and the final silhouette region is extracted while sequentially dividing the silhouette region. Instead of repeating a predetermined number of times, the extraction process may be terminated if a new non-background noise region is not found. Fig. 29 (e

) Indicates the extracted silhouette region. Then, the area other than the silhouette area is filled with the background pixel value as a background area (S1005).

[0090] [Pretreatment] (Step S904, S905)

The pre-processing means 29 performs a process of removing a thin edge and a background process of edge noise on the area extracted by the silhouette area extracting means 28 in the silhouette image.

[0091] 1. Fine !, edge noise (background noise) removal processing (step S904)

Figure 30 (a) shows an example of thin edge noise that can be generated inside the operator silhouette when a window frame such as a sash is in the background and the operator's hair overlaps. Fig 3

The A part of 0 (a) is the edge noise.

[0092] In the following, according to Fig. 26, such a thin noise is silhouetted (the white part in Fig. 30 (a)). The procedure for filling by edge extraction and synthesis will be described.

First, a contour extraction filter is applied to a silhouette image to extract silhouette edges (S1101). For edge extraction, for example, the weighted Laplacian filter shown in FIG. 33 is used. An image extracted by this processing is shown in FIG. 30 (b).

Next, the edge extracted image (FIG. 30 (b)) and the original silhouette image (FIG. 30 (a)) are synthesized (S1102).

Since both the silhouette part and the edge part have a value of 255, when they are overlaid, the silhouette edge is expanded by one pixel.

Thin edge noise is usually only about 1 or 2 pixels wide, so by applying the above processing, it can be filled as non-background as shown in Fig. 30 (c).

[0094] 2. Background noise processing of edge noise

An example of edge noise that has become a large hole is shown in Fig. 31 (a). This edge noise is likely to occur when the color of the operator's clothes is close to that of a single color area with a wide background. This noise can be removed using the background noise elimination algorithm described in the first embodiment by closing the noise mouth (B in the figure) and converting it to background noise. Become.

Hereinafter, the procedure for closing the mouth portion of the noise will be described with reference to FIG.

First, the contour extraction filter is applied to the silhouette image (Fig. 31 (a)) to obtain an edge image (Fig. 3).

1 (b)) is generated (S1201), and this edge image and the original silhouette image are combined (Fig. 3

1 (c)) is generated (S 1202). Furthermore, for this synthesized silhouette image,

S1201—The process of step S1202 is repeated a plurality of times. (Fig. 31 (d) (e)). Since it is a small noise, if it is repeated several times, the edge swells and the mouth closes.

When the mouth is closed, this edge noise can be removed by performing background noise removal processing on the silhouette image. (Fig. 31 (f))

The size of the mouth of the noise varies depending on the situation, so the operator can specify how many times the edge extension is repeated.

For example, by using the threshold setting means 23 described in the second embodiment, a threshold setting screen corresponding to the screen illustrated in FIG. If the return count is set and registered, image processing can be performed using an appropriate number of repetitions.

[0098] By the above pre-processing, thin edges are removed, and for edge noise that has become a large hole, a noise image is created by closing the noise mouth.

[0099] [Background noise removal processing] (Step S906)

Next, the background noise is removed from the silhouette image created by the preprocessing means 29 by the background noise removing means 24. Since this process is the same as that in the first embodiment, a description thereof will be omitted. The background noise of the silhouette image is removed by the background noise removal process.

[0100] [Post-processing] (Step S907)

The silhouette image created by the above process is expanded from the original sketch image, although noise is removed. For this reason, the post-processing means 30 is activated and this expanded silhouette image is brought close to the original silhouette image by the procedure shown in FIG.

First, an outline extraction filter is applied to extract the inner edge of the silhouette image (S1301). At this time, a Laplacian filter illustrated in FIG. 34 may be applied. Next, the pixel value of the original silhouette image (extended image) is subtracted from the pixel value of the edge image created by applying this filter (S1302). If the pixel value becomes smaller than 0, it is set to 0 (S1303, S1304). This process reduces the edge partial force by 1 pixel. The above steps S 1301 to S 1304 are repeated as many times as the number of applied extended processes.

[0102] [Noise mitigation processing, image output processing] (Step S908, S909)

Further, the noise reduction means 25 performs the remaining non-background noise and edge noise reduction processing (S908), and the image output means 26 outputs the composite image (S909). Since the processing procedures of these means 25 and 26 are the same as those in the first embodiment, the description thereof is omitted.

The operation of the image processing apparatus according to the present embodiment has been described above with a focus on differences from the first embodiment. However, the preprocessing unit 29 and the postprocessing unit of the present embodiment are compared with the second embodiment. 30 etc. can be provided. [0104] In the present embodiment, the filtering (noise removal, mitigation) process is applied only to the region that cannot be determined as noise by the silhouette region extraction means, so that the filtering speed is improved.

[0105] Further, the pre-processing means removes thin edge noise in advance, and closes the mouth for edge noise that has a narrow mouth portion and a large spread in a non-background area, and background noise is generated. Since the noise is removed by the noise removing means, it is possible to obtain a composite image with better quality than the noise mitigation effect of the first embodiment.

[0106] The present invention is not limited to the embodiments described above, and various modifications can be made without departing from the scope of the invention. For example, when both the background noise removal means and the noise reduction means are executed, the background noise removal means and the noise reduction means are executed in accordance with the performance of the iS CPU that must be executed in the above order. Even just doing it can reduce noise. Further, the post-processing means 30 can obtain a certain effect even if it is activated at any timing after the background noise removal processing. For example, it may be activated after noise reduction processing. If the number of loops of the edge enlargement processing by the preprocessing means is small, the postprocessing means can be omitted. Industrial applicability

[0107] The image processing apparatus and image processing method according to the present invention can be used for all purposes such as extracting a specific image such as a person and replacing the other background portion with another image. In addition to the game machine described in the above embodiment, for example, in a videophone, only the caller is extracted and the background is replaced with another image, or only a specific subject is extracted in camera shooting and other backgrounds are extracted. It becomes possible to replace with. The image processing apparatus includes a game machine, a video phone, a camera, and the like having such functions.

Brief Description of Drawings

FIG. 1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention.

2 is a functional block diagram of the processing unit in FIG.

3 is an example of a background image input by the image input means of FIG.

[Fig. 4] An example of an extraction target image that includes an extraction target (person) photographed against the background of Fig. 3. The

FIG. 5 is an explanatory diagram of a silhouette image created by the image separation means of FIG.

[Fig. 6] An illustration of noise generated in the silhouette image of Fig. 5, Fig. 6 (a) is an illustration of background noise, Fig. 6 (b) is an illustration of non-background noise, and Fig. 6 (c) is an illustration It is explanatory drawing of edge noise. [7] FIG. 7 is an explanatory diagram of the processing procedure of the background noise removing means of FIG.

FIG. 8 is an explanatory diagram of a processing procedure of the noise mitigation means of FIG.

9] FIG. 9 is an explanatory diagram of image composition by the image output means of FIG.

圆 10] Functional block diagram of the image processing apparatus processing unit according to the second embodiment of the present invention. 圆 11] Explanatory diagram of how to limit the standing position of the player at the preliminary preparation stage according to the second embodiment of the present invention. It is.

12] An explanatory diagram of a threshold setting screen according to the second embodiment of the present invention. Figure 12 (a) shows an example where a lot of noise occurs when the threshold is low. Figure 12 (b) is an example where the threshold is appropriate and almost no noise is generated. Fig. 12 (c) shows an example in which the extraction target is bitten into the background by edge noise when the threshold is high.

13] It is explanatory drawing of the other threshold value setting screen by the 2nd Embodiment of this invention. FIG. 13A is a diagram showing a low threshold value, FIG. 13B is a diagram showing a threshold value, and FIG. 13C is a diagram showing a high threshold value.

FIG. 14 is a flowchart showing the image processing procedure of the game player according to the second embodiment of the present invention.

15] An example of data associating a player image and a game background frame according to the second embodiment of the present invention.

FIG. 16 is a flowchart showing a processing procedure of the image output means of FIG.

FIG. 17 is a flowchart showing a processing procedure of the background noise removing means of FIGS. 2 and 10.

FIG. 18 is a flowchart of the background noise candidate area mark processing routine of FIG.

FIG. 19 is a flowchart of the candidate area upper confirmation processing routine of FIG.

20 is a flowchart of the candidate area downward confirmation processing routine of FIG.

FIG. 21 is a flowchart of the candidate area conversion processing routine of FIG. FIG. 22 is a flowchart showing a processing procedure of the noise mitigation means of FIGS. 2 and 10.

FIG. 23 is a functional block diagram of an image processing device processing unit according to the third embodiment of the present invention.

24 is a flowchart of a processing procedure executed by each means of the processing unit in FIG. 23.

FIG. 25 is a flowchart showing a processing procedure of the silhouette region extraction means of FIG. 23.

FIG. 26 is a flowchart showing a procedure of thin edge noise removal processing executed by the preprocessing means of FIG. 23.

FIG. 27 is a flowchart showing a procedure of edge noise background noise processing executed by the preprocessing means in FIG. 23.

FIG. 28 is a flowchart showing a procedure of extended edge reduction processing executed by the preprocessing means of FIG. 23.

FIG. 29 is an explanatory diagram of the silhouette region extraction process of FIG. 25.

FIG. 30 is an explanatory diagram of the thin edge noise removal process of FIG. 26.

FIG. 31 is an explanatory diagram of edge noise background noise processing in FIG. 27;

FIG. 32 is an explanatory diagram of the extended edge reduction process of FIG. 28;

FIG. 33 is an example of a weighted Laplacian filter used in the thin edge noise removal process of FIG. 26 or the edge noise background noise conversion process of FIG.

FIG. 34 is an example of a weighted Laplacian filter used in the extended edge reduction process of FIG.

Explanation of symbols

1 Image processing device (game device)

11 AZD conversion means

12 Processing section

13 memory

14 Display memory

15 DZA conversion means

16 External storage device Input section

Image input means Image separation means Threshold adjustment means Background noise removal means Noise mitigation means Image output means Game execution means Silhouette region extraction means Preprocessing means

Post-processing means

The camera

TV monitor

Claims

The scope of the claims

[1] An image input means for inputting a background image and an extraction target image including the extraction target with the same background as the background image;

Image separation means for creating a silhouette image having a non-background pixel and a background pixel based on a difference between pixel values of the background image and the extraction target image;

When the silhouette image is input, an area surrounded by non-background pixels is marked for each line of the input image, and when this mark area touches a background pixel in an adjacent row or column, the mark area is set as the background area. A background noise removal means that edits the silhouette image by converting the remaining mark area into a non-background area,

Based on the digitized non-background level, the non-background level averaging process of the neighboring area of the pixel is sequentially performed for each pixel from the edge of the image area of the silhouette image edited by the background noise removing means.ヽ, and then noise reduction means to correct the image area range, and the O-composition image based on the averaged non-background level and the extraction target image, together with the separately input background image Image output means for outputting;

An image processing apparatus comprising:

[2] Create an edge extraction image of the silhouette image created by the image separation means, and combine the edge extraction image and the silhouette image to expand the non-background area of the silhouette image to thereby reduce the background of the edge noise. Pre-processing means to convert to noise,

2. The image processing apparatus according to claim 1, wherein the background noise removing unit uses the silhouette image created by the preprocessing unit as an input image.

[3] A vertical or horizontal histogram of the silhouette image created by the image separation unit is taken, and a range to be edited or corrected by the background noise removal unit or the noise reduction unit based on the width of the histogram. 2. The image processing apparatus according to claim 1, further comprising a silhouette region extracting means for determining.

[4] The image separating means determines whether the force is a non-background pixel or a background pixel by comparing a difference between pixel values of a background image and an extraction target image with a threshold value, and A threshold adjustment unit is provided that displays an image that can be determined based on the threshold and is capable of determining the degree of noise, and that changes the threshold based on the threshold change request input from an input unit. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized.

[5] Provided with means for displaying range designation information indicating the extraction target area on the display unit,

4. The image processing apparatus according to claim 1, wherein the image input unit inputs an extraction target image in the range based on the range designation information.

[6] An image processing method for extracting a specific image from an input image and combining it with another image.

Inputting a background image and an extraction target image including the extraction target with the same background as the background image;

Creating a silhouette image having a non-background pixel and a background pixel based on a difference between pixel values of the background image and the extraction target image;

If you mark the area surrounded by non-background pixels for each line of this silhouette image, and this mark area touches the background pixel of the adjacent row or column, change the mark area to the background area sequentially, After converting the remaining mark area to a non-background area and editing the silhouette image, based on the digitized non-background degree, the edge area of the image area of the silhouette image Sequentially performing the process of averaging the non-background level of the image, and then correcting the image area range;

Generating an O composition image based on the averaged non-background degree and the extraction target image, and outputting the generated image together with a separately input background image;

An image processing method comprising:

[7] An image processing method that extracts a specific image from an input image and combines it with another image.

Edge noise is reduced by expanding the non-background area of this silhouette image. Step to convert to

An image processing method comprising:

A computer game program for displaying a game player image captured by a camera in the background of a game and executing the game,

Input and save a background image,

A process of inputting an extraction target image including a player with the same background as the background image; and a non-background pixel and a background pixel based on a difference between pixel values of the background image and the extraction target image stored Image separation processing to create a silhouette image that has, and mark the area surrounded by non-background pixels for each line of this silhouette image, if this mark area touches the background pixel of the adjacent row or column, the mark area A background noise removal process that edits the shot image by converting the remaining mark area into a non-background area,

Based on the digitized non-background level, a process that averages the non-background level of the neighboring area of each pixel from the edge of the image area of the silhouette image edited by the background noise removing means.ノイズ, then noise reduction processing to correct the image area range,

Based on the averaged non-background level and the extraction target image, an O composition image is created, and a background image created by a process of executing a game progress or input from an external storage device Image output processing to be output together with,

A program that causes a computer to execute.