JP6711031B2 - Image processing apparatus, image processing method, image processing system and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, an image processing system, and a program.

In the field of image processing, with respect to an unknown area, which is a defective area of an image, pixel values that should be present may be predicted from surrounding areas (Inpainting processing). Also, image processing such as expressing the texture of the cutout boundary may be performed. Furthermore, a matting process may be performed in which a pixel value representing the degree of foreground is represented by multivalues at the boundary between binary images separated into the foreground and the background.

Non-Patent Document 1 describes that graffiti on a photograph is removed, a fold area of an old scanned picture is designated, and a pixel value of the designated area is predicted from surrounding pixels.

In Non-Patent Document 2 and Non-Patent Document 3, regarding the boundary portion of the region separated into the foreground and the foreground in the image, whether the pixel belonging to the foreground or the foreground is a multi-valued (degree ) Method (Matting process) is described. In this method, the boundary pixels specify an unknown region that is considered to be neither the foreground nor the foreground, and the pixel values that can be approximated as the foreground component and the background component can be approximated to the pixels in the unknown region. Both pixel values are obtained, and then a multivalued image (Matting image) representing the foreground degree is created.

M. Bertalmio, G. Sapiro, V. Caselles and C. Ballester, "Image Inpainting", Proc. of Siggraph 2000, New Orleans, LA, July 2000. J. Sun, J. Jia, C.-K. Tang, and H.-Y. Shum, "Poisson matting," Proc. Siggraph, pp. 315-321, 2004. M. Gong, L. Wang, R. Yang, and Y-H. Yang. Real-Time Video Matting Using MultiChannel Poisson Equations. In Proc. Graphics Interface (GI), 2010.

For example, when performing the matting process, the foreground component and the foreground component for the unknown region at the boundary between the foreground and the foreground are predicted. That is, when performing the matting process, it is necessary to perform a process of predicting pixel values corresponding to the foreground and the foreground at the boundary between the foreground and the foreground. Then, for pixels near the boundary, the foreground component and the foreground component estimated from each of the foreground region and the foreground region are predicted, and then they are connected by a coupling coefficient to obtain the pixel value of the original image. At this time, the coefficient for the foreground component becomes an image (Matting image) representing the foreground degree.
However, the foreground component prediction or the foreground component prediction for pixels in an unknown region basically requires a search process such as searching for a pixel whose position is the closest, and a process for searching for a pixel closest to the periphery for each pixel. It becomes necessary to do this for all pixels in the unknown area.
The present invention provides an image processing apparatus or the like that can perform a process of predicting a pixel value in an unknown region whose pixel value is to be determined at a higher speed than a case of performing a search process of searching for a nearest neighboring pixel for each pixel. The purpose is to provide.

The invention according to claim 1 provides an image information acquisition unit that acquires image information of an image, an unknown region setting unit that sets an unknown region whose pixel value is to be determined from the image, and a known region other than the unknown region. A pixel having a numerical value indicating the degree of possibility of belonging to a plurality of known regions as the included strength is set as a starting pixel, and the strength of the starting pixel and the unknown region of the starting pixel are included. A pixel value prediction unit that predicts a pixel value of a pixel in the unknown region based on a numerical value that represents the closeness of the pixel value of the pixel of the starting point and the pixel included in the unknown region as a weight exerted on the pixel. The image processing apparatus includes the image processing apparatus.
The invention according to claim 2, wherein the pixel value predicting unit of a pixel included in the unknown region on the basis of the weighting on pixels included in the unknown region of the pixel intensity and the origin of the pixels of the starting point The pixel value of a pixel in the unknown region is predicted by determining a pixel value, and further repeating the determination with the pixel having the pixel value determined as a pixel of a new starting point. The described image processing apparatus.
Invention of claim 3, wherein the pixel value prediction unit, an image processing apparatus according to pixel values of pixels that are newly starting point to claim 2, characterized in that the pixel values of the pixels of the starting point Is.
The invention according to claim 4 is the image processing apparatus according to any one of claims 1 to 3 , wherein the unknown area setting unit sets an unknown area according to a user's instruction.
The invention described in claim 5, wherein the unknown region setting unit, an image according to any one of claims 1 to 4, characterized in that to set the defective area is not in the original image as an unknown region It is a processing device.
According to a sixth aspect of the present invention, an image information acquisition unit that acquires image information of an image, an unknown region setting unit that sets an unknown region whose pixel value is to be determined from the image, and a known region other than the unknown region are provided. A pixel having a numerical value indicating the degree of possibility of belonging to a plurality of known regions as the included strength is set as a starting pixel, and the strength of the starting pixel and the unknown region of the starting pixel are included. A pixel value predicting unit that predicts a pixel value of a pixel in the unknown region based on a weight exerted on the pixel, and the pixel value predicting unit determines the strength of the starting point pixel and the unknown value of the starting point pixel. a images processing apparatus for predicting a pixel value of a pixel in the unknown region based on the pixel value of the origin of the pixel value obtained by multiplying the weight on the pixel becomes maximum included in the area.
The invention according to claim 7 is an image information acquisition unit that acquires image information of an image, an unknown region setting unit that sets an unknown region whose pixel value is to be determined from the image, and a known region other than the unknown region. A pixel having a numerical value indicating the degree of possibility of belonging to a plurality of known regions as the included strength is set as a starting pixel, and the strength of the starting pixel and the unknown region of the starting pixel are included. a pixel value predicting unit for predicting a pixel value of a pixel in the unknown region based on the weighted on pixels, the binary image generating unit, e Bei and the multi-level image generating unit, wherein the binary image generating unit Generates a binary image represented by a binary value indicating the foreground or the foreground from the image information, and the unknown region setting unit uses the binary image to define a boundary between the foreground and the foreground. Section sets the unknown region, and the pixel value prediction unit uses the image information to set reference pixels for pixels belonging to the foreground and pixels belonging to the foreground in the known region, respectively. A pixel value for a pixel in an unknown region is predicted for each case, and the multi-valued image generation unit represents a multi-value representing a foreground degree from the pixel value for the pixel in the unknown region predicted by the pixel value prediction unit. determining the pixel values are images processing device according to claim.
In the invention according to claim 8, the unknown region setting unit sets the unknown region by applying a filter for canceling the setting of a binary pixel value centering on a pixel at the boundary between the foreground and the foreground. The image processing apparatus according to claim 7, characterized in that.
The invention according to claim 9 acquires image information of an image, sets an unknown region for which a pixel value is desired to be determined from the image, and includes a plurality of known regions as strengths included in a known region other than the unknown region. A pixel having a numerical value indicating the magnitude of the possibility of belonging to the pixel of the starting point, and the pixel of the starting point as a weight exerted on the strength of the pixel of the starting point and the pixel included in the unknown region of the pixel of the starting point. And an image processing method for predicting the pixel value of a pixel in the unknown region based on a numerical value indicating the closeness of the pixel value to the pixel included in the unknown region .
The invention according to claim 10 comprises a display device for displaying an image, and an image processing device for performing image processing on image information of the image displayed on the display device, wherein the image processing device comprises: An image information acquisition unit that acquires image information of an image, an unknown region setting unit that sets an unknown region whose pixel value is desired to be determined from an image, and a plurality of known regions as strengths included in a known region other than the unknown region. Of the starting point, a pixel having a numerical value indicating the magnitude of the possibility of belonging to the starting point is defined as the starting point pixel, and the strength of the starting point pixel and the weighting of the starting point pixel on the pixel included in the unknown area of the starting point. An image processing system comprising: a pixel value prediction unit that predicts a pixel value of a pixel in the unknown region based on a numerical value indicating a pixel value proximity of the pixel and a pixel included in the unknown region .
In the invention according to claim 11, an image information acquisition function of acquiring image information of an image, an unknown area setting function of setting an unknown area for which a pixel value is desired to be determined from an image, and an area other than the unknown area A pixel having a numerical value indicating the possibility of belonging to a plurality of known regions included in the known region is set as a starting pixel, and the strength of the starting pixel and the unknown value of the starting pixel A pixel value prediction function for predicting the pixel value of a pixel in the unknown area based on a numerical value indicating the closeness of the pixel value of the pixel of the starting point and the pixel included in the unknown area as a weight exerted on the pixel included in the area It is a program that realizes and.

According to the invention of claim 1, the process of predicting the pixel value in the unknown region for which the pixel value is desired to be determined can be performed faster than the case of performing the search process of searching for the nearest neighboring pixel for each pixel. An image processing device can be provided.
According to the invention of claim 2, the process of predicting the pixel value can be further speeded up.
According to the invention of claim 3 , the determination of the pixel value becomes easier.
According to the invention of claim 4, the portion desired by the user can be set as the unknown region.
According to the invention of claim 5 , it is possible to perform a process of predicting a defective area.
According to the invention of claim 6 , the pixel value in the unknown region can be determined more accurately.
According to the seventh aspect of the present invention, it is possible to perform a matting process in which the pixel value representing the foreground degree is represented by multiple values at the boundary between the binary images separated into the foreground and the background.
According to the invention of claim 8, the setting of the unknown region becomes easier.
According to the invention of claim 9, the process of predicting the pixel value in the unknown region in which the pixel value is desired to be determined can be performed faster than the case of performing the search process of searching for the nearest neighboring pixel for each pixel. An image processing method can be provided.
According to the invention of claim 10, it is possible to provide an image processing system capable of performing image processing more easily.
According to the invention of claim 11, the process of predicting the pixel value in the unknown region for which the pixel value is desired to be determined can be performed at a higher speed than the case of performing the search process of searching for the nearest neighboring pixel for each pixel. The function can be realized by a computer.

It is a figure which shows the structural example of the image processing system in this Embodiment. It is a block diagram showing an example of functional composition of an image processing device in this embodiment. (A)-(b) is the figure which showed the example of the method of performing the work which designates an unknown area|region interactively by a user. It is the figure shown about the other example of a deficient area. (A)-(b) is a figure explaining weighting. (A)-(b) is the figure shown about the method of determining a weight. (A) shows the image before the smoothing, and (b) shows the image after the smoothing. (A)-(i) is the figure which showed the process which determines the pixel value of an unknown area|region based on the pixel value of a known area|region. FIG. 5A is a diagram similar to FIG. 5-4(i) and shows the state of pixel values when one loop is completed. (B) shows the state of the pixel value when the process converges by performing a plurality of loops. (A)-(i) is the figure which showed the process which determines the pixel value of an unknown area|region by another method based on the pixel value of a known area|region. (A) is a diagram similar to FIG. 5-6(i), and shows the state of pixel values when one loop is completed. (B) shows the state of the pixel value when the process converges by performing a plurality of loops. (A)-(b) has shown the example which determined the pixel value of the pixel in an unknown area about the image of FIG.3(b). (A)-(b) has shown the example which input the unknown area about the image of FIG. 4 and determined the pixel value of the pixel in this unknown area. (A)-(c) is a conceptual diagram at the time of performing Retinex processing and improving the visibility with respect to an original image. (A)-(b) has shown the example which the unknown area setting part set the unknown area to the area|region where the noise in the image has generate|occur|produced. FIG. 7A is a diagram showing that a region in which noise is generated has a pixel value whose value is significantly different from that in the surrounding area. (B) is a smoothed version of the pixel value shown in (a). (A) is a figure showing a case where a pixel value in a region where noise is generated is replaced with another pixel value. (B) is a smoothed version of the pixel value shown in (a). FIGS. 8A and 8B show an example in which the original image of FIG. 8A is improved in visibility and noise is removed. 6 is a flowchart illustrating an operation of the image processing apparatus according to the first and second embodiments. It is a block diagram showing an example of functional composition of an image processing device in a 3rd embodiment. (A)-(b) is a figure shown about processing which a binary image generation part performs. (A)-(b) is the figure which showed the process which an unknown area setting part sets an unknown area. The case where an unknown region is set at the boundary between the foreground and the foreground for the original image is shown. It is a conceptual diagram of the process which a pixel value prediction part performs. (A)-(d) is a figure for demonstrating the mode of the process which a pixel value prediction part specifically performs. (A)-(i) has shown the case where a pixel value from only a foreground area is propagated. 16A is a diagram similar to FIG. 16C, and shows the state of the pixel values when the pixel values of the unknown region are generally determined. (B) shows the state of pixel values when the loop is further repeated and converged. (A)-(i) has shown the case where a pixel value is propagated only from a foreground area. FIG. 16A is a diagram similar to FIG. 16-5(i) and shows the state of the pixel values when the pixel values of the unknown region are generally determined. (B) shows the state of pixel values when the loop is further repeated and converged. (A) shows the result of setting the seed 1 and obtaining the pixel value of the pixel in the unknown region from the foreground. (B) shows the result of setting the seed 2 and obtaining the pixel value of the pixel in the unknown region from the background. The foreground degree obtained by the multivalued image generation unit is imaged. 9 is a flowchart illustrating an operation of the image processing apparatus according to the third exemplary embodiment. It is a figure showing the hardware constitutions of an image processing device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Explanation of the entire image processing system>
FIG. 1 is a diagram showing a configuration example of an image processing system 1 according to the present embodiment.
As shown in the figure, the image processing system 1 according to the present embodiment inputs an image processing device 10 that performs image processing on image information of an image displayed on the display device 20, and image information created by the image processing device 10. The display device 20 displays an image based on this image information, and the input device 30 for the user to input various information to the image processing device 10.

The image processing device 10 is, for example, a so-called general-purpose personal computer (PC). Then, the image processing apparatus 10 is configured to create image information and the like by operating various application software under the control of an OS (Operating System).

The display device 20 displays an image on the display screen 21. The display device 20 is composed of, for example, a liquid crystal display for a PC, a liquid crystal television or a projector, and the like having a function of displaying an image in an additive color mixture. Therefore, the display system of the display device 20 is not limited to the liquid crystal system. In the example shown in FIG. 1, the display screen 21 is provided in the display device 20, but when a projector is used as the display device 20, the display screen 21 is, for example, a screen provided outside the display device 20. Becomes

The input device 30 is composed of a keyboard, a mouse and the like. The input device 30 activates and terminates application software for performing image processing, and the user inputs an instruction for performing image processing to the image processing device 10 when performing image processing, which will be described in detail later. Used to.

The image processing device 10 and the display device 20 are connected via a DVI (Digital Visual Interface). Note that the connection may be made via HDMI (registered trademark) (High-Definition Multimedia Interface), DisplayPort, or the like instead of DVI.
The image processing device 10 and the input device 30 are connected via, for example, a USB (Universal Serial Bus). It should be noted that instead of USB, it may be connected via IEEE1394, RS-232C, or the like.

In such an image processing system 1, the display device 20 first displays an original image which is an image before image processing. When the user uses the input device 30 to input an instruction for performing image processing to the image processing device 10, the image processing device 10 performs image processing on the image information of the original image. The result of this image processing is reflected on the image displayed on the display device 20, and the image after image processing is redrawn and displayed on the display device 20. In this case, the user can perform the image processing interactively while looking at the display device 20, and can more intuitively and easily perform the image processing work.

The image processing system 1 according to the present embodiment is not limited to the one shown in FIG. For example, a tablet terminal can be exemplified as the image processing system 1. In this case, the tablet terminal includes a touch panel, which displays an image and inputs a user instruction. That is, the touch panel functions as the display device 20 and the input device 30. Similarly, a touch monitor can be used as a device in which the display device 20 and the input device 30 are integrated. This uses a touch panel as the display screen 21 of the display device 20. In this case, the image processing device 10 creates image information, and the image is displayed on the touch monitor based on this image information. Then, the user inputs an instruction for performing image processing by touching the touch monitor or the like.

<Description of image processing device>
[First Embodiment]
Next, a first embodiment of the image processing device 10 will be described.
In the first embodiment, an example will be described in which the image processing apparatus 10 is applied to a process of filling a defective area in an image (Inpainting).
FIG. 2 is a block diagram showing a functional configuration example of the image processing apparatus 10 according to the first embodiment. Note that, in FIG. 2, among various functions of the image processing apparatus 10, those relating to the first embodiment are selected and shown.
As shown in the figure, the image processing apparatus 10 according to the present embodiment includes an image information acquisition unit 11, a user instruction reception unit 12, an unknown area setting unit 13, a pixel value prediction unit 14, and an image information output unit 15. Prepare

The image information acquisition unit 11 acquires image information of an image on which image processing is performed. That is, the image information acquisition unit 11 acquires the image information of the original image before performing the image processing. The image information is, for example, RGB (Red, Green, Blue) video data (RGB data) for display on the display device 20.

The user instruction receiving unit 12 receives a user instruction input by the input device 30 regarding image processing.
Specifically, the user instruction receiving unit 12 receives, as user instruction information, an instruction to specify an unknown region in which the user wants to determine a pixel value from the image displayed on the display device 20.

In the present embodiment, a method of performing the work of designating an unknown area by user interaction described below is adopted.
3A and 3B are diagrams showing an example of a method of interactively performing the work of designating an unknown region.
Of these, FIG. 3A is an image before image processing, and shows the original image G1 displayed on the display device 20 based on the image information acquired by the image information acquisition unit 11. The original image G1 is an image obtained by reading a photograph including a person appearing as the foreground and a background appearing behind the person with a scanner or the like. Further, the original photograph has a crease, which is also read as the crease image D1 in the original image G1. In this case, the fold image D1 does not exist in the original image and can be considered as a defective region. That is, this defective region is a region where the original pixel value is unknown. Here, the location of the fold image D1 that is the missing area is set as the unknown area. The unknown region is a region to be filled in based on the image information of other pixels, and is a region for which the pixel value is desired to be determined.

FIG. 3B shows a case where the user inputs the portion of the fold image D1 as the unknown region M into the original image G1. In this case, the user inputs the unknown region M so as to cover the portion of the fold image D1.

This unknown region M can be input by drawing with the input device 30. For example, when the input device 30 is a mouse, the unknown region M is drawn by operating the mouse to drag the original image G1 displayed on the display screen 21 of the display device 20. If the input device 30 is a touch panel, the unknown area M is similarly drawn by swiping the original image G1 with the user's finger, touch pen, or the like.

The unknown region M is not limited to the case of drawing in the same direction, and may be drawn by performing a reciprocating operation. In addition, it is easier to input the unknown region M with a thicker line than a thin line. This may be realized by, for example, a method of mounting a brush tool having a thick pen size among brush tools used in image processing software for performing image processing.

Note that the image having the defective area is not limited to the case where the fold image D1 is provided.
FIG. 4 is a diagram showing another example of the defective region.
The illustrated original image G2 is an image obtained by scanning, with a scanner or the like, a photograph including a person who appears in the foreground and a background which appears behind the person. Then, the original photograph has graffiti, which is also read as the graffiti image D2 in the original image G2. In this case, the graffiti image D2 is a missing area that is not in the original image.

Also in this case, as shown in FIG. 3B, the user can input the unknown region M so as to cover the portion of the graffiti image D2. Further, since the graffiti image D2 is usually monotonous in color, the user can define the unknown region M by inputting this color.

The unknown area setting unit 13 sets an unknown area for which a pixel value is desired to be determined from the image.
Here, the unknown area setting unit 13 sets the location input by the user as the unknown area M by the method described above. In this case, the unknown area setting unit 13 sets the unknown area M according to a user's instruction. However, the present invention is not limited to this, and the unknown area setting unit 13 can automatically set the unknown area M, so to speak. It is not difficult for the unknown area setting unit 13 to detect the missing area and set the unknown area M if the missing area is clearly distinguishable from the surrounding image. For example, the unknown area setting unit 13 may set an area in the image having a color different from the surroundings as the unknown area M. In this case, the user instruction receiving unit 12 does not necessarily have to be provided.

The pixel value prediction unit 14 predicts the pixel value of the pixel in the unknown region M. Specifically, the pixel value prediction unit 14 predicts the pixel value in the unknown region M by propagating the pixel value from the pixel values of the pixels around the unknown region M to the pixels in the unknown region M. To do. Hereinafter, a specific example of this pixel value propagation method will be described.

Here, a method for determining the pixels of the unknown region M based on the pixels of the peripheral region other than the unknown region M with respect to the unknown region M described in FIG. Since the pixel values of the pixels in the peripheral area other than the unknown area M are fixed, this area may be referred to as “known area” hereinafter.
As described above, in the present invention, it is fundamental to determine the pixel value of the unknown region M by propagating the pixel value of the pixel of the known region other than the unknown region M to the pixel of the unknown region M. Is the way. As this method, the principle of expanding the area by propagating the strength of the pixel can be applied.

Specifically, the following method can be applied. This method is the method described in JP-A-2016-006645.
For example, when a region of an image is separated into a foreground and a foreground, a rough curve called a seed, which is specified by the user, is given, and a pixel to which a seed is given is labeled (for example, 1 for the foreground and 0 for the foreground). Provide a label). Then, the strength 1 is set to the seed-given pixel, and the strength is propagated from the seed-given pixel to the pixel to which the seed has not been applied yet. There is a method in which the stronger label is adopted while comparing the sizes. According to this method, pixels having respective labels are expanded from the seeds given to the foreground and the background, respectively, and finally the regions are separated into the foreground and the foreground.
At this time, weighting is considered as the degree of influence of strength from one pixel to an adjacent pixel. Then, for example, when propagating the strength from this one pixel to an adjacent pixel, it is basically performed by multiplying the strength possessed by one pixel by a weight and the multiplied value becomes the strength of the adjacent pixel. To do. At this time, the “strength” is the strength of belonging to the foreground or the foreground corresponding to the label, and represents the magnitude of the possibility that a certain pixel will belong to the foreground or the background of the label. The higher the strength, the higher the possibility that the pixel will belong to the foreground or the foreground corresponding to the label, and the lower the strength, the lower the possibility that the pixel will belong to the foreground or the background corresponding to the label.
The "weight" can be considered as follows.

FIGS. 5-1(a)-(b) are the figures explaining weighting.
In FIG. 5A, the adjacent pixel R that determines the weighting on the target pixel T is shown. In this case, the adjacent pixel R is 8 pixels adjacent to the target pixel T. The weight is then determined using the pixel information of the original image. That is, the weight of the adjacent pixel R with respect to the target pixel T is determined such that the closer the pixel value is, the larger the weight is, and the farther the pixel value is, the smaller the weight is. Whether or not the pixel values are close to each other can be determined by using, for example, the Euclidean distance of pixel values (for example, RGB values).

For example, if the pixel value of the target pixel T is P ₀ =(R ₀ , G ₀ , B ₀ ), and the pixel value of the adjacent pixel R is P _i =(R _i , G _i , B _i ), the RGB value The Euclidean distance d _i can be defined by the following formula 1.

Further, instead of the Euclidean distance d _i of the RGB value, the Euclidean distance d _i ^w using the YCbCr value shown in the following Expression 2 may be considered. In Equation 2, the pixel value of the target pixel T is P ₀ =(Y ₀ , Cb ₀ , Cr ₀ ), and the pixel value of the adjacent pixel R is P _i =(Y _i , Cb _i , Cr _i ). Is the Euclidean distance d _i ^w . Further, the Euclidean distance d _i ^w of the equation 2 is a weighted Euclidean distance using the weighting factors W _Y , W _Cb , and W _Cr .

Further, the pixel value is not limited to one having three components. For example, it is possible to use an n-dimensional color space and consider the Euclidean distance d _i ^w by n color components.
For example, the following Expression 3 is a case where the color components are X ₁ , X ₂ ,..., X _n . Then, in Expression 3, the pixel value of the target pixel T is P ₀ =(X ₁₀ , X ₂₀ ,..., X _n0 ), and the pixel value of the adjacent pixel R is P _i =(X _1i , X _2i , , X _ni ), the Euclidean distance d _i ^w is shown. Note that the Euclidean distance d _i ^{w in} Equation 3 is also a weighted Euclidean distance using the weighting factors W _X1 , W _X2 ,..., W _Xn .

FIG. 5-1(b) shows the magnitude of the weight determined for the target pixel T. Here, the adjacent pixel R having a larger weight determined for the target pixel T is indicated by a thicker line, and the adjacent pixel R having a smaller weight determined for the target pixel T is indicated by a thinner line. ing.

Note that the weight is determined from the Euclidean distance d _i by the following method.
FIGS. 5-2(a) and 5-2(b) are diagrams showing a method for determining the weight. In FIGS. 5-2(a) and 5-2(b), the horizontal axis represents the Euclidean distance d _i and the vertical axis represents the weight.
The Euclidean distance d _i is the Euclidean distance d _i of the pixel value determined between the pixels located around the pixel and the pixel given strength. Then, for example, as shown in FIG. 5-2(a), a non-linear monotonic decreasing function is determined, and a value determined by this monotonic decreasing function is weighted with respect to the Euclidean distance d _i .
That enough Euclidean distance d _i is small, the weights become larger, as the Euclidean distance d _i is large, weighting is smaller.
The monotonically decreasing function is not limited to the one having the shape shown in FIG. 5-2(a), and is not particularly limited as long as it is a monotonically decreasing function. Therefore, a linear monotonic decreasing function as shown in FIG. Further, it may be a piecewise linear monotone decreasing function that is linear in a specific range of the Euclidean distance d _i and nonlinear in other ranges.

As described above, as a result of the propagation of strength and the comparison of strength, the “label” is propagated and the regions can be separated. In this case, it can be considered that the label and the strength are propagated and the region is divided.

In the present embodiment, this method is further applied to propagate the “pixel value” by propagating the strength and comparing the strength, and thereby the pixel value of the unknown region M is determined based on the pixel value of the known region. It is characterized by making a decision. Then, in this case, it can be considered that the pixel value and the strength propagate and the pixel value of the unknown region M is determined.

In order to determine the pixel value of the unknown region M based on the pixel value of the known region, and to perform the above-mentioned principle of pixel value propagation, as described in FIGS. 5-1(a) and (b). It is necessary to calculate the weight. The pixel value used to calculate the weight is the pixel value of the original image, but the pixel value can be used as it is for the defective area. The missing area can be interpreted as the pen color specified by the user in the case of the fold image D1 shown in FIG. 3, and can be interpreted as the color of the scribble pen itself in the scribble image D2 shown in FIG. Since the color of the pen is known in advance, it may be replaced with another color convenient for the processing.

Further, if the color of the pixel in the defective area is used as it is, for example, smoothing is performed, the relationship with the adjacent pixel R is smoothed, and the calculation for obtaining the weight becomes easy.
FIG. 5-3(a) shows an image before smoothing, and FIG. 5-3(b) shows an image after smoothing.

5-4(a) to (i) are diagrams showing a process of determining the pixel value of the unknown region M based on the pixel value of the known region.
FIG. 5-4(a) is an image before the pixel value of the unknown region M is determined, and the unknown region M exists. The area around the unknown area M is a known area. At this time, it is assumed that the weight calculation has already been completed based on the pixel values of the original image. Further, all the pixel intensities in the unknown region M are set to 0, and all the pixel intensities in the known regions other than the unknown region M are set to 1. In this case, "strength" can be considered to represent the likelihood of having the current pixel value. For example, the probability of having the current pixel value in the known region is 1. Further, the probability of having the current pixel value in the unknown region M is 0.

Here, the pixels whose pixel values are to be determined are the pixels of the unknown region M shown in FIG. As described above, the pixels in the known area have a strength of 1, and the pixels in the known area having these strengths propagate the pixel value to the pixels in the unknown area M. For example, in this example, the target pixel T to be updated first is the pixel in the center of the frame shown in FIG. 5B, and the pixel strength and the target pixel T among the neighboring pixels in 8 neighborhoods. Receive the strength calculated according to the weight of and. At this time, it is the pixels that already have the strength that exert the strength on the target pixel T, and in this case, the pixels in the known area. Then, the pixel value of the adjacent pixel having the (strength×weight) among the neighboring pixels in 8 neighborhoods is accepted, and the one having the maximum (strength×weight (value multiplied by the strength and weight)) is accepted. Is updated to. In this example, as shown in FIG. 5-4(b), assuming that the strength (strength×weight) received from the adjacent right pixel R1 belonging to the known region is the strongest, the pixel value of the target pixel T is adjacent to the right. It can be seen that the pixel value has been changed. For example, when the strength of the adjacent pixel R1 on the right side is 1 and the weight from the target pixel T to the adjacent pixel R1 is 0.9, the strength given to the target pixel T is 1×0.9=0. It becomes .9.

In other words, in this case, the pixels in the known area that already have strength become the “starting point pixel”, and the pixels in the unknown area M are determined based on the strength of the starting point pixel and the weight exerted on the pixels included in the unknown area M of the starting point pixel. Predict the pixel value of a pixel.

Further, in this case, the propagation (update) of the pixel value and the strength is immediately performed by comparing (strength×weight) of the target pixel T and the adjacent pixel R (asynchronous type). Further, the strength of the target pixel T is updated (passive type) by comparing (strength×weight) received from the adjacent pixel R. Further, when the target pixel T already has strength, if (strength×weight) of the adjacent pixel R is stronger than (strength×weight) of the target pixel T at that time, both the strength and the pixel value are Will be updated.

The target pixel T is sequentially selected from the pixels in the unknown region M, and the same process is performed.
Next, the target pixel is the pixel in the center of the frame in FIG. 5C, and this is the target pixel T. Then, the pixel value is changed as in the case of FIG. 5-4(b). After that, the same processing is repeated from FIGS. 5D to 5D.

That is, in this case, the pixel whose pixel value has been determined together with the pixel in the already known area having the strength is further determined as a new "starting pixel".

In FIG. 5-4, an arrow shown by a solid line is an arrow when the target pixel T moves to the next pixel, and an arrow shown by a dotted line shows an arrow when the target pixel T is a pixel after the next next pixel. At this time, the state of the pixel values when all the pixels in the unknown region M are processed once is as shown in FIG. 5-4(i) with one loop of FIGS. 5-4(a) to (i). .. Then, the same processing is repeated further until the convergence.

FIG. 5-5(a) is a diagram similar to FIG. 5-4(i) and shows the state of pixel values when the above-mentioned one loop is completed. Further, FIG. 5-5(b) shows a state of the pixel value when the process is performed in a plurality of loops and converges.
At the stage of FIG. 5-4(a), the intensity of the pixel in the unknown region M is 0. However, as shown in FIG. And the propagated pixel value. Then, the second and subsequent loop updates start from that state. If the received strength is stronger than the current strength, the strength and the pixel value are both updated according to the rule.

5A to 5I are diagrams showing a process of determining the pixel value of the unknown region M based on the pixel value of the known region by another method.
In this case, the pixel value and the strength are similar to those in FIG. 5-4 in that the propagation of the pixel value and the strength is immediately performed by comparing (strength×weight) of the target pixel T and the adjacent pixel R (asynchronous type). ). On the other hand, the target pixel T propagates the pixel value and strength to the adjacent pixel R, and updates the pixel value and strength of the adjacent pixel R (attack type).

For example, in FIG. 5-6(b), the target pixel T propagates the strength and the pixel value to the adjacent pixel R2 immediately below.
Also in FIG. 5-6(d), the target pixel T propagates the strength and the pixel value to the adjacent pixel R in the vicinity of 8. However, for example, the pixel at the position of row 4 and column 3 is shown in FIG. ), it already has the strength propagated from the pixel that was the target pixel T located in row 4 and column 2. In this case, in FIG. 5-6(d), the strength is propagated from the pixels located in the third row and the third column, but at this time, the strength (strength×weight) from the target pixel T already has If it is above, it will be updated to the one above.

In this case as well, the pixel in the known area that already has strength becomes the “starting pixel” (in this case, the starting pixel is also the target pixel T), and the strength of the starting pixel and the unknown area M of the starting pixel The pixel values of the pixels in the unknown region M are predicted based on the weights exerted on the pixels included in. Then, the pixel whose pixel value has been determined together with the pixel in the already known region having the strength is further determined as a new "starting pixel".

5-6(a) to (i) are set as one loop, and the state of the pixel value when all the pixels in the unknown region M are processed once is as shown in FIG. 5-6(i). Then, the same processing is repeated further until the convergence.

FIG. 5-7(a) is a diagram similar to FIG. 5-6(i) and shows the state of the pixel value when the above-mentioned one loop is completed. Further, FIG. 5-7(b) shows a state of the pixel value when the process is performed in a plurality of loops and converges.
The pixel of the unknown area M has a strength of 0 at the stage of FIG. 5-4(a), but as shown in FIG. 5-7(a), one loop ends. The rest has some strength and propagated pixel values. Then, the second and subsequent loop updates start from that state.

As shown in FIGS. 5-4 and 5-7, the convergence state may change if the updating method is different. However, since it is not decided which is the true state, either one may be adopted.

In addition to this, the updated pixel state (strength and pixel value) is not immediately reflected, but is saved in another memory until the processing of one loop is completed, and when the one loop is completed, Update method may be used (synchronous type).

As described above, in the present embodiment, the strength of the pixel of the first unknown area M is set to 0, and the strength of the pixel of the known area other than the unknown area M is set to 1. Then, by propagating the strength and the pixel value to the pixel of the unknown area M, and comparing the strength in the process of updating, the pixel value of the larger strength is propagated to predict the pixel of the unknown area M. The basic feature is to do.

This is because the pixel value prediction unit 14 sets a pixel having a strength included in a known area other than the unknown area M as a starting point pixel, and a weight applied to the strength of the starting point pixel and a pixel included in the unknown area of the starting point pixel. It can also be said that the pixel value of the pixel in the unknown region M is predicted based on. Further, at this time, the pixel value prediction unit 14 determines the pixel value of the pixel included in the unknown region M based on the strength of the pixel of the starting point and the weight exerted on the pixel included in the unknown region M of the pixel of the starting point. It can also be said that the pixel value of the pixel in the unknown region M is predicted by repeating the determination with the pixel determined in step 1 as the pixel of the new starting point. Further, at this time, the pixel value prediction unit 14 determines the unknown region based on the pixel value of the starting pixel, which has the largest value obtained by multiplying the strength of the starting pixel and the weight exerted on the pixels included in the unknown region M of the starting pixel. Determine the pixel values in M. The determined pixel value is the pixel value of the starting pixel, and the pixel value propagates as it is.

The image information output unit 15 outputs the image information that has been subjected to the image processing as described above. The image information after the image processing is sent to the display device 20. Then, an image is displayed on the display device 20 based on this image information.

FIGS. 6A and 6B show an example in which the pixel value of the pixel in the unknown region M is determined for the image of FIG.
FIG. 6A is a diagram similar to FIG. 3A and is an original image G1 in which the fold image D1 is present. Then, when the unknown area M is input as shown in FIG. 3B and the pixel value prediction unit 14 predicts the pixel value of the unknown area M, an image G1′ as shown in FIG. 6B is obtained. In the image G1′ of FIG. 6B, the fold image D1 disappears.

Further, FIGS. 7A and 7B show an example in which the unknown area M is input for the image of FIG. 4 and the pixel value of the pixel in the unknown area M is determined.
FIG. 7A is a diagram similar to FIG. 4, and is an original image G2 in which a graffiti image D2 exists. Then, when the unknown area M is input and the pixel value prediction unit 14 predicts the pixel value of the unknown area M, an image G2′ as shown in FIG. 7B is obtained. In the image G2′ of FIG. 7B, the graffiti image D2 disappears.

[Second Embodiment]
Next, a second embodiment of the image processing device 10 will be described.
In the second embodiment, an example in which the image processing device 10 is applied to remove noise in an image will be described. A block diagram showing a functional configuration example of the image processing apparatus 10 according to the second embodiment is the same as that of the first embodiment.
Noise may be generated or conspicuous when originally present in an image, or by performing image processing such as contrast adjustment and visibility improvement.

As a means for improving the visibility of an image, for example, there is a method of performing Retinex processing.
Assuming that the pixel value (luminance value) at the pixel position (x, y) of the image is I(x, y) and the pixel value I′(x, y) of the image with improved visibility, the following Retinex processing is performed. The visibility can be improved by using the equation (4).

k is a parameter for emphasizing the reflectance, and R(x, y) is an estimated reflectance component. In the Retinex model, the visibility can be enhanced by emphasizing the reflectance component. In the present embodiment, R(x, y) may be calculated by any method of the existing Retinex model. When 0≦k≦1, k=0 represents the original image, and k=1 represents the reflectance image (maximum visibility). The k may be adjusted by the user or may be associated with the darkness of the image.

8A to 8C are conceptual diagrams when the Retinex process is performed to improve the visibility of the original image.
Of these, FIG. 8A is an original image G3, and FIG. 8B is an image G3′ after the Retinex process.

When the visibility is improved by the Retinex process or the like, the process of increasing the visibility of the dark part is performed with respect to the lightness. Therefore, the pixels in the dark area where the color information such as the hue and saturation of the pixels originally varies may become noise.

FIG. 8C is an enlarged view of a part of the image G3′ of FIG. 8B, and illustrates a case where noise occurs in the pixels in the dark area.
Since the dark area has a low lightness, it is difficult to visually recognize the color information, but a chromaticity component that is a color component other than the lightness (for example, in the HSV color space, the hue H and the saturation S, the L ^* a ^* b ^* color space). In this case, the chromaticity a ^* b ^* ) greatly varies, and may appear as color noise. Alternatively, the brightness itself may vary due to the Retinex process. Further, these may be combined to appear as noise.

In the present embodiment, the unknown area setting unit 13 sets a pixel or area that is considered to be noise as the unknown area M in which the pixel value is to be determined from the image G3'. In this case, noise is not present in the original image and can be considered as a defective region.

The unknown region setting unit 13 can set the unknown region M by the user inputting a noise region with the input device 30 and the user instruction receiving unit 12 receiving this as user instruction information.

On the other hand, the unknown area setting unit 13 may detect a noise area and set this area as the unknown area M. For example, in the state of the original image G3, a region that was a dark part is set as an unknown region M. Further, in the pixels in the area that is a dark part, the pixels whose color information such as lightness and chromaticity is far from the surroundings, or the average of the pixel area such as 3×3 or 5×5 is far from the average of the surrounding pixels. The pixel area can be

FIGS. 9-1(a) and 9-1(b) show an example in which the unknown area setting unit 13 sets the unknown area M in the area in the image G3′ where noise is generated.
Of these, FIG. 9-1(a) is an image G3′ similar to that of FIG. 8(b), and shows that noise is generated in the illustrated area.
Further, FIG. 9-1(b) shows a case where the unknown region M is set in a region in the image G3′ where noise is generated.

Specifically, as shown in FIG. 9-2(a), the area N in which noise is generated has a pixel value whose value is significantly different from that of the surrounding area. Therefore, for example, it is possible to detect by setting a threshold value by paying attention to the difference between the pixel values of the adjacent pixels. That is, when the adjacent pixel exceeds a predetermined threshold value, it is determined to be noise.

Further, when calculating the above-mentioned weight, the pixel value of the area N in which noise is generated may be used as it is. Alternatively, the weight may be calculated by using a smoothed pixel value shown in FIG. 9-2(a) as shown in FIG. 9-2(b).
Alternatively, as shown in FIG. 9-3(a), the pixel value of the area N in which noise is generated is replaced with another pixel value (in the figure, the case where the pixel value is replaced with a black pixel is shown). -3(b) may be smoothed.

In addition, after setting the unknown region M, the pixel value prediction unit 14 determines the pixel value of the unknown region M based on the pixel value of the known region, as in the first embodiment.

FIGS. 10A and 10B show an example in which the original image G3 of FIG. 8A is improved in visibility and noise is removed.
FIG. 10A is a diagram similar to FIG. 8A and shows an original image G3. And FIG.10(b) has shown the image G3" which improved the visibility and removed the noise about the original image G3 of FIG.8(a). That is, the image G3" of FIG.10(b) is Retinex process. After the visibility is improved by, the unknown area M is set in the area where noise is generated as shown in FIG. 9B, and the pixel value prediction unit 14 causes the pixel value of the pixel in the unknown area M to be increased. It is an image after determining.

FIG. 11 is a flowchart illustrating the operation of the image processing device 10 according to the first and second embodiments.
The operation of the image processing apparatus 10 according to the first and second embodiments will be described below mainly using FIGS. 2 and 11.

First, the image information acquisition unit 11 acquires RGB data as image information of an image to be subjected to image processing (step 101). This RGB data is sent to the display device 20, and an image before image processing is displayed.

Then, the user sets the missing area of the fold image D1 or the graffiti image D2 as the unknown area M, for example. The user instruction receiving unit 12 receives the user instruction information (step 102).

Next, the unknown area setting unit 13 sets an unknown area M for which a pixel value is desired to be determined from the image (step 103). This is performed based on the user instruction information received in step 102. The unknown area setting unit 13 can also automatically set the unknown area M as described above. In this case, step 102 is unnecessary.

Further, the pixel value prediction unit 14 predicts the pixel value of the pixel in the unknown area M (step 104). In the present embodiment, the pixel value prediction unit 14 predicts the pixel value of the pixel in the unknown area M by the area expansion method based on the pixel values around the unknown area M.

Then, the image information output unit 15 outputs the image information after the image processing (step 105). This image information is RGB data, and this RGB data is sent to the display device 20, and the image after image processing is displayed on the display screen 21.

[Third Embodiment]
Next, a third embodiment of the image processing device 10 will be described.
In the third embodiment, an example in which the image processing device 10 is applied to Matting will be described.
In image processing, when cutting out a specific area from an image, it is classified into two values, that is, the foreground to be cut out and the other foreground, and the method of Matting is used to represent an area with an ambiguous boundary in multivalue. May generate a mask. An example of an image having an ambiguous boundary is an image of fluffy clothing hair or human hair. Such an image can be rephrased as an image having a region in which light is transmitted through the foreground and the background is transparent. Hereinafter, a method of creating a mask using the method of Matting will be described.

FIG. 12 is a block diagram showing a functional configuration example of the image processing apparatus 10 according to the third embodiment.
As shown in the figure, the image processing apparatus 10 according to the present embodiment includes an image information acquisition unit 11, a binary image generation unit 16, an unknown area setting unit 13, a pixel value prediction unit 14, and a multi-valued image generation unit 17. And an image information output unit 15.
The illustrated image processing apparatus 10 does not include the user instruction receiving unit 12 in the image processing apparatus 10 of FIG. 2, but includes a binary image generating unit 16 and a multivalued image generating unit 17.

The image information acquisition unit 11 is the same as in the first embodiment. That is, the image information acquisition unit 11 acquires image information of an image on which image processing is performed.

The binary image generation unit 16 generates a binary image in which the image information is represented by a binary value indicating the foreground or the foreground.
13A and 13B are diagrams showing the processing performed by the binary image generation unit 16.
Here, FIG. 13A is an image before image processing, and shows an original image G4 displayed on the display device 20 based on the image information acquired by the image information acquisition unit 11. The original image G4 is composed of an image of human hair, which is the foreground, and an image behind it, which is the background. And the boundary between the foreground and the foreground is ambiguous.
Further, FIG. 13B shows a binary image G4′ after the original image G4 is processed by the binary image generation unit 16.
Here, the image information acquired by the image information acquisition unit 11 is binarized by setting the foreground hair portion to, for example, “1” and the rearground portion to “0”, and the binary image G4. '

In order for the binary image generation unit 16 to binarize the image information, the processing performed by the pixel value prediction unit 14 in the first embodiment can be applied. That is, the binary image generation unit 16 sets the seed 1, which is the reference pixel, to the pixels belonging to the foreground. Further, the seed 2 which is the reference pixel is set to the pixel belonging to the background. Then, based on the seed 1 and the seed 2, each area is expanded by the area expansion method. As a result, the pixel value of the area to which the label 1 that is the foreground is set to "1", and the pixel value of the area to which the label 2 that is the foreground is set to "0".

The unknown area setting unit 13 uses the binary image G4' to set the unknown area M at the boundary between the foreground and the foreground. It can also be said that this resets the label given to the pixel at the boundary between the foreground and the foreground.

In the present embodiment, the unknown area setting unit 13 applies a filter that sets the unknown area M to the pixels in the binary image G4' where the adjacent pixels have different pixel values. Specifically, the unknown region M is set by applying a filter that cancels the setting of the binary pixel value (resets the label) around the pixel at the boundary between the foreground and the foreground.

FIGS. 14A and 14B are diagrams showing a process of setting the unknown region M by the unknown region setting unit 13.
Of these, FIG. 14A shows a filter for setting the unknown region M. This filter is set centering on a pixel whose adjacent pixels have different pixel values. In FIG. 14A, the center pixel is set as the center pixel, and a filter set for the center pixel is illustrated. The pixel value of the central pixel may be either "1" or "0". The size of the filter is, for example, 10 pixels×10 pixels.

FIG. 14B is a diagram showing the state after setting the unknown region M using this filter. As shown in FIG. 14B, the unknown region M is set at the boundary between the pixel values “1” and “0” shown in FIG. 14A.

FIG. 15 shows a case where an unknown region M is set at the boundary between the foreground and the foreground for the original image G4.
As shown in the figure, the unknown region M set by the unknown region setting unit 13 is applied to the boundary between the image of the human hair, which is the foreground, and the image behind it, which is the background.

The pixel value prediction unit 14 uses the image information to set reference pixels for pixels belonging to the foreground and pixels belonging to the background in the known region, and sets pixel values for pixels in the unknown region M to the respective pixel values. Predict about the case.

FIG. 16-1 is a conceptual diagram of a process performed by the pixel value prediction unit 14.
As shown in the figure, the pixel value of the pixel in the unknown region M is obtained in the case of predicting the pixel value from the foreground and the case of predicting the pixel value from the foreground, and two kinds of results are obtained.

16-2(a) to 16(d) are diagrams for specifically explaining this state.
For example, FIG. 16-2(a) shows a boundary that separates image regions. In addition, in FIG. 16-2(b), areas are separated by the binary image generation unit 16, and binary labels representing the foreground and the foreground are attached. Further, FIG. 16-2(c) shows the boundary of the unknown region M generated in FIG. 14, and FIG. 16-2(d) shows the pixel of the unknown region M in black.

Here, the present embodiment is different from the first and second embodiments in that the pixel value is propagated from only the foreground area and the pixel value is propagated from only the foreground area. There are two ways.

FIGS. 16-3(a) to 16(i) show a case where pixel values are propagated only from the foreground area.
In this case, the propagation of the pixel value and the strength is immediately performed (asynchronous type) by comparing (strength×weight) of the target pixel T and the adjacent pixel R. Further, the strength of the target pixel T is updated (passive type) by comparing (strength×weight) received from the adjacent pixel R.

Calculation of the weight in this case may be performed on the original image shown in FIG. 16-2(a), or may be performed after smoothing the original image shown in FIG. 16-2(a). Alternatively, it is possible to set some pixel value in the unknown region M of FIG.

16-3(a) to (i), only the pixels in the known area other than the unknown area M and belonging to the foreground have the strength 1. First, the target pixel T at the center of the frame shown in FIG. 16C is affected only by the adjacent pixel R3 to which the lower right foreground belongs, and the strength multiplied by the weight is propagated. Similarly, in FIG. 16C, the target pixel T at the center of the frame is affected by the adjacent pixel R4 on the left side and the adjacent pixel R5 on the lower side, and is updated to a pixel value having a stronger strength. In this case, the target pixel T does not yet have strength, so comparison of these two strengths is sufficient.
The target pixel T is sequentially selected from the pixels in the unknown region M, and the processes of FIGS. 16-3(a) to 16(i) are performed. In this example, as a result of the number of loops, the pixel values of the unknown region M are generally determined, and FIG. 16C(i) shows the result of the determination.
In the case of FIG. 16C, the pixels belonging to the foreground are not affected, and the pixels propagated are those belonging to the foreground.

FIG. 16-4(a) is a diagram similar to FIG. 16-3(i) and shows the state of the pixel values when the pixel values of the unknown region M are generally determined. Further, FIG. 16-4(b) shows a state of pixel values when the loop is further repeated and converged.
As described above, the pixel value from the foreground can be predicted for the pixels in the unknown region M.

16-5(a) to 16(i) show a case where the pixel value is propagated only from the foreground area.
The processes performed in FIGS. 16-5(a) to (i) are the same as those in FIGS. 16-3(a) to (i). However, in this case, the pixels belonging to the foreground are not affected, and the pixels propagated from the background are propagated.

FIG. 16-6(a) is a diagram similar to FIG. 16-5(i) and shows the state of the pixel values when the pixel values of the unknown region M are generally determined. Further, FIG. 16-6(b) shows the state of the pixel value when the loop is further repeated and converged.

FIG. 17A shows the result of obtaining the pixel value of the pixel in the unknown region M from the foreground. Further, FIG. 17B shows the result of obtaining the pixel value of the pixel in the unknown region M from the background.

Assuming that the pixel value of the image information (pixel value of the original image G4) in the pixel in the unknown region M is C, C can be expressed by the following formula 5.

Here, F is a pixel value predicted from the foreground. B is a pixel value predicted from the background. The C, F, and B pixel values can be represented by a vector in the RGB color space or the L ^* a ^* b ^* color space, and can be represented by a scalar in the case of gray scale.

Further, α is 0≦α≦1, and is a parameter indicating the foreground degree. That is, when α is 1, C=F and the pixel value of C becomes the same as the pixel value predicted from the foreground. Further, when α is 0, C=B, and the pixel value of C becomes the same as the pixel value predicted from the background. Therefore, it can be considered that the degree of the foreground increases as α increases.

That is, it is considered that the pixel at the boundary has a pixel value in which the pixel value predicted from the foreground and the pixel value predicted from the foreground are mixed at a certain ratio, and the pixel value C is expressed by the above-mentioned expression 5.

The multi-valued image generation unit 17 obtains a multi-valued pixel value representing the foreground degree α from the pixel values of the pixels in the unknown region M predicted by the pixel value prediction unit 14. That is, a multivalued pixel value representing the foreground degree α is obtained from the two types of pixel values predicted by the pixel value prediction unit 14.
By solving the equation 5 for α, the following equation 6 is obtained.

The multi-valued image generation unit 17 obtains the foreground degree α by using, for example, Formula 6, but the present invention is not limited to this. In the present embodiment, the pixel value of the unknown area M is calculated at high speed from the known pixel value by the area expansion method. Therefore, based on the calculated pixel values of F and B and the basic equation (5), the foreground degree α may be obtained by any method. As the simplest calculation, Expression 6 is given as an example here, but when the pixel value is represented by a vector, it may be difficult to accurately represent the pixel value of C by the foreground degree α. In that case, a method of finely adjusting the pixel values of F or B, a method of calculating an approximate solution, or the like may be applied.
Further, in the image shown in FIG. 17, the portions corresponding to the foreground and the foreground may be smooth, so that the prediction of the foreground and the foreground generated in FIGS. 16-4(b) and 16-6(b) is performed. A smoothed version of the resulting image may be used.

FIG. 18 is an image of the foreground degree α obtained by the multivalued image generation unit 17.
In this image G4″, the pixel value of the image of the human hair, which is the foreground, is “1”. The pixel value of the image behind it, which is the background, is “0”. Further, at the boundary between the foreground and the foreground, the numerical value is 0 to 1 represented by the foreground degree α. This image G4″ can be used as a mask that represents an ambiguous region at the boundary between the foreground and the foreground with multiple values. Then, the mask can cut out the foreground from the original image G4.

FIG. 19 is a flowchart illustrating the operation of the image processing device 10 according to the third embodiment.
The operation of the image processing apparatus 10 according to the third embodiment will be described below mainly with reference to FIGS. 2 and 19.

First, the image information acquisition unit 11 acquires RGB data as image information of an image to be subjected to image processing (step 201).

Next, the binary image generation unit 16 generates a binary image in which the image information is represented by a binary value indicating the foreground or the foreground (step 202). As a result, a binary image G4' as shown in FIG. 13B is generated.

Next, the unknown area setting unit 13 sets the unknown area M at the boundary between the foreground and the foreground using the binary image (step 203). This process is performed by using a filter for setting the unknown area M as shown in FIG. As a result, the unknown area M as shown in FIG. 15 is set.

Further, the pixel value prediction unit 14 uses the original image information to set seeds for pixels belonging to the foreground and pixels belonging to the foreground, and predicts pixel values for pixels in the unknown region M in each case. (Step 204). As a result, two kinds of results as shown in FIG. 17 are obtained.

Then, the multi-valued image generation unit 17 obtains a multi-valued pixel value representing the foreground degree α from the pixel values for the pixels in the unknown region M predicted by the pixel value prediction unit 14 (step 205). The foreground degree α can be obtained, for example, by using Expression 6.

According to the image processing device 10 described above, the process of predicting the pixel value in the unknown region whose pixel value is desired to be determined can be performed at higher speed.

Note that the processing performed by the pixel value prediction unit 14 described above acquires image information of an image, sets an unknown region M for which a pixel value is desired to be determined from the image, and is included in a known region other than the unknown region M. Pixels in the unknown region M are defined based on the strength of the reference pixel and the influence of the reference pixel on the target pixel, with the pixel having the strength belonging to the region as the reference pixel and the pixel included in the unknown region M as the target pixel. It can be understood as an image processing method for predicting the pixel value of.

<Example of hardware configuration of image processing device>
Next, the hardware configuration of the image processing apparatus 10 will be described.
FIG. 20 is a diagram showing the hardware configuration of the image processing apparatus 10.
The image processing device 10 is realized by a personal computer or the like as described above. As shown in the figure, the image processing apparatus 10 includes a CPU (Central Processing Unit) 91 that is a calculation unit, a main memory 92 that is a storage unit, and an HDD (Hard Disk Drive) 93. Here, the CPU 91 executes various programs such as an OS (Operating System) and application software. Further, the main memory 92 is a storage area for storing various programs and data used for execution thereof, and the HDD 93 is a storage area for storing input data to the various programs, output data from the various programs, and the like.
Further, the image processing apparatus 10 includes a communication interface (hereinafter, referred to as “communication I/F”) 94 for performing communication with the outside.

<Explanation of program>
The processing performed by the image processing apparatus 10 according to the present embodiment described above is prepared as a program such as application software.

Therefore, in the present embodiment, the processing performed by the image processing apparatus 10 includes an image information acquisition function for acquiring image information of an image and an unknown area setting for setting an unknown area M whose pixel value is to be determined from the image in the computer. The function and a pixel included in a known area other than the unknown area M and having a strength is set as a starting pixel, and the unknown area M is determined based on the strength of the starting pixel and the weight of the starting pixel on the pixels included in the unknown area. It can be understood as a program that realizes a pixel value prediction function of predicting the pixel value of a pixel inside.

It should be noted that the program that realizes the present embodiment can be provided not only by communication means but also by being stored in a recording medium such as a CD-ROM and provided.

Although the present embodiment has been described above, the technical scope of the present invention is not limited to the scope described in the above embodiment. It is apparent from the scope of the claims that the various modifications and improvements made to the above embodiment are also included in the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1... Image processing system, 10... Image processing apparatus, 11... Image information acquisition part, 12... User instruction acceptance part, 13... Unknown area setting part, 14... Pixel value prediction part, 15... Image information output part, 16... Two Value image generation unit, 17... Multi-valued image generation unit, 20... Display device, 30... Input device

Claims (11)

An image information acquisition unit that acquires image information of an image,
An unknown area setting unit that sets an unknown area whose pixel value is to be determined from the image,
A pixel having a numerical value indicating the magnitude of the possibility of belonging to a plurality of known regions included in a known region other than the unknown region as a starting pixel, and the strength of the starting pixel and the starting point Predict the pixel value of the pixel in the unknown area based on the numerical value indicating the closeness of the pixel value of the pixel of the starting point and the pixel included in the unknown area as a weight exerted on the pixel included in the unknown area of A pixel value prediction unit that
An image processing apparatus including. The pixel value prediction unit determines a pixel value of a pixel included in the unknown region based on the weighted on pixels included in the unknown region of the pixel intensity and the origin of the pixels of the starting point, pixel value The image processing apparatus according to claim 1, wherein the pixel value of the pixel in the unknown region is predicted by repeating the determination by using the determined pixel as a pixel of a new starting point. The image processing apparatus according to claim 2 , wherein the pixel value prediction unit sets a pixel value of a pixel which is a new starting point as a pixel value of the pixel of the starting point. The unknown region setting unit, an image processing apparatus according to any one of claims 1 to 3, characterized in that setting the unknown region in accordance with a user instruction. The unknown region setting unit, an image processing apparatus according to any one of claims 1 to 4, characterized in that to set the defective area is not in the original image as an unknown region. An image information acquisition unit that acquires image information of an image,
An unknown area setting unit that sets an unknown area whose pixel value is to be determined from the image,
A pixel having a numerical value indicating the magnitude of the possibility of belonging to a plurality of known regions included in a known region other than the unknown region as a starting pixel, and the strength of the starting pixel and the starting point A pixel value prediction unit that predicts a pixel value of a pixel in the unknown region based on a weighting of the pixel of the pixel included in the unknown region,
Equipped with
The pixel value prediction unit, based on the pixel value of the pixel of the starting point that the value obtained by multiplying the strength of the pixel of the starting point and the weight of the pixel of the starting point that affects the pixels included in the unknown images processing device for predicting a pixel value of a pixel in. An image information acquisition unit that acquires image information of an image,
An unknown area setting unit that sets an unknown area whose pixel value is to be determined from the image,
A pixel having a numerical value indicating the magnitude of the possibility of belonging to a plurality of known regions included in a known region other than the unknown region as a starting pixel, and the strength of the starting pixel and the starting point A pixel value prediction unit that predicts a pixel value of a pixel in the unknown region based on a weighting of the pixel of the pixel included in the unknown region,
A binary image generating unit, and the multi-level image generating unit, a Bei example,
The binary image generation unit generates a binary image represented by a binary value indicating the foreground or the foreground from the image information,
The unknown area setting unit uses the binary image to set the unknown area at the boundary between the foreground and the foreground,
The pixel value prediction unit uses the image information to set reference pixels for pixels belonging to the foreground and pixels belonging to the foreground in the known area, and sets pixel values for pixels in the unknown area. Make predictions for each case,
The multi-value image generation unit, images processing device and obtains the pixel values of the multi-level representative of the foreground degree from the pixel values for the pixels in the predicted the unknown regions by the pixel value prediction unit. 8. The unknown region setting unit sets the unknown region by applying a filter that cancels the setting of a binary pixel value centered on a pixel at the boundary between the foreground and the foreground. The image processing device described. Get the image information of the image,
Set an unknown region whose pixel value you want to determine from the image,
A pixel having a numerical value indicating the magnitude of the possibility of belonging to a plurality of known regions included in a known region other than the unknown region as a starting pixel, and the strength of the starting pixel and the starting point Predict the pixel value of the pixel in the unknown area based on the numerical value indicating the closeness of the pixel value of the pixel of the starting point and the pixel included in the unknown area as a weight exerted on the pixel included in the unknown area of Image processing method. A display device for displaying an image,
An image processing device that performs image processing on image information of the image displayed on the display device,
Equipped with
The image processing device,
An image information acquisition unit that acquires image information of the image,
An unknown area setting unit that sets an unknown area whose pixel value is to be determined from the image,
A pixel having a numerical value indicating the magnitude of the possibility of belonging to a plurality of known regions included in a known region other than the unknown region as a starting pixel, and the strength of the starting pixel and the starting point Predict the pixel value of the pixel in the unknown area based on the numerical value indicating the closeness of the pixel value of the pixel of the starting point and the pixel included in the unknown area as a weight exerted on the pixel included in the unknown area of A pixel value prediction unit that
An image processing system including. On the computer,
Image information acquisition function to acquire the image information of the image,
An unknown area setting function that sets an unknown area whose pixel value you want to determine from the image,
A pixel having a numerical value indicating the magnitude of the possibility of belonging to a plurality of known regions included in a known region other than the unknown region as a starting pixel, and the strength of the starting pixel and the starting point Predict the pixel value of the pixel in the unknown area based on the numerical value indicating the closeness of the pixel value of the pixel of the starting point and the pixel included in the unknown area as a weight exerted on the pixel included in the unknown area of Pixel value prediction function
A program that realizes.