JP2019125269A - Method and device suitable for foreground/background separation - Google Patents

Abstract

To enable highly accurate foreground/background separation by accurately determining propriety of foreground/background separation with a small amount of calculation.SOLUTION: A difference mask image generation part 12 generates a difference mask image Im by a background difference method. A background edge image generation part 13 extracts an edge from a background image to generate a background edge image Ib_edge. A difference mask boundary image generation part 15 extracts a boundary from the difference mask image Im to generate a difference mask boundary image Im_edge. A difference mask error boundary image generation part 16 extracts an error boundary overlapping the background edge image Ib_edge from the difference mask boundary image Im_edge to generate a difference mask error boundary image. A determination part 18 determines whether to execute foreground/background separation using the difference mask image Im on the basis of the error boundary. A reflection part 19 executes the foreground/background separation by using the difference mask image Im, and executes secret processing such as removal to at least one of a foreground and a background.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a method and apparatus suitable for foreground separation, which separates the foreground from the background of an input image taken by a fixed camera, and in particular, for foreground separation using a difference mask image generated by background subtraction. Method and apparatus

  When a home worker transmits his / her own video at a TV conference or the like, a technique for separating the background (home background) and the foreground (human body upper body) and concealing the home background for privacy protection is required. Furthermore, at that time, it is also required to be light processing and to accurately separate into a human shape in order to display a gesture.

  There is also a method of separating the foreground from the background using distance information obtained from the depth sensor, but from the image / video obtained from a popular optical camera without the depth sensor mounted, the foreground of the person is taken from the stationary background Several approaches to separation have been proposed.

  As a first approach, there is a method in which a face is detected by image processing, and the periphery thereof is cut out with a fixed shape human shape (a human upper body shape) or the like. In the case of a moving image, the foreground and the background can be separated by processing for tracking the feature points of the detected face, and re-detecting the face every predetermined time.

  As a second approach, there is a method of separating the boundary between the foreground and the background with high accuracy in combination with the area division such as the grab cut method. Non-Patent Document 1 discloses a method of detecting a face, using it as a foreground label, and separating it. Non-Patent Document 3 proposes an image matting technique in which separation with appropriate transparency is performed at the boundary between the foreground and the background, and the like, and fine details such as hair are retained.

  As a third approach, there is a method based on the background subtraction method in which the foreground and the background are separated by the difference between the background model prepared in advance and the input image.

  Among the above approaches, the third approach, the background subtraction method, has basic configuration of difference calculation processing and binary morphological calculation, and although it is the lightest processing, such as illumination condition, background, foreground color, etc. If conditions etc. are satisfied, the outline at the time of separation of the foreground and the background can be performed with high accuracy as the shape. The basic flow of the general background subtraction method is as follows.

  Step 1: First create a background model. The background model is created from the photographed still image and video. When the background is acquired as a moving image, the mean value and variance value over time can also be used to reflect background fluctuations such as noise and lighting, and the threshold value is changed dynamically by the “threshold processing” described later. It is also possible. Here, for the sake of simplicity, the background model is assumed to be a still image Ib (x, y). If it is a color, a background model can be created with each color component such as RBG and YCbCr.

Step 2: Next, the difference between the input image (image) Ia and the background image Ib is calculated for each pixel by the following equation (1).
Id (x, y) = Ia (x, y)-Ib (x, y) (1)

  For simplicity, the difference image Id here has an absolute value. If it is color, it can be processed with each color component such as RBG or YCbCr. Alternatively, it is also possible to measure a difference image Id in which each color component is integrated without obtaining a difference image for each color component.

Step 3: Next, the difference image Id is subjected to threshold processing of the following equation (2) to detect the foreground to generate a difference mask image Im.
Im (x, y) = 1; if | Id (x, y)> T (foreground) (2)
0; otherwise (background)

  When the difference image Id is obtained for each color component, one difference mask image Im can be created by applying the logical sum operation after threshold processing.

  Step 4: As an option, an opening and closing process is performed to remove sesame salt noise and the like.

  Step 5: As an option, micro area removal which is a connected component equal to or less than a designated fixed value is removed by an opening closing process.

  On the other hand, in the background subtraction method, (i) weak to illumination change, (ii) weak to non-static area of background (leaf fluctuation etc.), (iii) detection of target object (foreground) if background and foreground are similar colors Is difficult, there is a problem. The foreground detection omission when the background color is similar color can be corrected to some extent by the sesame salt noise removal processing and the micro area removal processing in the steps 4 and 5 described above, but the detection omission of a larger area than a certain level If the area or the like of the minute region to be removed is set large in step 5 in order to eliminate it, an erroneous determination occurs, and it is not appropriate as a correction when the detection leakage region is wider than a certain amount.

  In the background subtraction method, many problem solving methods in (i) and (ii) above are proposed, but it is said that (iii) can not be fundamentally solved, and there are few improvement methods, but There is an improvement method.

  Non-Patent Document 2 proposes a method in which background difference and color detection are integrated, and the accuracy may be improved as compared with the case where only background difference is used. Here, in the six-dimensional space of the foreground image (Yf, Uf, Vf) and the background image (Yb, Ub, Vb) having YUV gradations as axes, the identification plane of the background and the foreground (target) is obtained by prior learning. In the prior art, techniques for detecting the foreground are disclosed.

  In Patent Document 1, it is determined that a background difference is not suitable for a block having a certain area of overexposure or underexposure with respect to an input image (in this case, there are white or black areas in the background, and the foreground color It can be inferred that the similarity of background color is a problem), and when the background difference is not suitable, there is disclosed a technique of performing an object detection process by another method.

  In this patent document 1, an edge image of an input image is created, pattern matching is performed with a background edge image created in advance, and an object having a matched shape is regarded as a background. The area of the input image which is not matched and in which the outline of the object appears is detected as the object.

  Patent Document 2 discloses a technique that uses a background subtraction method to stably extract the outline of a moving object (in particular, a person). In this patent document 2, an edge image extracted using an outline extraction filter and an expanded foreground image obtained by performing expansion processing on an image determined to be foreground by background difference are created, and the logic between the edge image and the expanded foreground image By obtaining the product, the other edges are removed as unnecessary edges, and a mask edge image is created only from the correct edges.

  Furthermore, a foreground boundary image is generated by extracting the boundary between the mask edge image and the expanded foreground image, and the mask edge image and the foreground boundary image are synthesized to create an outline image. After that, follow the contours dynamically with the particle filter. As a result, the moving object can be tracked with higher accuracy than dynamic contour tracking, and foreground / background separation can also be performed.

  Non-Patent Document 4 discloses a technique of adjusting the foreground concealment level after foreground / background separation with an abstraction operator.

JP, 2014-157453, A JP, 2010-061478, A

Seon Heo, Hyung Il Koo, Hong Il Kim, Nam Ik Cho, "Human Segmentation Algorithm for Real-time Video-call Applications", APSIPA 2013. (DOI: 10.1109 / APSIPA.2013.6694320) Kato Takekazu, Wada Toshikazu, "Integration of background difference and color detection by nearest neighbor identification-Information integration based on examples", IPSJ Transactions Journal, Vol. 45, No. SIG13 (CVIM 10), pp. 110-117 , 2004. Eduardo S. L. Gastal 1 and Manuel M. Oliveira, "Shared Sampling for Real-Time Alpha Matting", Proceedings of Eurographics 2010, Number 2, pp. 575-584. Noboru Baba, "Image Processing Technology for Safety and Security: 5. Video Surveillance Considering Privacy," Information Processing, vol. 48, no. 1, pp. 30-36, Jan., 2007.

  In the first approach that does not use the background subtraction method, there is a problem that the outline of a person can not be cut out with a correct shape because it is a certain type of cutout assuming the shape of the person. In addition, gestures such as hand movements are considered as background and can not be transmitted to the other party. Furthermore, the problem may be that the amount of calculation is large compared to the background subtraction method.

  For example, in the case of working at home, it is considered that the specifications of a home PC or a PC distributed from a company are often low. A low spec PC may have difficulty in real-time processing or may degrade the performance of other processes in parallel. In addition, even if real-time operation is possible, it is conceivable that the PC is likely to have heat and the trouble is increased by operating at full-time for a long period of time such as full teleworking.

  The second approach solves the problem of the first approach 1 and enables separation with the correct contour shape. On the other hand, the amount of calculation is generally greater than that of the first approach 1. Therefore, in order to realize real-time processing, it is often necessary to use a PC with a certain specification or more, which enables parallel processing on GPUs.

  Also, even if real-time processing is possible, the performance of other processes in parallel work may be degraded as in the first approach. Long-term home work, etc., by operating for a long time for a long time, it is possible to have a problem that the PC is likely to have heat, failure increases.

  According to the basic background subtraction method of the third approach, in the case of similar colors in which the background and foreground colors are similar, the foreground is missed. Even if the threshold or color calculation method is changed, foreground leakage can not be corrected if the foreground and background belong to the same color space and there is no difference.

  As in Non-Patent Document 2, the background subtraction method and the foreground color detection method are integrated, and the approach for enhancing the detection accuracy for the foreground trained by the foreground color detection method allows more than the general background subtraction method. There is a possibility that the accuracy can be improved. However, training data acquisition work and learning are separately required. In addition, with regard to similar colors, it is also conceivable that foreground detection omission occurs because the identification surface can not be configured. Furthermore, the detection speed is also lower than the background difference, and there is also a problem that the PC is likely to have heat due to the long time operation, and the failure increases.

  In Patent Document 1, a portion where the input image and the edge image of the background image are matched is determined to be the background, so the determination of the foreground of the foreground can not be determined, and it is determined incorrectly as the background. In Patent Document 2, the main purpose is not to separate the foreground and the background, and as in Patent Document 1, it is not possible to judge the foreground detection omission.

  The object of the present invention is to solve the above-mentioned technical problems, and to evaluate whether the foreground can be separated lightly from the background of the input image taken by the fixed camera by the background subtraction method with high accuracy and further, the background It is an object of the present invention to provide a method and apparatus for properly separating the foreground from the background by the subtraction method.

  In order to achieve the above object, the present invention is characterized in that the method and apparatus suitable for foreground / background separation for separating the foreground from the background of the input image comprises the following configuration.

  (1) The present invention comprises means for acquiring a background image, means for generating a difference mask image by a background subtraction method, means for extracting an edge from the background image to generate a background edge image, and boundary from the difference mask image Means for extracting a difference mask boundary image, means for extracting an error boundary overlapping the background edge image from the difference mask boundary image, and foreground / background separation using the difference mask image based on the error boundary And means for determining the propriety of the above.

  (2) The present invention is characterized in that it further comprises reflecting means for performing foreground / background separation using the difference mask image in response to the appropriate determination by the determining means.

  (3) The present invention sets the “missing” area as the correction target based on means for selecting the “missing” area from the differential mask image and whether or not the position of the “missing” area corresponds to the error boundary. The present invention is characterized in that it comprises means for determining whether or not to do so, and means for correcting the difference mask image so that the "dropout" area to be corrected is the foreground.

  (4) The present invention is characterized in that it comprises means for selecting a "chip" area from a differential mask image based on an error boundary, and means for correcting the "chip" area.

  According to the present invention, the following effects are achieved.

  (1) A background shape that may be generated when the foreground border portion and the background edge portion are similar in background subtraction in which the difference between the background image acquired in advance and the input image is acquired and the foreground is separated from the background Since it is possible to estimate with a small amount of calculation whether there is a foreground detection omission that appears partially in the difference mask image, the appropriateness of foreground / background separation using the difference mask image Im generated by the background difference method is accurately determined by light processing. become able to.

  (2) When foreground / background separation using the difference mask image Im generated by the background difference method is determined to be appropriate, foreground / background separation is performed using the difference mask image Im, while it is determined that foreground / background separation is not suitable Then, switching to foreground / background separation by a method other than the background subtraction method makes it possible to realize the application of the optimum foreground / background separation method to the input image by light processing.

  (3) Even if a “dropout” area occurs in the differential mask image Im, it is possible to evaluate whether or not each “dropout” area is a foreground detection omission by referring to the erroneous boundary image Imb_edge. Therefore, in the difference mask image Im, it is possible to correct only the erroneous “dropout” region to the foreground by light processing while maintaining the original “dropout” region as the background.

  (4) Even if a “missing” region occurs in the differential mask image Im, it is possible to evaluate whether or not the “missing” region is a foreground detection failure by referring to the erroneous boundary image Img_edge. Therefore, an appropriate response to the "missing" area of the differential mask image Im is possible.

It is the block diagram showing the composition of the principal part of the device suitable for foreground background separation concerning a 1st embodiment of the present invention. FIG. 13 schematically shows a process of determining the suitability of foreground / background separation using the difference mask image Im generated from the frame image Ia by the background subtraction method in the first embodiment. It is the block diagram which showed the structure of the principal part of the apparatus suitable for foreground background isolation | separation which concerns on 2nd Embodiment of this invention. In 2nd Embodiment, it is the figure which showed typically the process of implementing foreground-background isolation | separation using the difference mask image Im produced | generated by the background subtraction method, and reflecting the isolation | separation result in at least one of a foreground and a background. It is the block diagram which showed the structure of the principal part of the apparatus suitable for foreground background isolation | separation which concerns on 3rd Embodiment of this invention. FIG. 17 is a diagram schematically showing a process of correcting a “missing” area of the difference mask image Im generated by the background difference method in the third embodiment. It is the block diagram which showed the structure of the principal part of the apparatus suitable for foreground background isolation | separation which concerns on 4th Embodiment of this invention. FIG. 21 is a diagram schematically showing a process of correcting a “missing” region of the difference mask image Im generated by the background difference method in the fourth embodiment. It is a figure explaining the "missing" field and the "missing" field which generate | occur | produce in a difference mask image. It is the figure which showed the evaluation method of the "chipping" area. FIG. 7 is a diagram showing an example in which a differential mask including a small area of “missing” is to be corrected. FIG. 7 is a diagram showing an example in which a differential mask including a large area of “missing” area is not to be corrected. It is the figure which showed the example which does not set the difference mask with many numbers of "missing" area | regions as correction | amendment object.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present invention, in particular, the omission of detection of foreground is detected from the following three viewpoints to evaluate the suitability of foreground / background separation by the background subtraction method, and also when applying the background subtraction method: To achieve light processing and high precision foreground / background separation.

  (1) A foreground detection omission which is difficult to process by removing a small area tends to occur in a flat area having a certain area, such as a hair part (black) or a cheek part (skin color) serving as the foreground or a clothes part having the same color as the background. Further, since the shape of the background object is likely to leak, the present invention is directed to the detection and correction of such foreground detection omission.

  (2) As a feature of the background, in the case of the house, the escape is often linear (posts, furniture, doors, calendars, etc.). In addition, non-linear omissions also exist in the house. For example, objects such as laundry and round figurines are not linear, but the missing shape in that case is often a closed region surrounded by the foreground having the same shape as the object.

  (3) Generally, home objects do not move. The outline of the foreground detection leakage area generated in the above (1) and (2) does not move even if the foreground (human body) moves a little.

  In the present invention, it is determined whether or not there is a foreground detection failure as described above, and if the simple correction is possible, the simple correction is performed, and if not, the switching to another method other than the background subtraction method is urged.

  FIG. 1 is a block diagram showing the configuration of the main part of a device suitable for foreground / background separation by the background subtraction method according to the first embodiment of the present invention, and FIG. 2 is a background of the input frame image Ia. It is the figure which showed typically the process of evaluating whether the foreground can be light-processed and separated with high accuracy by the background subtraction method.

  The frame image acquisition unit 10 acquires a moving image from the fixed camera Ca in units of frames. The background image storage unit 11 stores, in advance, a background image Ib obtained by photographing the room where no person is present with the fixed camera Ca. As shown in FIG. 2, the difference mask image generation unit 12 obtains the difference between the current frame image Ia acquired from the fixed camera CA and the background image Ib, and the foreground and the background of each pixel based on the pixel value. The initial differential mask image Im_t is generated by separating into two values of.

  As shown in FIG. 2, the background edge image generation unit 13 extracts an edge component from the background image Ib to generate a background edge image Ib_edge. The edge extraction can be realized by applying a Laplacian filter, a Sobel filter, a Canny edge filter or the like and performing general threshold processing used for edge detection. For example, if the absolute value of the pixel value after the application of the filter is equal to or more than the predetermined threshold A, the contour can be used. In addition, when the absolute value of the pixel value is between the thresholds A and B, if the adjacent pixel has a contour, it can be contoured, otherwise it can be non-contoured.

  The target image for edge extraction may be an image obtained by performing some pre-processing on the background image Ib to make a change. For example, a gray-scaled background image Ib may be adopted, or edge extraction is carried out for each color component such as YCbCr, and finally a single background edge image Ib_edge is generated by obtaining a logical sum. It is good.

  As shown in FIG. 2, the micro area removing unit 14 performs a removing process of salt and salt noise on the difference mask image Im_t or a micro area removing process of inverting the mask value of the area less than or equal to a predetermined value T_area_mp. , A noise-removed difference mask image Im is generated. Morphological domain openings can be applied to these processes. Here, if T_area_mp is set too large, the area that should be the background will be the foreground, so T_area_mp should not be set too large, and for large closed areas, either the foreground or the background and all There are many cases where it is better to judge the condition separately. The determination method will be described below.

  As shown in FIG. 2, the difference mask boundary image generation unit 15 extracts an edge boundary from the difference mask image Im after the micro area deletion to generate a difference mask boundary image Im_edge. As a method of generating a difference mask boundary image, (i) using the boundary where the foreground and the background of the mask change as the boundary of the difference mask, (ii) performing edge extraction, and (iii) opening (or opening) the foreground of the difference mask It is possible to adopt a processing such as closing, making the boundary before and after the opening (closing) processing. Alternatively, the logical sum of boundary images subjected to both the opening process and the closing process may be acquired. However, when the foreground is in contact with the side of the image, it is desirable to use that portion as the boundary.

  Also, with regard to the area, in order to reduce the influence of illumination flicker and the texture of the object, a difference mask image or a difference mask boundary image of a fixed period may be accumulated and then determined. For example, a logical product of differential mask images for a predetermined time may be calculated to obtain an area, or only an area where the foreground is present at a predetermined rate or more during a predetermined time may be obtained as a foreground area. Furthermore, the difference mask boundary image Im_edge may be a logical sum of a difference mask boundary image for a predetermined time, or a boundary existing at a predetermined ratio or more in a predetermined period.

  The difference mask error boundary image generation unit 16 specifies the position where the shapes of the difference mask boundary image Im_edge and the background edge image Ib_edge are similar. In the present embodiment, as shown in FIG. 2, the difference mask error boundary image Imb_edge is generated based on the logical product of the difference mask boundary image Im_edge and the background edge image Ib_edge. In addition, if the similar position of each image is extracted by the logical product operation, it is desirable to perform an opening process on the differential mask boundary image Im_edge in advance as needed.

  The noise removing unit 17 removes noise components from the pixel group of the error border remaining as the difference mask error border image Imb_edge. In the present embodiment, the number of pixels that can be connected out of the pixel group at the error boundary is counted based on the consideration that it is highly likely that the edge is in contact with the foreground detection leakage region. As a result, a pixel group having fewer connectable pixels than a predetermined number T_cnct is likely to be an intersection between the correct edge of the foreground and an edge of the background, so it is removed as noise while pixel groups greater than the fixed number T_cnct It can not be removed because it can be estimated as the original "error boundary".

  Such noise removal may be performed in consideration of not only the correlation within the frame but also the correlation between the frames. For example, in the difference mask error boundary image Imb_edge generated in the previous frame, at a location where a connected component of T_cnct or more is present, even if noise or minute motion occurs in the current frame and T_cnct or less, a certain number of edges are T_cnct If it is above, it may be regarded that the connected component in the current frame is connected to T_cnct or less as in the previous frame.

  Also, in order to eliminate the intersection of the correct boundary of the foreground and the background edge or the coincident line, the edge image Ia_edge is also obtained by the edge extraction filter for the input frame image Ia, and the difference mask error border image Imb_edge is obtained. It is also possible to remove the image included in the edge image Ia_edge out of the pixel group of the error boundary that exists.

  At this time, it may be determined over time whether or not the edge image Ia_edge and the edge of the difference mask error boundary image Imb_edge coincide by chance. For example, a minute movement of the foreground over time is detected, and if there is a line movement that does not match after a minute movement, processing is not considered as coincident by coincidence, and a location where the outlines match over time The correction may be performed only in the area in contact with the image. For detection of a minute movement, for example, it is possible to determine that there is a movement when a difference image with the previous frame is acquired and a change of a certain level or more occurs.

  In the present embodiment, as described above, the noise removing unit 17 removes various noises including temporal noise from the difference mask error boundary image Imb_edge to generate a difference mask error boundary image Imb_edge ′.

  The determination unit 18 determines the suitability of foreground / background separation using the difference mask image Im generated by the background subtraction method based on whether or not an error boundary remains in the difference mask error boundary image Imb_edge ′ after the noise removal. judge.

  In the present embodiment, for the present frame image Ia, if no error boundary remains in the difference mask error boundary image Imb_edge ', the foreground / background separation using the difference mask image Im generated by the background difference method is determined to be suitable. . On the other hand, if an error boundary remains in the difference mask error boundary image Imb_edge 'after noise removal, it is determined that foreground / background separation using the difference mask image Im generated by the background difference method is unsuitable.

  According to the present embodiment, it is possible to estimate with a small amount of calculation whether there is a foreground detection omission in which a part of the background shape appears in the difference mask image, which may be caused by similarity between the foreground border and the background edge. As a result, it is possible to lightly determine the suitability of foreground / background separation using the difference mask image generated by the background subtraction method for each frame image.

  Therefore, if the determination result of the determination unit 18 is notified to the system having various foreground / background separation functions including the background subtraction method via the operator or directly, the foreground / background separation suitable for each input frame image As a method, either one of background subtraction and background separation can be accurately selected.

  FIG. 3 is a block diagram showing the configuration of the main part of a device suitable for foreground / background separation by the background subtraction method according to the second embodiment of the present invention, and FIG. 4 is a background area of the input frame image Ia. It is judged whether the foreground area can be separated with light processing and high accuracy by the background subtraction method, and if it is judged that the foreground area can be separated, the foreground area is separated from the background area using the difference mask image generated by the background subtraction method. Is a diagram schematically showing the process of The same reference numerals as above denote the same or equivalent parts in any of the drawings, so the description thereof is omitted.

  If the error boundary does not remain in the differential mask error boundary image Imb_edge ′ after the noise removal, the determination unit 18 determines that foreground / background separation by the background difference method is appropriate, and notifies that effect to the reflection unit 19. . On the other hand, if an error boundary remains in the differential mask error boundary image Imb_edge 'after the noise removal, foreground separation by the background difference method is stopped, and separation by a method other than the background difference method is recommended.

  In response to the notification of the appropriateness determination from the determination unit 18, the reflection unit 19 performs foreground / background separation on the current frame image Ia using the difference mask image Im generated by the background subtraction method, From the foreground and the background, a concealment process such as "blur", "mosaic", "replacement (to another texture)" or "removal" is performed on at least one from the viewpoint of privacy protection. If the concealment process is performed on any of the foreground and the background, it is preferable to perform different concealment processes on each of the separation boundaries so as not to be lost.

  According to the present embodiment, when it is determined that foreground / background separation can be performed with high accuracy and high accuracy by using the difference mask image created by the background difference method, this determination result is reflected on the current frame image Ia. On the other hand, when it is determined that the background subtraction method is not suitable, foreground separation by other than the background subtraction method is promoted, so that the optimum foreground / background separation can be realized according to the input image.

  FIG. 5 is a block diagram showing the configuration of the main part of an apparatus suitable for foreground / background separation by the background subtraction method according to the third embodiment of the present invention, and FIG. 6 is a difference mask image generated by the background subtraction method. It is the figure which showed typically the process of evaluating and correct | amending the "missing" area | region of Im. In any of the drawings, the same reference numerals as described above denote the same or equivalent parts, and thus the description thereof will be omitted.

  In this embodiment, a correction candidate area selection unit 20 selects a closed area of the difference mask image Im as a correction candidate based on the erroneous boundary image Imb_edge, and a difference mask image based on the selection result and the determination result by the determination unit 18. The first correction processing unit 21 is further provided to correct Im and output a corrected differential mask image Im1.

  The reflection unit 19 performs foreground / background separation on the frame image Ia using one of the difference mask image Im and the corrected difference mask image Im1, and performs the “blurring” on at least one of the foreground and the background. , “Mosaic”, “replacement” or “removal” etc.

  The correction candidate area selection unit 20 recognizes a background area [FIG. 9 (a)] surrounded by the foreground in the difference mask image Im as the “missing” area, and corrects all “missing” areas as correction candidates. The area is listed up and notified to the first correction processing unit 21.

  The first correction processing unit 21 corrects only the “dropout” region in contact with the erroneous boundary on the error boundary image Img_edge ′ after the noise removal on the difference mask image Im, and sets the mask value in the “dropout” region. By inverting and correcting from the background to the foreground, a corrected difference mask image Im1 is generated. The main correction area may be effective for a predetermined period (between a plurality of frames).

  As a result, as shown in FIG. 6, while the mask value of the “dropout” area candidate Re where the image group of the erroneous boundary exists on the difference mask image Im is corrected to the mask value of the foreground, the pixel group of the erroneous boundary The mask values of the "missing" region candidate Rt in which there is no exist maintain the background mask value without being corrected to the foreground mask value. The reflection unit 19 performs foreground / background separation by applying the corrected difference mask image Im1 to the current frame image Ia, and performs concealment processing on at least one of the foreground and the background (in this embodiment, the background). Carry out).

  According to the present embodiment, even if a “dropout” area occurs in the difference mask image Im, each “dropout” area can be accurately evaluated by referring to the erroneous boundary image Img_edge ′. It is possible to correct only the "missing" region based on false detection to the foreground while maintaining the region as the background.

  FIG. 7 is a block diagram showing the configuration of the main part of a device suitable for foreground / background separation by the background difference method according to the fourth embodiment of the present invention. FIG. 8 is a view schematically showing a process of correcting the “missing” area [FIG. 9 (c)] of the difference mask image Im generated by the background difference method. In any of the drawings, the same reference numerals as described above denote the same or equivalent parts, and thus the description thereof will be omitted.

  In the present embodiment, a “missing” area is selected from the difference mask image Im1 after the correction based on the pixel group of the error boundary on the error boundary image Imb_edge ′ after the noise removal, and further, as necessary, the “missing” region is selected. The second embodiment is characterized in that the second correction processing unit 22 is provided to evaluate and correct the "missing" area.

  In the second correction processing unit 22, the lack area determination unit 221 determines whether or not there is a “missing” area in the differential mask image Im <b> 1 after the first correction processing. In the present embodiment, in the area of the difference mask image Im1 which is the foreground in the correct difference mask image [FIG. 9 (b)] but is erroneously determined to be the background, an area in which only a part of the periphery is surrounded by the foreground [FIG. (c)] is a "missing" area. In the determination that there is no “missing” area, for example, the pixel group of all error boundaries on the error boundary image Imb_edge ′ is in contact with the “dropout” area, and has already been corrected by the first correction processing unit 21. Then, since there is a high possibility that the "missing" area does not exist, it is determined here that the "missing" area does not exist.

  On the other hand, as shown in FIG. 10, if there is an erroneous border pixel group [the same figure (a)] not in contact with the “dropout” area on the erroneous border image Imb_edge ′, It is determined that the "missing" area may exist. If it is determined that there is a possibility that there is a missing area, the “omission” correction process may not be performed, and it may be recommended to adopt another method other than the background subtraction method as the foreground / background separation method.

  Furthermore, if the shape feature of the foreground is known or inferred, it can be determined based on the consistency with the shape feature whether there is a high possibility of the "missing" region. For example, the foreground shape is an oval shape like a face, and as shown in FIG. 6B, it has a convex shape in which the foreground shape is smooth in the portion where the pixel group of the erroneous boundary of the erroneous boundary image Imb_edge 'is present. In the case of a concave shape or a complicated shape, it can be determined that there is a high possibility that the "chip" area is present.

  The foreground shape can also be inferred from the boundary of the current difference mask image Im. In that case, contour extraction is performed on the frame image Ia to obtain a differential mask image of FIG. Here, since the boundary coincident with the mask image boundary can be inferred to be correct, several points on the boundary are extracted, and the shape is estimated by interpolating the curve by spline interpolation or the like. Note that even if it is determined that there is a possibility that there is a missing area, it is possible to interrupt the missing correction if it is difficult to estimate the foreground shape because the foreground shape has many irregularities.

  Next, as shown in FIG. 6D, the contour of the estimated shape and the difference mask image are superimposed. At this time, when there is a background area surrounded by the outline and the foreground, it can be determined that the possibility of the "missing" area is high. Furthermore, if a part of the contour of the area coincides with the end point of the erroneous boundary on the erroneous boundary image Imb_edge ', it can be determined that the possibility of the "missing" region is higher.

  In this manner, when it is recognized that the "chip" area may be present or high, the second correction processing unit 22 first determines whether or not the "chip" area can be left. . For example, although there is a pixel group at an erroneous boundary not in contact with the already corrected “dropout” area, if the shape is not inconsistent with the foreground shape, or as shown in FIG. When the amount is smaller than the predetermined reference area, it is determined that the chip may be left as it is, and the difference mask image Im1 having the “chip” can be adopted as it is.

  On the other hand, when there is a "chipping" that can not be left, the area of the "chipping" area as shown in FIG. 12 or the number of "chipping" areas as shown in FIG. 13 is further determined. Then, if the area of the “chip” area or the number of “chip” areas exceeds the predetermined determination threshold, it is recommended to adopt another system other than the background subtraction method as the foreground / background separation system. Alternatively, if the number of pixels and the number of connected components counted as one for the number of pixels and the number of connected components of the erroneous border image Imb_edge that may be in contact with the “missing” area, the foreground / background separation method is used as the background. It is recommended to adopt other methods other than the difference method.

  Alternatively, if the area or the number of "missing" areas exceeds a predetermined determination threshold, the mask correction unit 222 performs correction on the difference mask image Im1 to replace the pixel values of the "missing" area with pixel values of the foreground. The difference mask image Im2 may be acquired, and the background may be deleted using the difference mask image Im2.

  Alternatively, the frame image Ia is divided into a plurality of blocks, and a mask image Im3 created by another method other than the background subtraction method is adopted only for the block including the edge pixel and further only for the area to which the adjacent block is added. It is good. That is, the difference mask image Im2 based on the background difference method and the mask image Im3 based on other than the background difference method are applied in a hybrid manner within one screen.

  Furthermore, as an option, methods such as the first approach and the second approach, fitting of a fixed mask shape for fitting a part of an ellipse, or the like as described in Patent Document 2 to the mask correction unit 222 Features such as dynamic contour tracking may be added.

  In addition, when generating another mask for combination, there are cases where a mask is generated for the entire image or a mask only around the correction area is generated. For example, in the case of face detection in the first approach, after creating a mask for the entire image, it is also possible to synthesize the mask for only the missing part of the difference mask image Im1, and fitting or active contour tracking In this case, even if a mask is created for the entire image, a mask may be created only around the “missing” area and then combined with the difference mask image Im1.

  In addition, processing such as changing the method of calculating the difference image Id of the background difference or reducing the threshold value is also effective for the block to be corrected without using another method. When reducing the threshold, it is used together with failure determination described later, or it is confirmed whether or not the foreground area is increased in the image immediately after the difference acquisition, and the threshold is reduced to such an extent that failure or increase in the number of foreground regions does not occur. It is also effective to do. Another method may be applied after changing the threshold. The correction determination changes with time, and can be applied as needed only to the region where the background difference is appropriate, by the “missing” correction determination and the mask composition.

  The reflecting unit 19 performs foreground / background separation on the frame image Ia using any of the difference mask images Im, Im1, Im2, etc., and performs predetermined concealment processing (in FIG. 8, at least one of the foreground and the background). Perform background removal).

  According to the present embodiment, when the “missing” area is generated in the difference mask image Im, the effect of each “notching” area is evaluated by referring to the erroneous boundary image Imb_edge ′, and the evaluation result is obtained. It is possible to take appropriate measures such as selection other than the background subtraction method, and allowance or correction of the "missing" area. Therefore, the appropriateness of foreground / background separation using the difference mask image generated by the background subtraction method can be determined with light processing and with high accuracy, and further appropriate concealment processing reflecting the result of foreground / background separation can be performed. become.

  In the second to fourth embodiments described above, the reflecting unit 19 performs foreground / background separation on the frame image Ia using the difference mask image Im, and “blurs” on at least one of the foreground and the background, Although the concealment processing such as "mosaic", "replacement" or "removal" has been described, the present invention is not limited to this and various methods using foreground / background information that can be acquired from difference mask information Application to various image processing is possible.

In Non-Patent Document 4, the concealment level of the foreground after foreground / background separation can be adjusted by the abstraction operator, and it is considered that making the foreground “transparent” has the highest abstraction degree (high concealment level). In the abstraction operator, “dot”, “bar”, “box”, “silhouette”, “edge”, “edge”, “mosaic”, “blur”, “monotone”, “see-through” in descending order of the degree of secrecy. In the case of "real shooting", the degree of secrecy is the lowest.
If it is desired to conceal both the foreground and the background but different levels of concealment are required, different processes can be applied to the foreground and the background. For example, when the background is slightly concealed at a low level and a person in the foreground wants to display only a silhouette and conceal it at a high level, the background is “blur”, the foreground is “silhouette”, or the like.

  Furthermore, when it is not desired to apply image processing to the foreground area, it can also be used for applications such as superimposing a balloon and a text message on the background area. Further, in the failure detection described later, it can be judged from the ratio of the number of pixels of the background area and the foreground area, but it may be used for processing using such foreground or background statistical information.

  In addition, when the determination unit 18 checks the presence of a foreground detection defect, failure detection is simultaneously performed, and when a failure is detected, it is determined that the background difference method is not applied to the current frame image Ia. Thus, it is possible to further optimize the use timing of the background difference method. Failures often occur due to changes in lighting conditions and the like, and failure detection can be determined, for example, when the area of the foreground region is rapidly increased to a certain value or more within a certain period of time.

  Further, in order to further optimize the use timing of the background difference method, it may be used in combination with other foreground detection omission correction. For example, if the foreground is a person with a white mask for a cold and the background wall is white, the background difference is likely to cause the omission of the white mask shape. In such a case, the mask face is detected at the time of activation, and if it is detected, the accuracy can be enhanced by using both the detection of a worn object such as a mask and the correction processing of the "dropout" region.

  DESCRIPTION OF SYMBOLS 10 Frame image acquisition part 11 Background image storage part 12 Difference mask image generation part 13 Background edge image generation part 14 Minute area deletion part 15 Difference mask boundary image generation part 16 Difference mask Error boundary image generation unit 17 17 noise removal unit 18 determination unit 19 reflection unit 20 correction candidate area selection unit 21 first correction processing unit 22 second correction processing unit

Claims (15)

In a device suitable for foreground background separation, which separates the background and the foreground of the input image
Means for acquiring a background image,
A means for generating a difference mask image by the background difference method;
Means for extracting an edge from the background image to generate a background edge image;
A means for extracting a boundary from the difference mask image to generate a difference mask boundary image;
Means for extracting an erroneous boundary overlapping with the background edge image from the differential mask boundary image;
A device suitable for foreground / background separation, comprising: means for determining the suitability of foreground / background separation using the difference mask image based on the erroneous boundary.   The apparatus according to claim 1, further comprising reflection means for performing foreground / background separation using the difference mask image in response to the appropriateness determination by the determination means.   The apparatus according to claim 2, wherein the reflecting means switches to foreground / background separation methods other than the background subtraction method in response to the improper determination by the determination means.   The apparatus according to claim 2, wherein the reflection unit performs a concealment process on at least one of the foreground and the background.   The apparatus according to claim 4, wherein the reflection means performs different concealment processing on each of the foreground and the background.   The apparatus suitable for foreground / background separation according to claim 4 or 5, wherein the concealment process is any of blur, mosaic, replacement and removal processes.   An apparatus suitable for foreground / background separation according to any one of claims 1 to 6, wherein said means for extracting an erroneous boundary finds the logical product of a difference mask boundary image and a background edge image. A means for selecting a "missing" area from the difference mask image;
A unit that determines whether or not the “dropout” area is to be corrected based on whether or not the position of the “dropout” area corresponds to the erroneous boundary;
A device suitable for foreground / background separation according to any one of claims 1 to 7, further comprising: means for correcting the difference mask image so that the "dropout" area to be corrected is the foreground.   The means for determining whether or not to make the correction object is characterized in that if an error boundary exists at a position corresponding to the boundary position of the “dropout” region, it is regarded as a correction object, and if not, it is not regarded as a correction object A device suitable for foreground / background separation according to claim 8. Means for selecting a "missing" region from the difference mask image based on the error boundary;
A device suitable for foreground / background separation according to claim 8 or 9, characterized in that it comprises means for correcting said "missing" area.   When an erroneous boundary exists at a position other than the position corresponding to the boundary position of the “dropout” region, the means for selecting the “dropping” region sets the position on the differential mask image corresponding to the position as the “lost” region. An apparatus suitable for foreground / background separation according to claim 10, characterized in that it is estimated. The means for correcting the "chip" area comprises means for evaluating the size of the "chip" area,
According to the evaluation result, any one of correction of the “chip” area, acceptance of the “chip” area, and switching to foreground / background separation by a method other than the background subtraction method is adopted. Or a device suitable for foreground / background separation according to 11.   The foreground / background separation unit according to claim 12, wherein the means for evaluating the size of the "chip" area evaluates the size based on at least one of the area and the number of the "chip" areas. apparatus. In a method suitable for foreground background separation which separates the foreground from the background of the input image
The steps to get the background image,
Generating a difference mask image by the background difference method;
Extracting an edge from a background image to generate a background edge image;
A step of extracting a boundary from the difference mask image to generate a difference mask boundary image;
Extracting an erroneous boundary overlapping the background edge image from the differential mask boundary image;
And e. Determining the appropriateness of foreground / background separation using the difference mask image based on the error boundary.   15. The method according to claim 14, further comprising: performing foreground / background separation using the difference mask image in response to the determination that foreground / background separation using the difference mask image is appropriate. Preferred method for foreground background separation.

Images