JP2013143069A - Image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such a problem that, when a person moves an installed object in a monitoring space, a change area integrated with the installed object is generated on a monitoring image and, if the change area is uniformly processed for protecting privacy, it is hard to visually recognize the shapes of the person and a dolly.SOLUTION: An image processing apparatus causes a storage unit 3 to store a background image 30 including an installed object. A change area extraction unit 40 obtains a change area between the background image 30 and an input image. An installation change detecting unit 42 collates the background image 30 with the input image to obtain an installed object-derived area generated by moving the installed object in the change area. A mask image generation unit 44 excludes the installed object-derived area from an object of a masking process when performing a masking process for the input image in the change area.

Description

  The present invention relates to an image processing apparatus that performs a privacy protection process on a monitoring image.

  In order to display a person or the like in a monitoring image with privacy protection processing, it is necessary to extract a person area in the image. Conventionally, a change area is extracted by a difference process between a background image and an input image, and privacy protection processing is performed using the change area as a person area.

JP 2009-225398 A

  However, if a person moves an arrangement that originally existed in the monitoring space such as a chair or a carriage, the person and the arrangement are detected as a group of change areas. For this reason, when the change area is uniformly processed with privacy, it is difficult to visually recognize the shape of the person or the carriage, and it is difficult for the observer to grasp the situation such as the presence or absence of suspicious behavior. In this case, in order for the monitoring person to grasp the situation, it is necessary to take a procedure for confirming the monitoring image before the processing, and the monitoring performance is deteriorated. In this case, there is a risk that privacy may be impaired by checking the monitoring image before processing. In addition, since the arrangement differs for each monitoring space, it is difficult to store the arrangement information in advance in order to detect the area of the arrangement.

  The present invention has been made in view of the above problems, and an object thereof is to provide an image processing apparatus capable of obtaining a monitoring image that can achieve both privacy protection and monitoring performance.

  An image processing apparatus according to the present invention sequentially inputs input images captured in a monitoring space, and performs mask processing for privacy protection on the input images, and an arrangement placed in the monitoring space A storage unit that stores a background image of the monitoring space in which a background is captured; a change region extraction unit that obtains a change region between the background image and the input image; and the input image in the change region in the input image A mask processing unit that performs a mask process, and an arrangement change area specifying unit that compares the background image with the input image and obtains an arrangement-derived area generated by movement of the arrangement among the change areas. The mask processing unit excludes the arrangement-derived region from the mask processing target.

  In another image processing apparatus according to the present invention, the arrangement change region specifying unit detects corresponding points having similar image characteristics between the background image and the input image, and moves between the corresponding points. An amount is calculated, and the arrangement-derived region is detected from the position of the corresponding point where the movement amount is a predetermined value or more.

  In addition, the arrangement change area specifying unit is configured to detect the arrangement-derived area generated at the movement destination of the arrangement from the position of the corresponding point on the input image side. Can do.

  Furthermore, the arrangement change area specifying unit can detect the arrangement-derived area generated at the movement source of the arrangement from the position of the corresponding point on the background image side.

  Further, the arrangement change area specifying unit registers the background image of the arrangement-derived area generated at the movement source as a template of the arrangement, and the input image is subjected to matching processing using the template after the registration. It is also possible to detect the arrangement-derived region in.

  According to the present invention, even when a person in a monitoring space moves an arrangement, a monitoring image that can easily grasp the situation of the monitoring space while protecting the privacy of the person is obtained.

1 is a block diagram illustrating a schematic configuration of an image monitoring apparatus according to an embodiment of the present invention. It is a schematic diagram explaining the process of a background flow detection part. It is a flowchart which shows the operation | movement of the outline of the image monitoring apparatus which concerns on embodiment of this invention. It is a general | schematic flowchart of a background flow detection process. It is a typical image which shows the example when the change area of a movement origin and the change area of a movement destination are the same, and different. It is a schematic diagram of the example of the change area extracted corresponding to the input image in three continuous times which the change area of the arrangement under tracking appears. It is a schematic diagram explaining the process in which a mask image is produced | generated by the image monitoring apparatus which concerns on embodiment of this invention using a specific example.

  Hereinafter, an image monitoring apparatus 1 according to an embodiment of the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings.

  In the monitoring space, there are articles that are arranged in advance and can be moved by the person (hereinafter referred to as an arrangement) in addition to the person entering and exiting. For example, the monitoring space is a room, and the arrangement is equipment or facilities in the monitoring space, or temporarily placed items. As an example, a chair or a carriage is the arrangement.

  The arrangement object is normally stationary and forms a part of the background of the monitoring space, and a change area due to the arrangement object is not extracted normally. However, if the placement object is moved by a person, a change area in which the placement object and the person are integrated is extracted. The image monitoring apparatus 1 performs an image processing process suitable for monitoring and privacy protection by separately detecting an arrangement-derived area generated by the movement of the arrangement from the change areas.

[Configuration of Image Monitoring Apparatus 1]
FIG. 1 is a block diagram illustrating a schematic configuration of the image monitoring apparatus 1. The image monitoring apparatus 1 includes an imaging unit 2, a storage unit 3, a control unit 4, and an output unit 5. Among these, the control unit 4 is an image processing apparatus according to the present invention, and the imaging unit 2, the storage unit 3, and the output unit 5 are connected to the control unit 4.

  The imaging unit 2 is a surveillance camera and photographs the surveillance space at predetermined time intervals. The captured surveillance images (input images) of the surveillance space are sequentially input to the control unit 4. In order to grasp the position and movement of an arrangement object or a person using a background image, the imaging unit 2 is installed with a field of view fixed at a height at which a bird's-eye view can be taken. For example, in this embodiment, the imaging unit 2 is fixedly installed on the ceiling of the room. The time interval at which the monitoring image is captured is 1/5 second, for example. Hereinafter, the unit of time recorded at the imaging time interval is referred to as time, the latest time is referred to as the current time, the input image at the current time is referred to as the current image, and the input image one hour before the current time is referred to as the immediately preceding image.

  The storage unit 3 is a storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 3 stores various programs and various data, and inputs / outputs such information to / from the control unit 4. Various data includes a background image 30, a previous image 31, arrangement information 32, and area information 33.

  The background image 30 is an image that includes only the background image of the monitoring space and does not include the human image. The background image 30 is generated prior to the mask image generation process and stored in the storage unit 3. Specifically, an input image captured in a state where a moving object such as a person does not exist in the monitoring space can be used as the background image 30, and the background image 30 can include an image of an arrangement object. The background image 30 is appropriately updated during the mask image generation process.

  The immediately preceding image 31 is an input image one hour before the current time, and the current image input at each time is written in the storage unit 3 as the immediately preceding image 31 at the next time.

  The arrangement information 32 includes the identification number of each arrangement, the state of the arrangement (arrangement state), the position and area of the arrangement in the monitoring image, and the image characteristics of the arrangement. When the arrangement is moved and the arrangement is detected, arrangement information 32 of the arrangement is created. While the arrangement is moving, the image monitoring apparatus 1 tracks the position of the arrangement, and the position history is stored as the arrangement information 32. Tracking and completion of movement are defined as attributes indicating the arrangement state. The image feature includes a feature point representing a characteristic part of the image of the arrangement, in addition to a color histogram representing the characteristics of the entire image of the arrangement.

  The area information 33 includes an identification number of each change area and information indicating whether or not the change area includes an arrangement.

  The control unit 4 functions as an image processing apparatus that sequentially receives input images captured in the monitoring space and generates an image including a concealed portion by mask processing as will be described later. The control unit 4 is configured by using an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit), and reads and executes a program from the storage unit 3. , Function as a background flow detection unit 41, an arrangement change detection unit 42, a background image generation unit 43, a mask image generation unit 44, and the like.

  The change area extraction unit 40 compares the input image and the background image, extracts a change area between them, and outputs information on the extracted change area to the background flow detection unit 41 and the arrangement change detection unit 42. The change area can be extracted by a known background difference process or background correlation process. That is, the degree of difference (difference or correlation value) of the pixel values at the same coordinates between the input image and the background image is calculated, and a pixel whose degree of difference exceeds the threshold value is detected as a change pixel. Then, change pixels adjacent to each other are collected, and the combined area is extracted as a change area. The detection threshold is set in advance based on a prior experiment or the like. Note that an area of about 1 to several pixels is excluded from the extraction target because it is noise.

  The extracted change area includes a change area caused by an arrangement change of the arrangement object and a change area caused by the presence and movement of a person. While the arrangement change of the arrangement by the person is in progress, a change area in which the arrangement and the person are integrated is extracted. Further, as a change area (arrangement derived area) derived from the movement of the arrangement, an area (movement) where the arrangement exists in the background image 30 in addition to an area (movement destination area) where the arrangement exists in the current image Original area) is extracted. Hereinafter, the movement source region that is the trace of the image of the arrangement object among the change areas derived from the arrangement object is referred to as a ghost. Here, at the initial stage of the arrangement change, a change area in which the arrangement object, the person, and the ghost are integrated can be extracted.

  The background flow detection unit 41 and the arrangement change detection unit 42 function as an arrangement change region specifying unit that compares the background image with the current image and specifies an arrangement-derived region from among the change regions extracted by the change region extraction unit 40. To do. In the present embodiment, collation is performed by detecting corresponding points whose image features are similar to each other.

  The background flow detection unit 41 detects corresponding points having similar image characteristics between the background image 30 and the current image, and detects a background flow that is a motion vector between the corresponding points. Both the background image 30 and the current image include an image of an arrangement, and when the arrangement moves, a movement amount (motion vector) corresponding to the movement of the arrangement is calculated between corresponding points of the background image 30 and the current image. Is done. At this time, the current image includes the image of the person who is moving the arrangement, but the background image 30 does not include the image of the person, and thus the background flow of the person is basically not detected. Therefore, if the movement amount obtained by the background flow detection unit 41 is used, the presence / absence of the arrangement change can be determined with high accuracy, and the arrangement-derived region can be specified from the position of the corresponding point. Note that when there is no change in arrangement, only a movement amount of an error that can be regarded as 0 or 0 is calculated.

  A motion vector calculated between input images is generally called an optical flow, but a background flow is different from a general optical flow in that it is calculated between a background image 30 and an input image. Hereinafter, a general optical flow calculated between the immediately preceding image 31 and the current image is referred to as an input flow in distinction from the background flow.

  Specifically, the background flow detection unit 41 sets a plurality of attention points on one image (reference image) of the background image 30 and the current image, and each attention point and image in the other image (comparison image). The most similar point having the most similar feature is detected, and a vector (motion vector) connecting the attention point and the most similar point is calculated. As an image feature, a block having a predetermined size (for example, 5 × 5 pixels) centering on each of the target point and the most similar point candidate is set, and the color distribution, luminance gradient, or light / dark distribution in the block is used. it can. Incidentally, the corresponding point on the background image 30 side is the start point of the motion vector, and the corresponding point on the current image side is the end point.

  In view of the purpose of the arrangement change area specifying unit, the range in which the attention point is set is preferably not the entire image but the part where the image change has occurred, that is, the inside of the change area obtained by the change area extraction unit 40. . Thereby, errors that erroneously detect background flows other than the arrangement object are reduced, and the processing amount can be reduced.

  The background flow detection unit 41 calculates a background flow and outputs the background flow to the arrangement change detection unit 42 every time a current image is input from the imaging unit 2.

  In order to specify the shape of the arrangement-derived area in detail, it is desirable to set each pixel as a point of interest and detect a spatially dense background flow (dense background flow). If it detects by, processing load will become large. It is sufficient to specify the shape in detail only when an arrangement change occurs, and the presence or absence of the arrangement change can be detected even in a background flow (sparse background flow) having a lower density than the pixel unit. The sparse background flow can be detected by setting a feature point on the reference image as a point of interest.

  That is, the background flow detection unit 41 detects a sparse background flow by extracting a feature point from the reference image at each time and setting the feature point as a point of interest for detection of a change in arrangement, and the arrangement-derived region Therefore, when a change in arrangement is detected, each pixel is set as a point of interest and a dense background flow is detected. As a result, the processing load for detecting corresponding points is reduced at a time when no change in the arrangement is detected. Note that a blob can be used as the feature point, and various other feature points such as a corner and a subpixel corner can be used.

  The background flow detection unit 41 can be configured to perform a search process on the comparison image to detect the most similar point. For this search, various search methods such as the steepest descent method such as the Lucas-Kanade method can be applied.

  Here, as described above, the input image at the time of moving the arrangement includes a human image, and the human image on the input image has characteristics similar to a part of the background image 30, A background flow may be erroneously detected between a part of the background and a person.

  On the other hand, the current image at the start of the movement of the arrangement and the immediately preceding image 31 both include the same person image, and it can be expected that the same person image is more similar to the person and the background. Therefore, compared to the background flow, the motion vector is less likely to be erroneously detected between the person and the background in the input flow.

  Focusing on this property, the background flow detection unit 41 of this embodiment includes an input flow detection unit 410 that detects an input flow, and a background flow selection unit 411 that selects a background flow that matches the input flow. By outputting the flow, false detection of background flow due to human influence is reduced. Hereinafter, these will be described.

  The input flow detection unit 410 detects corresponding points having similar image characteristics between the immediately preceding image 31 (preceding input image) and the current image (subsequent input image), which are successive input images. Specifically, among the current image and the immediately preceding image 31, a plurality of attention points are set on one image, and the most similar point whose feature is most similar to each attention point in the other image is detected. A motion vector connecting the most similar points is calculated and used as an input flow. Incidentally, the corresponding point on the immediately preceding image 31 side is the start point of the motion vector, and the corresponding point on the current image side is the end point. The calculated input flow is output to the background flow selection unit 411.

  Note that the attention point setting and the most similar point detection can be performed in the same manner as in the background flow detection described above. That is, the input flow detection unit 410 extracts a feature point from one image at each time and detects the feature point as an attention point in order to detect a low-density input flow (sparse input flow) used for arrangement change detection. Each pixel is set as a point of interest when an arrangement change is detected in order to detect a high-density input flow (dense input flow) that is set and used for specifying an arrangement-derived region. At this time, the input flow detection unit 410 can set the attention point only inside the change area obtained by the change area extraction unit 40 in order to reduce the processing amount. Then, the input flow detection unit 410 performs a search process on the other image to detect the most similar point.

  The background flow selection unit 411 selects a background flow that matches the motion vector (input flow) detected by the input flow detection unit 410.

  Here, the background flow is detected separately from the input flow detection, and the background flow obtained by this can be determined to match the input flow. However, in this method, the background flow is detected. In addition, a similar point search process with a large amount of processing must be performed.

  Therefore, focusing on the fact that the background flow to be detected is included in the input flow and that the current image is common among the background flow and the original data of the input flow, the background flow selection unit 411 of the present embodiment selects the background. The flow selection processing method is devised to reduce the processing load. Specifically, the background flow selection unit 411 includes the image feature of the immediately preceding image 31 at the start position of each input flow detected by the input flow detection unit 410 and the image feature of the background image 30 at the position. It is determined whether or not they match, and a motion vector corresponding to the input flow determined to match is determined as the background flow. That is, the start and end of the background flow are defined based on the start and end of the input flow. Considering that a minute change due to illumination fluctuation or noise may occur between the immediately preceding image 31 and the background image 30, a slight deviation (for example, for one pixel) at the start end is allowed and the coincidence determination is performed.

  The arrangement change detection unit 42 detects the start of movement of the arrangement, detects the area derived from the arrangement, and tracks the arrangement based on the background flow or the input flow detected by the background flow detection unit 41.

  The arrangement change detection unit 42 determines that the arrangement has started to move, that is, that an arrangement change has occurred when a sparse background flow having a movement amount, that is, a vector size equal to or greater than a preset motion detection threshold Ts is detected. . This determination is performed at each time. Here, Ts is a value for excluding a background flow of an error level detected by a minute illumination variation, and can be set to a size corresponding to a few pixels, for example. In addition, in order to increase the detection accuracy, when conditions of direction, number, and position are added and a movement amount is equal to or greater than Ts and a sparse background flow in the same direction is detected from a predetermined number Tn (for example, 10) or more from the same change area It may be determined that the arrangement has started moving.

  When the movement start of the arrangement is detected, the arrangement change detection unit 42 refers to the dense background flow detected by the background flow detection unit 41 and arranges the position of the corresponding point whose movement amount is equal to or greater than the motion detection threshold Ts. It is detected as an object-derived region. As described above, the motion detection threshold value Ts is set in advance to a value that can exclude the background flow having an error level. Furthermore, the placement change detection unit 42 refers to the dense input flow detected by the background flow detection unit 41 during the tracking of the placement object, and detects the position of the corresponding point as the placement object-derived region. For these reasons, the arrangement change detection unit 42 detects the arrangement object area detection unit 420 that detects the arrangement object area (movement destination area) in the arrangement object-derived area and the ghost (movement source area) in the arrangement object-derived area. And a ghost detection unit 421 for detecting.

  When the start of movement of the arrangement is detected, the arrangement object detection unit 420 arranges the position of the corresponding point on the current image side of the dense background flow whose movement amount is equal to or greater than the movement detection threshold Ts, that is, the position of the end. It is detected as an arrangement object area that is a movement destination area of an object. As described above, the corresponding points are detected in the change region. Therefore, a collection of pixels at the end of the dense background flow, which is a part of the change area, is detected as an arrangement area at the start of arrangement movement. In addition, if the size or orientation on the image changes, such as when the placement object moves forward, a gap is generated between the end points, so that the placement region detection unit 420 performs a filling process (expansion process and further contraction process) on the collection of end pixels. ). Then, a new identification number and an attribute “tracking” are assigned to the detected arrangement area and registered in the arrangement information 32.

  Further, when the arrangement object is being tracked, the arrangement area detection unit 420 extracts a dense input flow obtained from the background flow detection unit 41 that has a start in the arrangement area detected in the immediately preceding image. A set of terminations detected in the change area in the current image for the extracted dense input flow is detected as an arrangement area at the current time. At that time, a zero-vector dense input flow is also used for detection in order to detect the area of the arrangement object that is stopped, and in order to eliminate a gap between the terminal ends, a filling process is performed on the collection of terminal ends.

  As described above, the arrangement object area and the person area can be separated and detected from the change area in which the arrangement object area and the person area are integrated.

  The ghost detection unit 421 detects a ghost based on the position of the corresponding point on the background image side of the background flow. There are two types of processing of the ghost detection unit 421: processing when the ghost is in a mixed state integrated with the arrangement object region and the person region, and processing when the ghost is separated from the arrangement object region and the person region.

  When the beginning and end of the sparse background flow are in the same change region, the ghost is in a mixed state. In this case, the ghost detection unit 421 detects the position of the corresponding point on the background image side of the dense background flow whose movement amount is equal to or greater than the motion detection threshold value Ts, that is, the position of the start end as a ghost. Further, in order to eliminate a gap between the start ends, a filling process is performed on the gathering of the start ends. Thus, only the ghost can be detected separately from the change area in which the ghost is integrated with the arrangement object area and the person area.

  On the other hand, when the start end and the end of the background flow are separated in different change areas, the ghost is in a separated state. In this case, the ghost detection unit 421 detects, as a ghost, the position of the corresponding point on the background image side of the sparse background flow whose movement amount is equal to or greater than the motion detection threshold Ts, that is, the change area where the start edge is detected. That is, in the separated state, it is not necessary to use the dense background flow, and the ghost can be detected without applying a load only with the sparse background flow.

  The arrangement tracking unit 422 associates an arrangement area detected from the immediately preceding image and a change area detected from the current image as an area including the same arrangement if they are connected by a sparse input flow. Note that a zero-vector sparse input flow is also used for association in order to associate the stopped arrangement.

  Also, the arrangement tracking unit 422 compares the positions and shapes of the change areas including the same arrangement, and determines that the arrangement has been moved within the field of view when the positions and shapes are substantially the same. In this case, an attribute of “movement completed within the field of view” is given as the arrangement state of the arrangement information 32, and the arrangement is excluded from the tracking target. Whether or not they are the same position / shape can be determined based on, for example, the degree of overlap between the two change regions, and the degree of overlap can be thresholded, and the case where the degree of overlap is high can be made the same position / shape.

  On the other hand, when the arrangement area corresponding to the arrangement area detected from the immediately preceding image is not detected from the current image, the arrangement tracking section 422 determines that the arrangement has been moved out of the field of view. In this case, an attribute of “moving out of the field of view” is given as the arrangement state, and the object is not tracked. In this case, an abnormal signal indicating the take-out may be output to the output unit 5.

  FIG. 2 is a schematic diagram for explaining the processing of the background flow detection unit 41 and the arrangement change detection unit 42. Based on the specific examples of the background image 200, the immediately preceding image 210, and the current image 220, detection of the background flow and the arrangement-derived region is performed. Processing is shown.

  The background image 200 includes a chair 201 (image 201a) as an arrangement, and the background image 200 includes a background object 202 (image 202a) that can cause erroneous detection.

  In addition to the chair 201 and the background object 202, the person 203 is shown in the immediately preceding image 210 and the current image 220. The person 203 has a portion having a color similar to that of the background object 202. In the immediately preceding image 210, the arrangement has not changed yet, the position of the chair 201 (image 201b) is the same as that of the background image 200, and the person 203 (image 203b) is not yet in contact with the chair 201. On the other hand, the current image 220 is a state in which a person 203 (image 203c) is in contact with the chair 201, and the chair 201 (image 201c) has started to change its arrangement. The background object 202 (image 202c) remains in its original position.

  The schematic images 230 and 240 show the background flow and the input flow with arrows, respectively. The images 230 and 240 show the outlines of the change areas between the current image 220 and the immediately preceding image 210 and the background image 200, respectively. Specifically, the change area 231 in which the chair 201 and the person 203 are integrated is extracted from the current image 220 as the change area, and the ghost 232 of the chair 201 is extracted from the background image 200. Further, the change area 233 of the person 203 is extracted from the immediately preceding image 210.

  In the background flow shown in the image 230, in addition to the background flow 234 caused by the movement of the chair 201, an erroneous background flow 235 is detected between the chair 201 and the person 203. An incorrect background flow 236 has been detected.

  In the input flow shown in the image 240, an input flow 241 due to the movement of the chair 201 and an input flow 242 due to the movement of the person 203 are detected. Here, since the degree of similarity between the people 203 is higher than the degree of similarity between the chair 201 and the person 203, an input flow corresponding to the erroneous background flow 235 is not detected. Further, since the degree of similarity between the people 203 is higher than the degree of similarity between the background object 202 and the person 203, an input flow corresponding to the erroneous background flow 236 is not detected.

  Incidentally, the input flow 241 of the chair 201 and the input flow 242 of the person 203 are detected in substantially the same direction and size at positions close to each other. Therefore, it is difficult to distinguish the input flow of the arrangement from the human input flow. Therefore, it is difficult to specify the arrangement-derived region only from the input flow.

  A schematic image 250 shows the background flow shown in the image 230 that matches the input flow shown in the image 240, and the background flow is detected by the background flow detection unit 41. That is, the background flow 234 caused by the movement of the chair 201 in the background flow shown in the image 230 is in common with the input flow 241 caused by the movement of the chair 201 shown in the image 240, and is selected as the background flow 251. On the other hand, erroneous background flows 235 and 236 shown in the image 230 are excluded because they are not in common with the input flow of the image 240.

  The arrangement object region is specified from the end position of the background flow 251 selected in this way, and the ghost is specified from the start end position.

  Note that, as described above, the background flow and the input flow can be generated separately, and the common flow of them can be selected as the background flow. However, the background flow selection unit 411 of the present embodiment has the configuration of the input flow. Of these, a configuration in which the starting edge has the same image characteristics as the same position of the background image 200 is selected as the background flow, and the processing load associated with the background flow detection is reduced. With this configuration, the processing load for obtaining the background flow 234 is reduced, and waste of detecting erroneous background flows 235 and 236 that are motion vectors that do not occur in the input flow is eliminated.

  Here, since the background object 202 does not change the position and brightness and does not generate a change region, the background flow and the input flow are separately generated and the common flow is used as the background flow. If the above-described processing is performed to limit the search range for similar points to the change region, erroneous detection of the background flow 236 can be prevented.

  The background image generation unit 43 registers the unmanned input image as the background image 30 as an initialization process when the image monitoring apparatus 1 is activated. The background image generation unit 43 functions as a background image update unit that appropriately updates the background image 30 using the input image during the operation of the image monitoring apparatus 1. Specifically, for areas other than the change area between the current image and the background image 30, the background image 30 is replaced with the current image, or the background image 30 and the current image are updated by weighted averaging. Can do.

  Further, the background image generation unit 43 as the background image update unit updates the ghost region in the background image 30 where the arrangement object is absent in the current image with the current image. Specifically, when completion of movement of the arrangement is detected, the arrangement area in the background image 30 is replaced with the current image and updated. Thus, after the movement is completed, the arrangement object area can be prevented from being extracted as a change area. Further, when it is determined that the ghost area is in a separated state in the current image, the ghost area in the background image 30 is replaced with the current image and updated, and thereafter, the area is not extracted as a change area.

  In addition, in the update process of the background image 30 related to the arrangement object region described above, the arrangement image may not be included in the background image 30, but once the arrangement change is detected, the arrangement image is distinguished from the background. Since the position and shape are grasped, it is possible to continue processing such as detection of a change in arrangement and detection of an arrangement object area using the information. On the other hand, the update may be performed so that the image of the arrangement object at the movement destination is included in the background image 30. In this case, the process such as the detection of the arrangement change and the detection of the arrangement area is continued in the same manner as after the initialization process. The

  The mask image generation unit 44 performs mask processing (filter processing) on a change area (processing area) excluding an arrangement object area and a ghost (that is, an arrangement origin area) in the current image for the purpose of protecting the privacy of a person shown in the input image. This is a mask processing unit that generates an image (mask image) that has been processed to include and outputs the image to the output unit 5. By the way, the person in the image cannot be identified by processing to conceal the image, but if the shape and movement of the concealed part are known, the observer should grasp suspicious behavior based on it Can do.

  Here, when grasping the action, it may be difficult to visually recognize the shape and movement of the person because the person and the arrangement are integrated in the change area before the area derived from the arrangement. For this reason, in order to ascertain the presence or absence of suspicious behavior, the monitoring person needs a procedure for confirming the input image before processing. For example, the monitoring performance is deteriorated and the privacy of the subject is also impaired.

  On the other hand, since the placement object originally existed in the monitoring space, it is clear that the placement-derived area does not need to be processed for privacy protection. It becomes difficult to grasp the situation.

  In this respect, since the mask image generation unit 44 uses the change area excluding the arrangement-derived area as the processing target area, the mask image can visually recognize the shape and movement of the person as described above, and also the appearance of the arrangement. it can. Therefore, the monitoring person can grasp the situation only with the mask image, and both privacy protection and monitoring can be achieved.

  The mask processing is performed by generating a composite image obtained by transparently combining a single-color image having the same shape as the processing target region and the background image 30 of the processing target region, and replacing the processing target region in the current image with the composite image. Do. Further, instead of transparently synthesizing the background image, it is also possible to perform processing such as painting a processing target area in the current image with a single color, painting with a texture, or performing a blurring process on the processing target area in the current image.

  Here, at the edge of the image of the arrangement object, the pixel is affected by the quantization, and a pixel in which the color of the arrangement object and the background color are mixed, a noise pixel, or a shadow pixel of the arrangement object is generated. Since it tends to be extracted as a part of the change area, the change area is often extracted a little wider. Therefore, it is preferable to perform the contraction process a predetermined number of times on the area excluding the arrangement-derived area from the change area and then perform the same number of expansion processes to generate the processing target area. The number of times of the contraction process and the expansion process may be set to about 1 to several times, and is set to 2 in this embodiment.

  The output unit 5 includes a sound output unit such as a speaker or a buzzer that outputs a warning sound in response to an abnormal signal, a display unit such as a liquid crystal display or a CRT that displays a privacy-processed composite image, and the like.

[Operation of the image monitoring apparatus 1]
FIG. 3 is a flowchart showing a schematic operation of the image monitoring apparatus 1.

  The image monitoring apparatus 1 is activated in a state where there is no person in the monitoring space, and the imaging unit 2 starts an operation of imaging the monitoring space every predetermined time. The background image generation unit 43 acquires or generates an image in which no person is captured from an input image captured during a preset initialization period, and initializes the background image 30 of the storage unit 3 with the image (S1). For example, the background image generation unit 43 generates the background image 30 by averaging the input images at a plurality of times in the initialization period.

  After the initialization period, the control unit 4 starts a monitoring operation using the input image acquired from the imaging unit 2. The control unit 4 sequentially receives input images (hereinafter, the current image) (S2), and repeats the processes of steps S2 to S16 each time the image is input.

  The change area extraction unit 40 compares the current image sequentially input with the background image 30, and extracts a change area in the current image (S3). The control unit 4 determines whether or not a change region has been extracted (S4). If the change region is not extracted (in the case of “NO” in S4), the control unit 4 omits steps S5 to S14 and performs processing. To step S15. In this case, the background image generation unit 43 updates the entire background image 30 by weighted averaging with the current image (S15), the control unit 4 outputs the current image as it is to the output unit 5, and the output unit 5 outputs the current image as it is. Is displayed (S16).

  On the other hand, when a change area is extracted (in the case of “YES” in S4), the control unit 4 advances the process to step S5 and performs an arrangement change detection process (S5).

  The background flow detection unit 41 detects a sparse input flow by the input flow detection unit 410, and sparses a sparse input flow in which the image feature of the background image 30 matches the image feature of the immediately preceding image 31 by the background flow selection unit 411. Detect as background flow. The arrangement change detection unit 42 refers to the sparse background flow and checks whether a movement amount is equal to or greater than Ts and a sparse background flow in the same direction is detected from the same change area to Tn or more. If it is detected, it is determined that an arrangement change is detected. At the same time, the change areas on the start and end sides of the group of the sparse background flow are set as the movement source and the movement destination of the arrangement, respectively.

  The arrangement change detection unit 42 has an arrangement in which movement of a new arrangement has been detected in the arrangement change detection processing S5 or is being tracked (an arrangement in which movement has been detected by the previous time and tracking has not ended). In this case, it is determined that the arrangement is changing (in the case of “YES” in S6).

  Here, if there is no arrangement subject to arrangement change (in the case of “NO” in S6), the control unit 4 omits steps S7 to S13 and performs the processes of steps S14 to S16. In this case, all the change areas are processed as person areas. That is, the mask image generation unit 44 performs mask processing on all the changed regions, the background image generation unit 43 updates the background image 30 other than the change regions by weighted averaging with the current image, and the control unit 4 performs the mask processing. The image is output to the output unit 5, and the output unit 5 displays the masked image.

  When an arrangement being changed in step S6 is detected, the arrangement tracking unit 422 checks whether or not there is an object being tracked (S7). If there is (in the case of “YES” in S7), the arrangement is tracked to detect a change area corresponding to the current position (S8). On the other hand, if there is no tracked object ("NO" in S7), the process of step S8 is omitted. In addition, the arrangement | positioning object by which the movement start was detected in step S5 will be tracking after the next time, and is not made into the object of tracking process S8 at the present time.

  Next, in order to specify the arrangement-derived region in the change region, the background flow detection unit 41 performs background flow detection processing S9. FIG. 4 is a schematic flowchart of the background flow detection process S9. The background flow detection process will be described below with reference to FIG. The input flow detection unit 410 of the background flow detection unit 41 refers to the change region of the previous time stored as the region information 33 and the change region of the current time extracted in step S3, and within the change region of the previous time or the current time Each pixel of the immediately preceding image 31 that is within the change area is set as a point of interest (S101).

  The input flow detection unit 410 searches the current image for the most similar point having an image feature most similar to the image feature of the target point set in the immediately preceding image 31 (S103). Then, a vector starting from the attention point and ending at the most similar point is detected as a dense input flow (S104). A zero vector is also detected as a dense input flow. In addition, when only the most similar point with a similarity equal to or less than the threshold value is found in step S103, it can be assumed that the dense input flow corresponding to the point of interest is not detected.

  The background flow selection unit 411 selects a dense background flow that matches the dense input flow. Specifically, a predetermined deviation allowable range is set around the start coordinates of each dense input flow detected in step S104 (S105), and the immediately preceding image 31 is noticed in the background image 30 within the deviation allowable range. An attempt is made to detect a coincident point with the point (S106). The allowable shift range can be set to 3 × 3 pixels, for example, and a shift within one pixel is allowed.

  When a matching point is detected in the background image in step S106, the background flow selection unit 411 detects a vector having the matching point as a starting point and the corresponding most similar point as a terminal as a dense background flow (S107). On the other hand, when the attention point does not match the background image within the deviation allowable range, the dense background flow corresponding to the dense input flow starting from the attention point is not selected. Note that if the start of movement is not detected at the current time, steps S104 to S107 can be omitted.

  When the dense background flow is detected for the arrangement whose movement start is detected by the background flow detection process S9 described above, the arrangement area detection unit 420 of the arrangement change detection unit 42 terminates the dense background flow whose movement amount is Ts or more. Is detected as the arrangement area at the current time (S10). Further, for the arrangement being tracked, the arrangement area detection unit 420 reads the arrangement area at the immediately preceding time from the arrangement information 32 of the arrangement, and the area of the dense input flow detected in step S104. A collection of terminal ends of the dense input flow starting from the pixel is detected as an arrangement area at the current time (S10).

  The arrangement change detection unit 42 determines separation / mixing of the change area of the arrangement in the current image with a person change area or a ghost (S11). For the arrangement in which the movement start is detected at the current time, the arrangement change detection unit 42 confirms whether the movement source change area set in step S204 is the same as or different from the movement destination change area set in step S205. Based on the result, separation / mixing determination is performed.

  FIG. 5 is a schematic image showing an example in which the movement source change area and the movement destination change area are the same or different. FIG. 5A shows a case where the movement source area 500 and the movement destination area 501a are the same change area. The movement amount of the arrangement object whose movement start is detected is small, and the person and the arrangement object (movement destination in the current image). A change area in which the area 501a) and the ghost (movement source area 500) are integrated is extracted. The arrangement change detection unit 42 determines the change area of the arrangement in this case as “a mixed area with a person” and “a mixed area with a ghost”.

  On the other hand, FIG. 5B shows a case where the movement source area 500 and the movement destination area 501b are different change areas, and a person and an arrangement (movement destination area 501b) and a ghost (movement source area 500) in the current image. ) Are extracted as separated change areas. The arrangement change detection unit 42 determines the change area of the arrangement in this case as a “mixed area with people” and a “separated area with ghost”.

  For the arrangement being tracked, the arrangement change detection unit 42 checks whether or not the shape of the change area at the immediately preceding time and the change area at the current time matched in step S8 match. Based on this, separation / mixing judgment is performed.

  FIG. 6 is a schematic diagram of an example of an input image at three consecutive times in which a change area of an arrangement being tracked is reflected, and an example of a change area associated with the arrangement in the image. The time advances in the order of (c), and in each of the diagrams (a) to (c), the left side is an input image and the right side is an image showing a change area. When the shape of the change area associated with the arrangement does not match the previous time and the current time, for example, the person and the arrangement shown in FIG. Immediately after the person and the arrangement are separated. In this case, the arrangement change detection unit 42 does not immediately determine the separation, but maintains the determination of the change area of the arrangement as the “mixed area with people”. On the other hand, when the shape of the change area associated with the arrangement matches the previous time and the current time, for example, when the state is changed from the state shown in FIG. 6B to the state shown in FIG. In this way, the person and the arrangement are separated at both the previous time and the current time, and can be said to be a stable separation state. In this case, the arrangement change detection unit 42 determines that the change area associated with the arrangement object is a “separated area from a person”.

  The ghost detection unit 421 performs the process of detecting the ghost region described above for an arrangement whose movement start has been detected at the current time and an arrangement that is being tracked and for which ghost separation has not been determined. S12). If the movement start of the arrangement is detected at the current time, the ghost detection unit 421 refers to the dense background flow detected in step S10 and detects the set of attention points as a ghost.

  If it is determined in step S7 that the arrangement is changing, the arrangement change detection unit 42 performs the processes in steps S8 to S12 described above, and updates the arrangement information 32 and the area information 33 based on the result (S13). . Specifically, in step S10, the arrangement area newly detected at the current time is registered in the arrangement information 32 together with the new identification number and the attribute being tracked. Also, the information on the arrangement area in the arrangement information 32 of the arrangement being tracked is updated with the arrangement area of the arrangement detected at the current time. For an arrangement for which completion of movement out of the field of view is detected in S9, the arrangement state of the arrangement information 32 is changed to an attribute "completion of movement out of field of view". Further, the separation / mixing information of the arrangement information 32 of the corresponding arrangement is updated with the separation / mixing determination result of step S12, and the attribute of the arrangement for which “separation area with human” is determined in step S12 is displayed. Change to “Completed within view”.

  When the arrangement area and the ghost in the change area are detected by the above processing, the mask image generation unit 44 excludes the arrangement area and the ghost from the change area, and then performs the contraction process twice on the area after the exclusion, Furthermore, an expansion process is performed twice to generate a processing target area. Subsequently, a single color image having the same shape as the processing target area and the background image 30 of the processing target area are transparently synthesized to generate a composite image, and the processing target area in the current image is replaced with the composite image to replace the mask image. Generate (S14). In addition, when the arrangement object area | region is not detected, the process which excludes an arrangement object area | region is abbreviate | omitted. If no ghost is detected, the process of excluding the ghost is omitted.

  When the generation of the mask image is completed for the current image from which the change area is extracted, the background image generation unit 43 excludes the ghost whose separation is determined and the arrangement area of the arrangement where the movement completion is detected from the change area. The update target area is calculated, and the part of the update target area of the background image 30 is replaced with the current image in the update target area (S15). If there is no arrangement for which movement completion is detected, the process of excluding the arrangement area is omitted. If there is no ghost determined to be separated, the process of excluding the ghost is omitted.

  Further, the background image generation unit 43 updates the background image 30 other than the change area by performing a weighted average with the current image.

  The control unit 4 outputs the mask image to the output unit 5 when the mask image is generated by the mask image generation unit 44, and outputs the current image to the output unit 5 when the mask image is not generated. The output unit 5 displays the image input from the control unit 4 on the monitor (S16).

  When the above processes S2 to S16 are completed for the current image, the control unit 4 replaces the immediately preceding image 31 with the current image and also replaces the area information 33 with the information of the changed area extracted in step S3. Return to step S2.

  FIG. 7 is a schematic diagram illustrating a process of generating a mask image by the image monitoring apparatus 1 using a specific example. FIG. 7A shows a current image 600, which is an image obtained by capturing a state immediately after the person 601 moves the chair 602. An image 610 shown in FIG. 7B is an image showing a change area extracted from the current image 600. A region 611 in which the person 601 exists at the current time shown in the current image in FIG. 7A, a region in which the chair 602 exists, and a ghost of the chair 602 are extracted as an integrated change region 611. Yes. If the shape of the change area 611 is masked, it is difficult to visually recognize that the person 601 is moving the chair 602. Note that the boundary portion of the image is a pixel in which the chair 602 and the person 601 are mixed with the background due to the influence of quantization, and a shadow is generated below the chair 602 and the person 601, so the change region 611 is extracted slightly wider. The

  In an image 620 shown in FIG. 7C, a ghost 621 (shaded portion) and an arrangement object region 622 (vertical line portion) detected from the change region 611 are shown. FIG. 7D schematically shows a mask image 630. The image 630 protects the current image 600 with privacy by a mask 631 formed in a region excluding the ghost 621 and the arrangement region 622 from the change region 611. It is an image. The area (see FIG. 7C) in which the change area 611 is extracted slightly wider and remains thin around the chair 602 (see FIG. 7C) is removed in FIG. 7D by the contraction process and the expansion process. The current image is exposed in the area of the chair 602, the shape of the person 601 is easily visible, and the relationship between the chair 602 and the person 601 is also easily visible. On top of that, the image of the person 601 is masked to protect privacy.

[Modification]
As mentioned above, although embodiment of this invention was described, the said embodiment can be changed and it can also be set as other embodiment of this invention. Examples thereof will be described below.

  The arrangement change detection unit 42 holds the arrangement object area (movement destination area) registered in the arrangement object information 32 as a template without deleting the registration, or uses the ghost part (movement source area) of the background image as a template of the arrangement object. For an arrangement that has been registered in the arrangement information 32 and has been detected after registration, the arrangement area detection unit 420 can detect the movement destination area of the arrangement in the input image by matching processing using the template. .

  Further, it is possible to use the detection result of the arrangement change for person tracking. For example, a change area other than the arrangement-derived area can be tracked as a person area. In addition, until a change area determined to contain an arrangement is detected until a separation between the person and the arrangement is detected, the change area is characterized by a feature quantity (a feature quantity in which the person and the arrangement are mixed), During this time, there is a method in which the feature amount of the person alone is saved and switching to tracking by the feature amount of the person alone is detected if separation is detected.

  DESCRIPTION OF SYMBOLS 1 Image monitoring apparatus, 2 Image pick-up part, 3 Storage part, 4 Control part, 5 Output part, 30 Background image, 31 Immediately preceding image, 32 Arrangement information, 33 Area information, 40 Change area extraction part, 41 Background flow detection part, 42 arrangement change detection unit, 43 background image generation unit, 44 mask image generation unit, 410 input flow detection unit, 411 background flow selection unit, 420 arrangement object region detection unit, 421 ghost detection unit, 422 arrangement object tracking unit.

Claims (5)

An image processing apparatus that sequentially inputs input images captured in a monitoring space and performs mask processing for privacy protection on the input images,
A storage unit that stores a background image of the monitoring space in which a background including an arrangement placed in the monitoring space is captured;
A change area extraction unit for obtaining a change area between the background image and the input image;
A mask processing unit that performs the mask processing on the input image in the change area;
An arrangement change area specifying unit that compares the background image with the input image and obtains an arrangement-derived area generated by movement of the arrangement among the change areas;
With
The mask processing unit excludes the arrangement-derived region from the mask processing target;
An image processing apparatus. The image processing apparatus according to claim 1.
The arrangement change region specifying unit detects corresponding points having similar image characteristics between the background image and the input image, calculates a movement amount between the corresponding points, and the movement amount is equal to or greater than a predetermined value. An image processing apparatus comprising: detecting the arrangement-derived region from the position of the corresponding point. The image processing apparatus according to claim 2,
The image processing apparatus according to claim 1, wherein the arrangement change area specifying unit detects the arrangement-derived area generated at the movement destination of the arrangement from the position of the corresponding point on the input image side. The image processing apparatus according to claim 2 or 3,
The arrangement change area specifying unit detects the arrangement-derived area generated at the movement source of the arrangement from the position of the corresponding point on the background image side. The image processing apparatus according to claim 4.
The arrangement change area specifying unit registers the background image of the arrangement-derived area generated at the movement source as a template of the arrangement, and after the registration, the matching process using the template performs the matching in the input image. An image processing apparatus characterized by detecting an arrangement-derived region.

Images