JP7129271B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing apparatus that detects a detection target from within an image.

Conventionally, by comparing an image obtained by photographing a monitoring area with a camera and a reference image, a changed area, which is an area where there is a change between the two images, is obtained, and the size or shape of the changed area is determined. There is an image processing apparatus that detects a detection target such as an intruder from a photographed image by determining "likeness of a person" based on the image features.

The reference image can be set based on a past image taken in the past when the detection target did not exist in the monitoring area. Note that the reference image is generally updated every several minutes or every ten and several minutes, depending on the brightness of the monitored area depending on the time of day.

Further, conventionally, there are cases where the photographing method for photographing the monitored area is changed. If the imaging method is changed and the pixel values of the captured image are changed as a result, the standard image used before the imaging method change will extract a large amount of change, so the imaging method after the change will be used. It becomes impossible to appropriately compare with the captured image that has been acquired. Therefore, conventionally, when a photographing method of a photographed image is changed, after obtaining several frames of the photographed image after the change, the reference image is updated based on the few frames, and then the photographed image and the reference image are obtained. A comparison was made with

For example, Patent Document 1 discloses a daytime mode in which a color image of a monitoring area captured with visible light is output as a captured image, and a nighttime mode in which an image captured by illuminating the monitoring area with infrared light is output as a captured image. A security system is disclosed that updates the reference image each time the shooting mode (daytime mode or nighttime mode) is switched.

JP 2009-239739 A

As described above, the reference image is updated when the photographing method of the photographed image is changed. However, conventionally, immediately after the photographing method of the photographed image is changed, that is, between the time the photographing method is changed and the reference image is updated, there is no reference image corresponding to the photographing method after the change. It will be. Therefore, immediately after the photographing method of the photographed image is changed and until the reference image is updated, there is a problem that a monitoring disabled period occurs during which the detection target cannot be detected from the photographed image.

A similar problem may also occur when the brightness of the monitoring area changes abruptly (for example, when the illumination in the monitoring area that has been turned off is turned on). In other words, even if the brightness of the monitoring area suddenly changes, the pixel values of the captured image are changed as a whole, and it becomes impossible to appropriately detect the detection target by comparison with the reference image that has been used up to that point. is.

SUMMARY OF THE INVENTION It is an object of the present invention to detect a detection target from a photographed image without causing a monitoring disabled period even when the photographing method of the photographed image is changed or when the brightness of the monitored area is abruptly changed. It is to be detectable.

The present invention provides an image processing apparatus for extracting a change area from a reference image in a target image to be processed from among captured images sequentially captured by a capturing method according to a capturing environment, wherein a predetermined period including the target image is extracted. When the storage means for storing the photographed image of and the photographing method of the target image match the photographing method of the photographed image photographed immediately before the target image or the photographing method of the reference image, the photographing method is more temporal than the target image extracting the changed area from the reference image set using the past image which is the photographed image photographed before and the target image; extracting means for extracting the changed region from the target image and the reference image set using a future image, which is the captured image captured temporally later than the target image among the captured images; An image processing apparatus characterized by

Preferably, the storage means further stores the reference image set using the past image, and stores a method of photographing the target image, a photographed image photographed immediately before the target image, or a method of photographing the reference image. a reference image updating means for updating the reference image using the future image before the extraction processing by the extraction means when it is determined that the photographing method is different, and characterized by having

Desirably, when updating the reference image using the future image, the reference image updating means updates any of the future images used for comparison if a difference region between the plurality of future images is not extracted. One of them is used as the reference image.

Preferably, when updating the reference image using the future image, if a plurality of the inter-future-image difference areas are extracted, the reference image updating means updates each pixel in the future image other than the difference area. and each pixel in the difference area in the reference image before updating based on the pixel value change amount obtained from the pixel value of each corresponding pixel outside the difference area in the reference image before updating to set the pixel values of corresponding pixels in the difference area of the reference image after updating, and the pixel values of the areas other than the difference area in the future image are corrected to the difference area of the reference image after updating A pixel value of each pixel other than .

Preferably, the reference image updating means updates the pixel value of each pixel in the peripheral area of the difference area in the future image and the pixel value of each pixel in the image area of the reference image before updating corresponding to the peripheral area. and obtaining the pixel value change amount.

Further, the present invention is an image processing apparatus for extracting a change area from a reference image in a target image to be processed from among captured images in which a monitoring area is sequentially captured, wherein the brightness of the monitoring area and the captured image are brightness change determination means for determining whether or not there is a change of a level or more in the brightness of the monitoring area from at least one of the brightness of the; storage means for storing the captured images for a predetermined period including the target image; When it is determined by the brightness change determining means that there is no change equal to or greater than the level between when the target image is captured and when the captured image immediately before the target image or the reference image is captured, the target image is The changed area is extracted from the reference image set using the past image, which is the captured image captured earlier in time, and the target image. extracting means for extracting the changed region from the target image and the reference image set using a future image captured temporally later than the target image among the stored captured images; An image processing apparatus characterized by having

According to the present invention, detection targets can be detected from a captured image without causing a monitoring disabled period even when the method of capturing the captured image is changed or the brightness of the monitored area is changed. can be

1 is a schematic configuration diagram of a security system according to this embodiment; FIG. 1 is a schematic configuration diagram of an image sensor according to an embodiment; FIG. FIG. 4 is a conceptual diagram showing a plurality of captured images arranged in order of acquisition time; FIG. 10 is a diagram showing a second change area in the background image and the future image; FIG. 4 is a diagram showing peripheral areas in a background image and a future image; FIG. 4 is a diagram showing luminance values of an estimated human area and a peripheral area in a background image and a future image; It is a figure which shows the content of a change rate table. FIG. 10 is a diagram showing luminance values of an estimated human region in an updated background image; FIG. 10 is a diagram showing luminance values of an estimated human region and surrounding regions in an updated background image; FIG. 10 is a schematic configuration diagram of an image sensor according to a modified embodiment; 10 is a flowchart showing the flow of intruder determination processing;

Embodiments of the present invention will be described below.

FIG. 1 is a schematic diagram of the configuration of a security system 10 according to this embodiment. The security system 10 includes a security device 14 installed in each property 12 to be monitored such as a store, an office, an apartment building, a warehouse, a house, etc., and a security center connected to each security device 14 via a communication network 16 such as a public telephone line. It comprises a device 18 and a user device 20 . Further, the security system 10 includes one or more images for detecting a detection target (a human figure in this embodiment) from the monitored area within the monitored object 12 based on the photographed image of the monitored area of the monitored object 12. It includes a sensor 22 (image processing device) and a recording device 24 for recording an image captured by the image sensor 22 . Image sensor 22 and recording device 24 are communicatively connected to security device 14 .

When the security device 14 receives an alarm signal from the image sensor 22 connected to itself via a local LAN or the like, the alarm signal and the identification signal of the security device 14 itself, or the property 12 to be monitored or an abnormality is detected. The identification signal of the image sensor 22 is transmitted to the security center device 18 . Therefore, the security device 14 has a communication interface for communicating with the image sensor 22, a communication interface for communicating with the security center device 18 and the user device 20, and a control unit for controlling them.

The security center device 18 is composed of a so-called computer, and has a communication interface for communicating with the security device 14 via the communication network 16, a display device such as a liquid crystal display, and a notification unit composed of a buzzer, an LED, and the like. . When the security center device 18 receives an alarm signal from the security device 14 via the communication network 16, the security center device 18 notifies the property 12 to be monitored where the security device 14 that sent the alarm signal and the content of the detected abnormality are sent to the reporting unit and Notify the observer through the display device.

The user device 20 is also composed of a so-called computer, and includes a communication interface for communicating with the security device 14 via the communication network 16, a display device such as a liquid crystal display, and a keyboard, mouse, etc. for remotely operating the security device 14. A user interface for inputting operation commands for When the user device 20 is operated to observe the object 12 to be monitored which has been registered in advance via the user interface, the user device 20 is currently photographing the security device 14 installed at the object 12 to be monitored which has been registered. Various image request signals are transmitted to request the user device 20 to transmit the captured image inside or the captured image recorded in the recording device 24 . Then, when the photographed image is received from the security device 14, the user device 20 displays the requested photographed image on the display unit.

The recording device 24 includes recording media such as a magnetic disk device such as an HDD, a magnetic tape such as a DAT, and an optical recording medium such as a DVD-RAM. It consists of a device that reads and writes The recording device 24 receives the photographed image photographed by the image sensor 22 from the security device 14 and records it in association with the photographing time.

FIG. 2 is a schematic diagram of the configuration of the image sensor 22. As shown in FIG.

The communication unit 30 is an input/output interface for transmitting and receiving various setting signals and control signals between the image sensor 22 and the security device 14 via a communication network such as a local LAN. communication interface circuits and driver software for driving them. Specifically, when an intruder is detected by the signal processing unit 44 described later, the communication unit 30 outputs an intrusion alarm signal indicating that the intruder has been detected to the security device 14 .

The photographing unit 32 obtains a photographed image by photographing a monitoring area in the property 12 to be monitored. The photographing unit 32 according to the present embodiment is capable of photographing the monitoring area with the same angle of view using a plurality of photographing methods. A plurality of imaging methods here means methods in which the pixel values of the captured images obtained by imaging the same monitoring area are entirely different. For example, a method of photographing with visible light (capturing a visible image), a method of photographing with near-infrared illumination (capturing a near-infrared image), and a distance image showing the distance from the photographing unit 32 to the photographed object and a method of capturing a thermal image showing the temperature of the captured object. Note that the fact that the pixel values of the captured image are totally different includes the case where the pixel values of a small number of pixels are the same.

Therefore, the imaging unit 32 includes a photoelectric converter such as a CCD that is sensitive to visible light and infrared light, an imaging optical system that forms an image of the monitored area on the photoelectric converter, and an imaging engineering system. It includes an IR cut filter (infrared cut filter) provided, a projection LED for measuring the distance to the object to be photographed, or an infrared image sensor for detecting the temperature of the object to be photographed.

In this embodiment, the photographing unit 32 switches between two photographing modes as a plurality of photographing methods: a daytime mode in which the monitored area is photographed with visible light, and a nighttime mode in which the monitored area is photographed with near-infrared light illumination. shall be taken. Specifically, in the daytime mode, an IR cut filter is inserted into the optical path of the imaging optical system so as to cut near-infrared light, and the two-dimensional detector of the imaging unit 21 acquires visible light images. On the other hand, in the night mode, the IR cut filter is removed out of the optical path of the imaging optics and the two-dimensional detector acquires near-infrared images. In addition, the sensitivity wavelength range of the photoelectric converter can be controlled by a method in which a visible camera for acquiring visible light images and a near infrared camera for acquiring near infrared light images are separately installed, or by changing the voltage applied to the organic thin film having a laminated structure. , a visible image may be captured in the daytime mode, and a near-infrared image may be captured in the nighttime mode. Incidentally, the switching of the shooting mode is executed by the shooting mode switching means 48, which will be described later.

In the present embodiment, the imaging unit 32 acquires a captured image by performing imaging at regular time intervals (for example, 1/5 second), but the imaging interval of the imaging unit 32 is not limited to this. The captured image thus acquired is stored in the storage unit 38 . Note that although the imaging unit 32 is included in the image sensor 22 in this embodiment, the imaging unit 32 may be provided outside the image sensor 22 .

The illuminance sensor 34 includes, for example, a photodiode. The illuminance sensor 34 detects the illuminance (brightness) of the monitored area and outputs an illuminance signal indicating the detected illuminance to the signal processing unit 44 described later. Various well-known illuminance sensors can be used as the illuminance sensor 34 .

The illumination unit 36 includes, for example, a near-infrared LED. The illumination unit 36 irradiates the monitoring area with near-infrared light when the imaging mode of the imaging unit 32 is the night mode. On the other hand, when the shooting mode of the shooting unit 32 is the daytime mode, the lighting unit 36 is turned off.

The storage unit 38 (storage means) is composed of a semiconductor memory, a magnetic disk (HDD), or an optical disk drive such as CD-ROM, DVD-RAM, and its recording medium. The storage unit 38 stores an image processing program for operating each unit of the image sensor 22 . In addition, as shown in FIG. 2, the storage unit 38 stores a photographed image 40 and a background image 42 acquired by the photographing unit 32 .

The photographed image 40 is an image acquired by the photographing unit 32 sequentially photographing the monitoring area. The storage unit 38 stores captured images 40 for several tens of frames (images). The latest frame is stored after deleting the old frame. That is, the storage unit 38 accumulates and stores captured images 40 for a predetermined period.

FIG. 3 shows a conceptual diagram of a plurality of photographed images 40 arranged in order of acquisition time and stored in the storage unit 38 . In this specification, among the plurality of captured images 40 stored in the storage unit 38, an image to be processed for human image detection is referred to as a sensing image 40t (target image). An image captured in the past is called a past image 40p, and an image captured after the sensing image 40t is called a future image 40f. In this embodiment, the storage unit 38 stores 10 frames of the past image 40p, 1 frame of the sensing image 40t, and 10 frames of the future image 40f. When finished, the captured image 40 captured next to the sensing image 40t (the future image 40f one frame after the sensing image 40t) becomes the new sensing image 40t. Thus, in the present embodiment, the sensing image 40t is not the latest (most recently acquired) captured image 40 . Further, the photographing method by which the photographed image 40 stored in the storage unit 38 is photographed is, for example, the past image 40p is a visible image, and the sensing image 40t and the future image 40f are near-infrared images.

The background image 42 (reference image) is set using the past image 40p. Specifically, it is generated by the signal processing unit 44, which will be described later, based on the past image 40p when there is no moving object such as a person in the monitoring area among the past images 40p. For example, the signal processing unit 44 obtains an inter-frame difference between two past images 40p adjacent in time series, and obtains an average value of absolute values of luminance differences between corresponding pixels between frames. If there is no moving object in both frames, the average absolute value of the luminance difference between the two frames is considerably small. Therefore, the signal processing unit 44 causes the storage unit 38 to store one of the past images 40p whose average value is smaller than the predetermined reference as the background image 42 . The background image 42 is updated at regular intervals (for example, every 10 minutes) in order to deal with variations in the brightness of the monitored area due to variations in lighting conditions, diurnal variations in the sun, and the like. Note that the stored background image 42 may be a captured image captured before the past image 40p.

Note that in the present embodiment, the captured image 40 and the background image 42 are stored in the storage unit 38 within the image sensor 22, but the captured image 40 and the background image 42 are stored in a storage unit provided outside the image sensor 22. and stored in a storage accessible from the image sensor 22 .

The signal processing unit 44 has a built-in microprocessor unit, memories such as ROM and RAM, and peripheral circuits thereof, and executes various signal processing of the image sensor 22 . As shown in FIG. 2 , the signal processing unit 44 functions as brightness detection means 46 , shooting mode switching means 48 , background image update means 50 , extraction means 52 and determination means 54 . By the signal processing unit 44 exerting these means, even if the shooting mode of the shooting unit 32 is suitably switched and the shooting mode is switched, sensing is possible without causing a monitoring disabled period. In the image 40t, a person area, which is an image area corresponding to a person, is detected. Each unit included in the signal processing unit 44 will be described below.

The brightness detection means 46 detects the brightness of the monitoring area, and determines whether the brightness of the monitoring area is suitable for the daytime mode or the nighttime mode among the shooting modes of the imaging unit 32. determine whether the In this embodiment, the brightness detection means 46 detects the brightness of the monitored area based on the illuminance signal received from the illuminance sensor 34 .

For example, the brightness detection means 46 determines that the brightness is suitable for the daytime mode when the illuminance indicated by the illuminance signal is equal to or higher than the first illuminance threshold and continues for a predetermined time (for example, 30 seconds). If the illuminance indicated by the signal is less than the first illuminance threshold and less than the second illuminance threshold for a predetermined period of time, it is determined that the brightness is suitable for the night mode. Note that the first illuminance threshold and the second illuminance threshold may be the same. In addition, the requirement for the predetermined time may be omitted, that is, the brightness detection means 46 immediately detects the brightness suitable for the daytime mode when a state occurs where the illuminance indicated by the illuminance signal is equal to or higher than the first illuminance threshold. , and when the illuminance indicated by the illuminance signal is less than the second illuminance threshold value, it may be immediately determined that the brightness is suitable for the night mode.

Further, the brightness detection means 46 may estimate the brightness of the monitoring area from the pixel values of the captured image 40 (for example, the latest captured image 40). For example, the average brightness value of a plurality of pixels forming the whole or part of the captured image 40 is calculated, and based on the average brightness value, the brightness of the monitoring area is suitable for the daytime mode or the brightness suitable for the nighttime mode. You may make it determine whether it is. Specifically, when the average luminance value is equal to or greater than the threshold, it is determined that the brightness is suitable for the daytime mode, and when the average luminance value is less than the threshold, it is determined that the brightness is suitable for the nighttime mode. can. If the brightness detection means 46 estimates the brightness of the monitored area based on the captured image 40, the illuminance sensor 34 can be omitted.

A photographing mode switching unit 48 switches the photographing mode of the photographing unit 32 based on the determination result of the brightness detecting unit 46 in order to photograph a photographed image with a photographing method suitable for the brightness of the photographing environment. Specifically, when it is determined that the brightness of the monitoring area is suitable for the daytime mode, the imaging mode switching means 48 sets the imaging mode of the imaging unit 32 to the daytime mode and switches the monitoring area to the daytime mode. is determined to be suitable for the night mode, the photographing mode of the photographing section 32 is set to the night mode. Note that the photographing mode switching means 48 may switch the photographing mode of the photographing section 32 according to an instruction from the user.

Background image updating means 50 (reference image updating means) updates the background image 42 stored in the storage unit 38 . Specifically, the background image updating means 50 replaces the background image 42 with the photographing method of the sensing image 40t when the photographing method of the sensing image 40t and the photographing method of the past image 40p photographed immediately before the sensing image 40t are different. Update to a suitable image for the method. Note that the term “immediately before” includes not only the past image 40p that is one before the sensing image 40t but also a plurality of past images 40p that are a plurality of times before the sensing image 40t when arranged in order of acquisition time (see FIG. 3). may be included. The details of the processing of the background image updating means 50 will be described later.

The extracting means 52 compares the sensing image 40t and the reference image (the background image 42 in this embodiment) and extracts one or a plurality of changed regions composed of pixels having different pixel values between the two images. The extraction unit 52 also performs a labeling process of assigning an identifier (label) to each pixel forming the change area. As the label, for example, a numerical value unique within the change area image can be used. In the labeling process, when a certain extracted pixel is focused on, a cluster of extracted pixels consisting of other extracted pixels adjacent to the extracted pixel is regarded as one changing region. Note that the pixels adjacent to the extracted pixel may be pixels adjacent to the extracted pixel in the vertical and horizontal directions, or may include pixels adjacent to the extracted pixel in the oblique direction. Then, a process of assigning different labels to the respective changing regions and storing them as pixel values of the changing region image is performed. Furthermore, the extracting means 52 performs shaping processing such as integration, separation, or deletion on the labeled changed regions in order to obtain human regions more appropriately.

In the present embodiment, the extracting means 52 compares the sensing image 40t and the background image 42 and extracts a changed region from the sensing image 40t. , and the past image 40p acquired immediately before the sensing image 40t may be compared to extract the changed region. In that case, the past image 40p becomes the reference image.

The determining means 54 determines whether or not the changed area extracted by the extracting means 52 is a human area. For example, the determining means 54 obtains a feature amount such as the size (width/height) of the shaped change area, and compares the obtained feature amount with a pre-stored feature amount representing humanness to determine the It is determined whether the change area is human-like. In addition, after performing tracking processing of the changed regions between the plurality of captured images 40, the movement speed of the changed regions is calculated from the position of the changed regions in each captured image 40 and the acquisition time of each captured image 40, and the changed region is calculated. and a pre-stored human-like speed may be further compared to determine whether or not the changed region is a human region.

When the determining means 54 determines that the changed area extracted from the sensing image 40 t is the human area, the communication section 30 transmits an intrusion alarm signal to the security device 14 .

The outline of the configuration of the image sensor 22 is as described above. Details of the processing of the background image updating means 50 will be described below.

In the present embodiment, first, the background image updating unit 50 switches the shooting mode of the shooting unit 32 by the shooting mode switching unit 48, and changes the shooting mode of the sensing image 40t and the past image 40p shot immediately before the sensing image 40t. It is determined whether or not the photographing modes of are different from each other. When the shooting mode is switched by manual operation or the like and a mode change input is received from the shooting mode switching means 48, it may be determined whether the shooting modes are different from each other based on whether or not the input is received. Also, instead of the past image 40p captured immediately before the sensing image 40t, a background image 42 (hereinafter, the two images may be collectively referred to as a “determination target image”) is compared with the image capturing mode. are different from each other.

The shooting mode of each image may be recorded in the metadata (attribute information) of each image, and the background image updating means 50 can confirm the shooting mode by checking the metadata of each image. Alternatively, the background image updating means 50 can determine the shooting mode of each image based on the pixel values of each image. For example, when using an image pickup device having a color filter that allows the visible light RGB components to pass through separately, but the IR component to pass the same amount of light, the variation (dispersion) of the RGB values of each pixel of the image causes the The shooting mode of the image can be determined. Specifically, the variance value of the RGB values of all pixels of the image is calculated, and if the calculated variance value is equal to or greater than a predetermined value (the variation in the RGB values is large), the image is a visible image, that is, the daytime image. If the calculated variance value is less than a predetermined value (variation in RGB values is small), it is determined that the image is a near-infrared image, that is, that the image was shot in the night mode. In the daytime mode, visible images are captured in a bright environment, so the RGB values of each pixel vary greatly according to the color of the object in the captured image. Since the near-infrared image is captured by projecting light, the RGB values of each pixel show similar values, and the variation is small. Also, if the imaging device has an IR element, the imaging mode of the image may be determined based on whether the IR value is equal to or greater than a predetermined value. Alternatively, the shooting mode of each image may be determined based on the determination result of the brightness detection means 46 when each image was shot. Data indicating the result of switching the shooting mode by the shooting mode switching means 48 may be stored as metadata in the shot image 40 shot by switching the shooting mode. , and whether or not there is information indicating the result of switching the shooting modes, it may be determined whether or not the shooting modes are different from each other.

When the imaging mode of the sensing image 40t and the imaging mode of the determination target image are different, the background image updating unit 50 updates the background image using the future image 40f stored in the storage unit 38 before the extraction processing by the extraction unit 52. 42 is executed to update the background image.

In the background image updating process, first, the background image updating means 50 extracts a difference area by comparing a plurality of future images 40f stored in the storage unit 38, and determines whether or not a moving object is captured. In this embodiment, the moving object is assumed to be a person to be detected, but the moving object is not limited to a person, and may be other animals, for example. More specifically, the background image update means 50 compares a plurality of future images 40f to determine whether or not an estimated human area (second change area), which is an area changed from the image to be compared in the future image 40f, is extracted. do.

In this embodiment, the background image updating means 50 creates a difference region, which is a set of pixels having a difference of a predetermined value or more between the pixel values of the corresponding pixels of the future images 40f that are adjacent in time series. Generate a difference image that represents Thereby, a plurality of difference images are obtained. Then, it is determined whether or not there is an area extracted as a difference area a predetermined number of times or more in the plurality of obtained difference images. Specifically, the background image updating unit 50 sets the pixel value of the pixels included in the difference area to "1", and sets the pixel value of the other pixels (pixels whose difference in pixel value is less than a predetermined value) to "0". is generated as a difference image, the total value of the pixel values of pixels at the same coordinates in a plurality of binary images is calculated, and the difference region is a set of pixels whose total value is equal to or greater than a predetermined value be the estimated person region. In addition, instead of using a predetermined value set in advance, the average value obtained from pixels with pixel values other than "0" in the binarized image is compared with the total value to estimate a difference region in which the total value is equal to or greater than the average value. It may be a human area. Also, the difference area may be an area surrounded by a minimum rectangle that includes all the extracted areas.

If no presumed human region is extracted from the future image 40f, the background image updating means 50 sets one of the multiple future images 40f as the background image 42. FIG. In other words, the updated background image 42 is one of the multiple future images 40f. On the other hand, when the presumed human region is extracted in the future image 40f, the background image updating means 50 performs background image generation processing described below.

In the background image generation process, the background image updating means 50 updates the pixel value of each pixel other than the estimated human region in one of the plurality of future images 40f (hereinafter simply referred to as the “future image 40f”), A pixel value change amount from the background image 42 to the future image 40f is calculated based on the pixel value of each corresponding pixel other than the estimated human region in the background image 42 before updating.

The estimated person area in the background image 42 is an area made up of a plurality of pixels having the same coordinates as the pixels included in the estimated person area of the future image 40f. The background image 42 is shown in FIG. 4(a), and the future image 40f is shown in FIG. 4(b). FIG. 4(b) shows an estimated human area 60 detected in the future image 40f, and FIG. 4(a) shows an estimated human area 60' in the background image 42. As shown in FIG.

Preferably, as shown in FIG. 5, the background image updating means 50 updates the pixel value of each pixel in a peripheral region 62 (see FIG. 5B) of the estimated human region 60 in the future image 40f and the peripheral region 62 The pixel value of each pixel in the image area of the background image 42 before updating corresponding to pixel value), and the amount of change in pixel value is calculated.

In this embodiment, the background image updating means 50 calculates the magnification (rate of change) of the change in the luminance value of pixels at the same coordinates from the background image 42 to the future image 40f, and expresses the rate of change for each luminance value. Generate a rate of change table. For example, the brightness value of each pixel included in the peripheral region 62' of the background image 42 is shown in FIG. 6A, and the brightness value of each pixel included in the peripheral region 62 of the future image 40f is ) will be described as an example. It should be noted that FIG. 6 is a simplified diagram, and in reality the peripheral regions 62, 62' or the estimated human regions 60, 60' are composed of a larger number of pixels. Also, in the present embodiment, the luminance value is used as the pixel value for calculating the rate of change, but the pixel value is not limited to this, and may be, for example, a hue value.

Focusing on the upper leftmost pixel of the peripheral region 62', the brightness value of the pixel is "2" in the background image 42, but the brightness value of the corresponding pixel is "6" in the future image 40f. Therefore, in the rate of change table, the rate of change is "three times" with respect to the luminance value "2". Similarly, focusing on the uppermost pixel of the peripheral region 62', the brightness value of the pixel is "36" in the background image 42, but the brightness value of the corresponding pixel is "12" in the future image 40f. ing. Therefore, in the rate of change table, the rate of change is "1/3 times" with respect to the luminance value "36".

In this manner, the background image updating means 50 obtains the rate of change for each luminance value and generates a rate of change table as shown in FIG. Note that the background image updating means 50 complements the rate of change for the luminance value that did not exist as the luminance value of the pixels in the peripheral region 62′ (for example, the luminance value “1” or “3” in the example of FIG. 7). You may do so. For example, the rate of change of luminance values that are close to each other (for example, the rate of change for a luminance value of “1” is “three times” and the rate of change to a luminance value of “3” is “two times”), or Calculate the change rate based on the change rate (for example, the change rate for the luminance value "3" is calculated as the median of the change rate "3 times" for the luminance value "2" and the change rate "2 times" for the luminance value "4". 2.5 times”).

Next, the background image updating means 50 corrects the luminance value of each pixel in the estimated human region 60′ in the background image 42 before updating based on the generated change rate table, and updates the luminance value after the correction. is the luminance value of the corresponding pixel of the estimated human region 60'' of the background image 42'. For example, as shown in FIG. 6(a), the luminance value of the upper leftmost pixel in the estimated human region 60′ is “5”, and as shown in FIG. The rate of change corresponding to the value "5" is "3 times". Therefore, as shown in FIG. 8, the background image updating means 50 sets the luminance value of the upper leftmost pixel in the estimated human region 60'' in the updated background image 42' to 5×3=“15”.

Further, the background image updating means 50 sets the pixel values of the areas other than the estimated human area 60 of the future image 40f as the pixel values of the pixels of the updated background image 42' other than the estimated human area 60''. Thereby, as shown in FIG. 9, an updated background image 42' is generated, that is, the background image 42 is updated.

As described above, the background image updating unit 50 updates the background image 42 corresponding to the photographing mode before the change to the future image 40f corresponding to the photographing mode after the change based on the amount of change in the pixel value of the person who is the moving object. Each pixel value in the estimated human region 60′ of the background image 42 before updating, in which no image exists, is corrected to obtain the pixel value of the pixels in the estimated human region 60″ of the background image 42′ after updating. As a result, each pixel value in the estimated human region 60'' of the updated background image 42' can be set to a pixel value in which there is no human image and which corresponds to the changed shooting mode. Further, the pixel value of each pixel other than the estimated human region 60'' of the background image 42' after updating is the estimated human region 60 of the future image 40f corresponding to the shooting mode after change, where there is no human image from the beginning. The pixel value of each pixel other than As a result, the background image 42' after the update does not include the human image, which is the moving object, and can correspond to the shooting mode after the change.

In particular, since the peripheral regions 62, 62' are located in the vicinity of the estimated human regions 60, 60' that require pixel value correction, the pixel values of the pixels in the peripheral region 62' of the background image 42 and the future By obtaining the pixel value change amount based on the pixel value of the pixels in the peripheral region 62 of the image 40f, the pixel value change amount suitable for correcting the pixel value of the pixels in the estimated human region 60′ of the background image 42 is obtained. can be asked for. In other words, when the brightness values of the pixels in the estimated human area and the pixels in the adjacent peripheral area are close, there is a high possibility that the object captured in the peripheral area continues to the estimated human area, and furthermore, the image changes before and after switching the shooting mode. Identical objects in the image undergo similar luminance changes. Therefore, by obtaining the pixel value change amount from the peripheral areas 62 and 62', it is possible to obtain the pixel value change amount suitable for the object captured in the estimated human area 60''. This reduces the possibility that the pixel values of the pixels in the estimated human region 60'' will be unnatural values in the updated background image 42', and a more natural background image 42' can be generated. .

When the photographing mode of the sensing image 40t and the photographing mode of the past image 40p or the background image 42 photographed immediately before the sensing image 40t match, the extracting means 52 uses the past image 40p or the background image 42 as a reference image. , to extract a changed region from the sensing image 40t. On the other hand, if the imaging mode of the sensing image 40t is different from the imaging mode of the past image 40p or the background image 42 shot immediately before the sensing image 40t, the reference image updated by the background image updating means 50 as described above Based on, the changed area is extracted from the sensing image 40t.

In this way, when the shooting method of the shot image 40 is changed, the background image 42 is updated using the future image 40f so as to be suitable for the current shooting method. In the sensing image 40t, it is possible to detect a person area, which is an image area corresponding to a person.

In the present embodiment, the imaging unit 32 images the monitoring area using two types of imaging methods, the visible image and the near-infrared image. . For example, the monitoring area is photographed by three types of photographing methods of a visible image, a distance image, and a thermal image.

Further, in the present embodiment, when extracting the estimated human region, a difference image is generated for each future image, and the region extracted as the difference region a predetermined number of times or more from the obtained multiple difference images is extracted as the estimated human region. However, the number of difference images to be generated may be one. For example, if a difference image is created from two future images captured immediately after the sensing image, and if no difference region is extracted in the created difference image, one of the future images used to create the difference image is used as the background. When the difference area is extracted from the difference image, the extracted difference area may be used as the estimated human area.

Further, in the present embodiment, the background image updating means 50 generates the change rate table, but when correcting the luminance value of each pixel in the estimated human region 60' of the background image 42, the change rate table is generated. It doesn't have to be. For example, the average value of the luminance values of the plurality of pixels in the region other than the estimated human region 60′ of the background image 42 and the average value of the luminance values of the plurality of pixels in the region other than the estimated human region 60′ of the future image 40f. A pixel value change amount may be obtained from the difference between . In particular, the pixel value change amount is calculated from the difference between the average value of the brightness values of the plurality of pixels in the peripheral region 62′ of the background image 42 and the average value of the brightness values of the plurality of pixels in the peripheral region 62 of the future image 40f. you may ask.

Further, in the present embodiment, the pixel value change amount is obtained based on one future image 40f selected from a plurality of future images 40f and the background image 42. It is also possible to generate an average future image by averaging the pixel values of , and obtain the pixel value change amount based on the average future image and the background image 42 .

Further, when the background image updating unit 50 updates the background image 42 when the imaging mode of the sensing image 40t and the imaging mode of the determination target image are different, the imaging mode of the future image 40f is determined and the imaging mode of the sensing image 40t is determined. It is even more preferable to update the background image using the same future image 40f as in . That is, when the shooting mode is switched in the future image 40f, the background image 42 is updated using the future image 40f before the shooting mode is switched, that is, the future image 40f shot in the same shooting mode as the sensing image 40t. do. The imaging method of the future image 40f is switched for each frame (for example, the sensing image 40t is a near-infrared image, one frame after the sensing image 40t is a visible image, and two frames after the sensing image 40t is a near-infrared image), the background image 42 may be updated using the future image 40f captured by the same imaging method as the sensing image 40t.

Further, when the extracting unit 52 extracts the changed region from the sensing image 40t by the inter-frame difference described above, the background image updating unit 50 changes the shooting mode of the sensing image 40t and the past image 40p shot immediately before the sensing image 40t. , the future image 40f (preferably the future image 40f one frame after the sensing image 40t) is used as the reference image. After that, the extracting means 52 compares the sensing image 40t with the future image 40f, and extracts a changed region in the sensing image 40t.

Further, in the present embodiment, the background image updating unit 50 updates the background image 42 when the shooting mode of the sensing image 40t and the shooting mode of the determination target image are different from each other. , the background image 42 may be updated when it is determined that there is a change in the brightness of the monitoring area that is greater than or equal to a predetermined level between when the sensing image 40t is captured and when the determination target image is captured. Even in this case, the method of updating the background image 42 may be the same as described above.

In this case, as shown in FIG. 10, the image sensor 22' according to the modified example has the brightness detection means 46 and the shooting mode switching means 48 instead of or in addition to the image sensor 22 according to the basic embodiment. , a brightness change determination means 70 is provided.

Brightness change determination means 70 determines whether or not there is a change in brightness of a predetermined level or more in the monitored area from one point in time to another point in time.

For example, the brightness change determining means 70 can determine a change in brightness of the monitored area based on the illuminance signal from the illuminance sensor 34 . In this case, each time the photographed image 40 is photographed by the photographing unit 32, illuminance information indicating the brightness of the monitored area at the time of photographing is added as metadata to the photographed image 40, and the brightness change determination means 70 can obtain the change (amount) of the brightness of the monitoring area from the time when a certain captured image 40 is captured to the time when another captured image 40 is captured.

Further, for example, the brightness change determination unit 70 calculates the average brightness value of the sensing image 40t and the average brightness value of the past image 40p immediately before the sensing image 40t, and calculates the difference between the calculated average brightness values (average brightness difference ) is equal to or greater than a preset value. If it is equal to or greater than the predetermined value, it is determined that a change in brightness equal to or greater than the standard has occurred in the monitoring area. Alternatively, when the average luminance value of the sensing image 40t is equal to or less than a first threshold and the average luminance value of the previous image 40p immediately before the sensing image 40t is equal to or greater than a second threshold higher than the first threshold, or When the average luminance value of the image 40t is equal to or greater than the second threshold and the average luminance value of the past image 40p immediately before the sensing image 40t is equal to or less than the first threshold, the change in brightness above the standard level occurs in the monitoring area. may be determined to have occurred. Instead of the previous image 40p immediately before the sensing image 40t, the background image 42 and the sensing image 40t may be compared to determine whether or not there has been a change in brightness above the standard level in the monitoring area.

Note that, in order to determine whether or not there is a change in brightness equal to or higher than the standard, another index such as average brightness may be used instead of the average luminance value of the sensing image 40t and the past image 40p. Further, it is also possible to determine whether or not there is a change in the brightness of the monitoring area that exceeds the standard based on the average luminance value of a predetermined area within the image, instead of the average luminance value of the entire image. Alternatively, it may be determined whether or not there is a change in brightness above a standard level in the monitoring area based on whether or not the pattern of brightness change (using a graph of brightness change, etc.) matches a preset pattern. .

The extraction unit 52 determines the level of brightness of the monitored area by the brightness change determination unit 70 when the sensing image 40t is captured and when the previous image 40p or the background image 42 captured immediately before the sensing image 40t is captured. If it is determined that there is no such change, the past image 40p or the background image 42 is used as a reference image to extract a changed region from the sensing image 40t. On the other hand, when the sensing image 40t is captured, and when the past image 40p or the background image 42 captured immediately before the sensing image 40t is captured, the brightness change determination unit 70 determines that the brightness of the monitoring area has changed by a level or more. If it is determined that there is, the changed area is extracted from the sensing image 40t based on the reference image updated by the background image updating means 50 as described above.

The flow of intruder determination processing in the basic embodiment will be described below according to the flowchart shown in FIG.

In step S10, the background image updating unit 50 switches the shooting mode of the shooting unit 32 by the shooting mode switching unit 48, and changes the shooting mode of the sensing image 40t to the past image 40p shot immediately before the sensing image 40t or the background image. 42 is different from each other. If the shooting modes of both images are different, the process proceeds to step S12. If the shooting modes of both images are the same, the process proceeds to step S22.

In step S12, the background image updating unit 50 determines whether or not the estimated human area 60 exists in the future image 40f based on the difference image between the multiple future images 40f.

When the estimated human region 60 does not exist in the future image 40f, the background image updating means 50 sets one of the plurality of future images 40f as the background image 42 in step S14, and proceeds to step S22.

When the estimated human region 60 exists in the future image 40f, in step S16, the background image updating means 50 updates the pixel values of each pixel in the peripheral region 62 of the estimated human region 60 in the future image 40f and the estimated Based on the pixel value of each pixel in the peripheral region 62' of the human region 60', a rate of change table is generated that represents the rate of change for each luminance value from the background image 42 to the future image 40f.

In step S18, the background image updating means 50 corrects the luminance value of each pixel in the pre-updated background image 42 in the estimated human region 60' based on the change rate table generated in step S16. Let the brightness value be the brightness value of the corresponding pixel in the estimated human region 60'' of the updated background image 42'.

In step S20, the background image updating means 50 sets the pixel values of the regions other than the estimated human region 60 in the future image 40f to the pixel values of the pixels other than the estimated human region 60'' of the updated background image 42'. . As a result, an updated background image 42' is generated.

In step S22, the extracting unit 52 compares the sensing image 40t and the background image 42, and extracts a changed region from the sensing image 40t. Here, if it is determined in step S10 that the shooting mode of the sensing image 40t is different from the shooting mode of the past image 40p or the background image 42 shot immediately before the sensing image 40t (that is, steps S12 to S20 are passed through). case), the background image 42 used in step S22 is the background image 42 updated in the processes of steps S12 to S20. On the other hand, if it is determined in step S10 that the shooting mode of the sensing image 40t matches the shooting mode of the past image 40p or the background image 42 shot immediately before the sensing image 40t, then the past image 40p or the background image Using 42 as a reference image, a changed region is extracted from the sensing image 40t.

In step S24, the background image updating means 50 determines whether or not the conditions for updating the background image 42 are satisfied. Specifically, it is determined whether or not the background image 42 has been updated for a predetermined period of time (for example, 20 frame intervals), and whether or not the change area of the sensing image is within a predetermined range.

If the condition is satisfied, the background image updating unit 50 sets the sensing image 42t as the background image 42 in step S26. If the condition is not satisfied, step S26 is bypassed, that is, the process proceeds to step S28 without updating the background image 42 .

In step S28, the determination means 54 determines whether or not the changed region extracted in step S22 is a human region. If it is in a human area, an intrusion alarm signal is sent from the communication unit 30 to the security device 14, otherwise no notification is given.

Although the embodiments according to the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the gist of the present invention.

10 security system, 12 object to be monitored, 14 security device, 16 communication network, 18 security center device, 20 user device, 22 image sensor, 24 recording device, 30 communication unit, 32 imaging unit, 34 illuminance sensor, 36 lighting unit , 38 storage unit, 40 captured image, 42 background image, 44 signal processing unit, 46 brightness detection means, 48 shooting mode switching means, 50 background image updating means, 52 extraction means, 54 determination means.

Claims (6)

An image processing device for extracting a change area from a reference image in a target image to be processed from among captured images sequentially captured by a capturing method according to a capturing environment,
a storage means for storing the captured images for a predetermined period including the target image;
If the method of photographing the target image matches the method of photographing the photographed image photographed immediately before the subject image or the photographing method of the reference image, the photographed image is the photographed image temporally before the subject image. extracting the changed area from the reference image set using a past image and the target image; extracting means for extracting the changed region from the reference image set using the future image, which is the photographed image photographed later, and the target image;
An image processing device comprising: The storage means further stores the reference image set using the past image,
determining whether or not a shooting method of the target image is different from a shooting method of a shot image shot immediately before the target image or the shooting method of the reference image, and if it is determined that the shooting methods are different, the extracting means A reference image updating means for updating the reference image using the future image before the extraction process by
2. The image processing apparatus according to claim 1, further comprising: When updating the reference image using the future image, the reference image updating means updates one of the future images used for comparison if a difference region between the plurality of future images is not extracted. is the reference image,
3. The image processing apparatus according to claim 2 , wherein: When updating the reference image using the future image, the reference image updating means, when a difference area between a plurality of the future images is extracted, is a pixel of each pixel in the future image other than the difference area. and the pixel value of each pixel in the difference area in the reference image before updating based on the pixel value change amount obtained from the pixel value of each corresponding pixel outside the difference area in the reference image before updating. The value is corrected to be the pixel value of each corresponding pixel in the difference area of the reference image after updating, and the pixel value of the area other than the difference area in the future image is corrected to the pixel value of the area other than the difference area of the reference image after updating. Let the pixel value of each pixel be
4. The image processing apparatus according to claim 2 , wherein: The reference image updating means calculates the pixel value of each pixel in the peripheral area of the difference area in the future image and the pixel value of each pixel in the image area of the reference image before updating corresponding to the peripheral area. find the amount of pixel value change,
5. The image processing apparatus according to claim 4, wherein: An image processing device for extracting a change area from a reference image for a target image to be processed from captured images in which a monitoring area is sequentially captured,
brightness change determination means for determining whether or not the brightness of the monitoring area changes by a level or more from at least one of the brightness of the monitoring area and the brightness of the captured image;
a storage means for storing the captured images for a predetermined period including the target image;
When it is determined by the brightness change determining means that there is no change equal to or greater than the level between when the target image is captured and when the captured image immediately before the target image or the reference image is captured, the target image is The changed area is extracted from the reference image set using the past image, which is the captured image captured earlier in time, and the target image. extracting means for extracting the changed region from the target image and the reference image set using a future image captured temporally later than the target image among the stored captured images;
An image processing device comprising:

Images