JP6273921B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus suitable for use in a monitoring system.

  In recent years, a monitoring system using an imaging device typified by a television camera has been developed for the purpose of measuring traffic on a road, identifying a traveling vehicle, measuring the speed of a traveling vehicle, or identifying an object that has entered a predetermined site. Proposed.

  In such a monitoring system, a background subtraction method that obtains an image of a detection target object by subtracting a background image from an input image including the detection target object and a background image, or a currently input image frame and a previously input image What detects an object using a method that detects an object based on a change in an image, such as an inter-frame difference method that calculates a difference from a frame and detects an area (change area) having a large difference value as a moving object. is there. In addition, there is one that specifies an intruding object by measuring a distance from the imaging device to the detected object.

  Conventionally, a stereo method is widely known as a method for measuring an image from an imaging apparatus and measuring a distance to a subject. In this method, a subject to be distance-measured is imaged by a plurality of imaging devices, and the distance is measured by the principle of triangulation using the distance (baseline length) between the imaging devices and the difference (parallax) on the image. is there.

  Another method is a DFD (Depth from Defocus) method. This measures the distance from the correlation information between the blur amount of the image and the subject distance, and can be realized with a monocular camera, which is advantageous for downsizing of the system.

  In addition, as a technique for estimating the amount of blur of an image, a technique of estimating a PSF (Point Spread Function) that expresses the characteristic of the blur from the radius of a dark circle appearing on the logarithmic amplitude spectrum of the image ( Non-Patent Document 1) and a technique (Non-Patent Document 2) for expressing a blur characteristic using a luminance gradient vector distribution on a logarithmic amplitude spectrum of an image and estimating a PSF (Non-Patent Document 2).

  However, since these methods involve image frequency analysis, there is a problem that the processing load is heavy and the processing takes time.

D. B. Gennery, “Determination of optical transfer function by inspection of frequency-domain plot,” Journal of the Optical Society of America, vol. 63, pp. 1571-1577, 1973. IEICE Transactions D Vol. J90-D, No.10, pp.2848-2857 “Noise robust defocus PSF estimation using luminance gradient vector distribution on logarithmic amplitude spectrum”, Morihiko Sakano, Suetake Noriaki, Eiji Uchino

  The present invention has been made to solve the above-described conventional problems, and an object of the present invention is to realize estimation of an image blur amount by a simple process without frequency analysis.

  An image processing apparatus according to the present invention includes an edge extracting unit that extracts an edge of an image, a difference between pixel values of a pair of pixels located immediately on both sides of the edge extracted by the edge extracting unit, and the pair of pixels A blur amount calculation unit that calculates a blur amount of the edge based on a difference between pixel values of at least a pair of pixels located on both sides of the image processing apparatus.

  According to the present invention, the blur amount of an image can be estimated by a simple process that does not involve frequency analysis.

It is a figure which shows the monitoring system containing the image processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the function structure of the blur amount estimation means in FIG. It is a figure for demonstrating operation | movement of the brightness | luminance difference calculation means and the blur amount calculation means in FIG. It is a figure which shows the relationship between the brightness | luminance of a pixel of an edge part and its both sides, and a blur amount. It is a flowchart which shows operation | movement of the blur amount estimation means in FIG. It is a block diagram which shows the function structure of the distance measurement means in FIG. It is a figure for demonstrating the preparation procedure of a blurring amount / distance table. It is a flowchart which shows operation | movement of the distance measurement means in FIG. It is a figure for demonstrating operation | movement of the brightness | luminance image extraction means and brightness | luminance image synthetic | combination means in FIG. It is a figure which shows the example of the measuring object of the distance measurement means in FIG. It is a block diagram which shows the function structure of the object detection means in FIG. It is a flowchart which shows operation | movement of the object detection means in FIG. It is a figure for demonstrating operation | movement of the object detection means in FIG. It is a block diagram which shows the function structure of the shadow removal means in FIG. It is a flowchart which shows operation | movement of the shadow removal means in FIG. It is a figure for demonstrating operation | movement of the shadow removal means in FIG. It is a figure for demonstrating the measurement principle of the speed measurement means in FIG. It is a graph of the characteristic curve which shows the correspondence of the brightness | luminance and the amount of blurs in the image which the movement blur generate | occur | produced. It is a block diagram which shows the function structure of the speed measurement means in FIG. It is a flowchart which shows operation | movement of the speed measurement means in FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<Monitoring system including image processing device>
FIG. 1 is a diagram showing a monitoring system including an image processing apparatus according to an embodiment of the present invention. This monitoring system includes a PC (personal computer) 1, a camera 2, a keyboard 3, a mouse 4, and a monitor 5 each connected to the PC 1.

  Here, the PC 1 is an image processing apparatus according to an embodiment of the present invention, and performs various image processing (details will be described later) by processing the image data from the camera 2.

  The camera 2 is an imaging device arranged so as to image a monitoring area. In this embodiment, a color camera is used, but a monochrome camera can also be used.

  The keyboard 3 and the mouse 4 are input devices for the user to give various instructions to the PC 1, and the monitor 5 is an output device for displaying an image generated by the camera 2 imaging the monitoring area.

  The PC 1 includes a CPU (Central Processing Unit) 11, an input unit 12, a camera control unit 13, a ROM (Read Only Memory) 14, a RAM (Random Access Memory) 15, a storage 16, and an output unit each connected to the CPU 11. 17 is provided.

  The input unit 12 is an interface for communicating with the camera 2, the keyboard 3, and the mouse 4, and the output unit 17 is an interface for communicating with the monitor 5. The camera control unit 13 is an interface for controlling the focal length, aperture, shutter speed, and the like of the camera.

  The CPU 11 is a processor that controls the operation of the entire PC 1. The ROM 14 is a memory (including rewritable ones) that stores programs executed by the CPU 11 and fixed data (such as various tables described later). The RAM 15 is a memory that serves as a work area for the CPU 11. The storage 16 is composed of a large-capacity storage device such as a hard disk, and stores image data generated by the camera 2 and subjected to image processing by the CPU 11, data generated by the CPU 11, application programs related to monitoring operations, and the like.

  The CPU 11 functions as a function block realized by executing a program stored in the ROM 14 or the storage 16, as a blur amount estimation unit 21, a distance measurement unit 22, an object detection unit 23, a shadow removal unit 24, and a speed measurement unit 25. It has.

  The blur amount estimation unit 21 estimates the blur amount of an image generated by the camera 2 imaging the monitoring area. The distance measuring unit 22 estimates a blur amount of an image generated by the camera 2 capturing an image of the monitoring area, and based on data indicating a correspondence relationship between the blur amount, the focal position of the camera 2, and the distance to the subject. Measure (estimate) the distance from 2 to the subject.

  The object detection unit 23 estimates the amount of blur of the background difference image of the image generated by the camera 2 imaging the monitoring area, and whether the amount of blur is the amount of blur when the subject in the monitoring area set in advance is imaged. Based on whether or not, an object that has entered the monitoring area is detected.

  The shadow removing unit 24 estimates the blur amount of the image generated by the camera 2 imaging the monitoring area, and identifies the shadow portion in the image based on the data indicating the correspondence between the blur amount and the shadow. , Remove the shadow part in the image.

  The speed measuring means 25 estimates the movement blur amount (blur amount) of the image of the object generated by the camera 2 imaging the monitoring area, and based on the data indicating the correspondence between the movement blur amount and the movement speed, Measure (estimate) the moving speed of the object.

<Functional block diagram of blur amount estimation means>
FIG. 2 is a block diagram showing a functional configuration of the blur amount estimation means 21 in FIG. As illustrated, the blur amount estimation unit 21 includes an image acquisition unit 211, a monochrome conversion unit 212, a noise removal unit 213, an edge vector calculation unit 214, an edge determination unit 215, a luminance difference calculation unit 216, and a blur amount calculation unit 217. It has. The edge vector calculation means 214 and the edge determination means 215 constitute an edge extraction means.

  The image acquisition unit 211 is a unit that acquires an image from the camera 2 by receiving an image generated by the camera 2 imaging the monitoring area via the input unit 12. At this time, the optical system of the camera 2 is set to a state where the depth of field is shallow, in other words, a state where the focusing range is narrow, by the control of the focal length and the diaphragm by the camera control unit 13.

  The monochrome conversion unit 212 converts a color image that is an image acquired by the image acquisition unit 211 into a monochrome image. This conversion is performed as generation of a luminance (Y) component by combining each color component of a color image, or extraction of an arbitrary color component (eg, G).

  The noise removing unit 213 includes a median filter and a Gaussian filter, and removes pixel noise of the monochrome image from the monochrome conversion unit 212. When a monochrome camera is used as the camera 2, the monochrome conversion means 212 is not necessary.

  The edge vector calculation means 214 is composed of a 3 × 3 Sobel filter or a Canny filter, and calculates an edge vector (edge intensity and edge direction) of each pixel of the monochrome image from which noise has been removed.

That is, assuming that the output of the 3 × 3 Sobel filter Sx in the horizontal direction (x direction) of the image is fx and the output of the 3 × 3 Sobel filter Sy in the vertical direction (y direction) of the image is fy, the following formula [ 1] and edge strength and edge direction are calculated by equation [2].
Edge strength = √ (fx ² + fy ² ) (1)
Edge direction = tan ⁻¹ (fy / fx) (2)

  The edge determination means 215 is larger than the edge vector strengths of the pixels on both sides of the edge vector strength (edge strength) of each pixel (target image) calculated by the edge vector calculation means 214, that is, a difference or An edge is extracted by determining an image of interest whose ratio exceeds a predetermined threshold value as an edge portion pixel.

  The luminance difference calculating unit 216 calculates a luminance difference as pixel values of a predetermined number of pixels located on both sides of the edge portion pixel in the edge vector direction. That is, for example, the luminance difference between a pair of pixels located on both sides of the edge portion pixel is calculated.

  The blur amount calculation unit 217 calculates an edge blur amount from the edge intensity of the edge pixel extracted by the edge vector calculation unit 214 and the luminance difference calculated by the luminance difference calculation unit 216.

<Calculation of luminance difference and blur amount at edge>
FIG. 3 is a diagram for explaining the operations of the luminance difference calculation unit 216 and the blur amount calculation unit 217 in FIG. In this figure, the horizontal axis represents pixels and the vertical axis represents luminance.

  In FIG. 3, a pixel Pe is an edge portion pixel. The pixel Pe + 1 and the pixel Pe-1 are a pair of pixels adjacent to each other in the edge vector direction of the pixel Pe (immediately on both sides), and the luminance difference: A is determined by the edge vector calculation means 214 by the edge intensity of the pixel Pe. Is calculated as The pixel Pe + 2 and the pixel Pe-2 are a pair of pixels adjacent to both the pixel Pe + 1 and the pixel Pe-1, and the luminance difference B is calculated by the luminance difference calculating means 216.

The blur amount calculating means 217 calculates the blur amount of the edge by comparing the luminance difference: B and the luminance difference: 2A by the following equation [3].
Bokeh amount = B ÷ 2A Formula [3]

In FIG. 3, when the luminance difference between the pixel Pe + 2 and the pixel Pe + 1 is B1, and the luminance difference between the pixel Pe-2 and the pixel Pe-1 is B2, The blur amount can also be calculated by the equation [4].
Bokeh amount = (A + B1 + B2) ÷ 2A (4)

<Relationship between brightness of pixel near edge and amount of blur>
FIG. 4 is a diagram illustrating the relationship between the luminance of the edge portion and pixels on both sides thereof and the amount of blur. The relationship between the brightness of the pixels near the edge and the amount of blur will be described using this figure.

  The horizontal axis of this figure is the pixel position, and the ninth pixel is the edge portion pixel. Also, the vertical axis in the figure is the luminance of the pixel. Further, σ is the square root of the variance value in the Gaussian distribution (normal distribution), and the numerical value is a value used as a parameter of the Gaussian filter. That is, the relationship between the brightness and the amount of blur of the edge portion and the pixels on both sides of the edge portion shown in this figure is the brightness versus pixel position characteristic using σ as a parameter.

  This characteristic is obtained by calculating a cumulative distribution function of a Gaussian distribution with the total number of luminance values of 18 pixels being 256, using σ as a parameter. Since it is known that the blur amount of an actual captured image has a Gaussian distribution, this characteristic can be said to be a numerical value of the blur amount. The value of σ is related to the amount of blur, and the larger the σ, the larger the amount of blur.

  That is, for example, when σ = 0, the luminance of the ninth pixel corresponding to the pixel Pe in FIG. 3 is 128, and the luminances of the tenth and eighth pixels corresponding to the pixel Pe + 1 and the pixel Pe-1 are 256, respectively. , 0, A = 256. Since the eleventh and seventh pixels corresponding to the pixel Pe + 2 and the pixel Pe-2 are 256 and 0, respectively, B = 256 (B1 = 0, B2 = 0). Accordingly, when the blur amount when σ = 0 is calculated by the equation [3], “B ÷ 2A = 256 ÷ 512 = 0.5” is obtained. This value is the minimum amount of blur.

  As the value of σ increases, the slope of the characteristic curve becomes gentle, so the value of A decreases, the values of B1 and B2 increase, and B and 2A converge to the same slope. In this case, the blur amount is 1 which is the maximum value.

<Operation of blur amount estimation means>
FIG. 5 is a flowchart showing the operation of the blur amount estimating means 21 in FIG.
First, the image acquisition unit 211 acquires an image from the camera 2 set in a state where the depth of field is shallow (step S1). This image is a color image.

  Next, the monochrome conversion means 212 converts the color image into a monochrome image (step S2), and the noise removal means 213 removes noise in the monochrome image (step S3).

  Next, the edge vector calculation unit 214 calculates an edge vector of each pixel of the monochrome image from which noise has been removed (step S4), and the edge determination unit 215 extracts an edge based on the strength of the edge vector (step S5). In the process of executing step S4, the value A in FIG. 3 is calculated.

  Next, the luminance difference calculation unit 216 has a predetermined number of pixels on both sides in the edge vector direction of the edge pixel constituting the edge extracted by the edge determination unit 215, for example, a pair of pixels on both sides of the edge unit pixel. Is calculated (step S6). By this step, the value B in FIG. 3 is calculated. Finally, the blur amount calculation unit 217 calculates the blur amount using the equation [3] (step S7).

  As described above, according to the blur amount estimation unit of the embodiment of the present invention, the blur amount of an image can be estimated by a simple process without frequency analysis. In the embodiment described above, the amount of blur is calculated by the equation [3]. However, instead of 2A of the denominator of the equation [3], an output of a 5 × 5 Sobel filter may be used. In this case, the pixel values of a pair of pixels on both sides of the pixel Pe + 2 and the pixel Pe-2 are also reflected in the blur amount.

<Functional block diagram of distance measuring means>
FIG. 6 is a block diagram showing a functional configuration of the distance measuring means 22 in FIG.
As illustrated, the distance measurement unit 22 includes an image acquisition unit 221, a monochrome conversion unit 222, a noise removal unit 223, an edge vector calculation unit 224, an edge determination unit 225, a luminance difference calculation unit 226, and a blur amount calculation unit 227. I have. The configuration of these means is the same as the means of the same name in the blur amount estimation means 21.

  The distance measuring unit 22 includes a blur amount / luminance image generating unit 228, a luminance threshold setting unit 229, a luminance image extracting unit 22a, and a luminance image synthesizing unit 22b.

  The blur amount / luminance image generation unit 228 as the distance image generation unit generates data (correlation information) indicating the correspondence between the blur amount of the image captured by the camera 2 and the distance between the focus of the camera 2 and the subject. A blur amount / luminance image as a distance image is generated from the blur amount of the edge calculated by the blur amount calculation means 227 using a table (hereinafter referred to as a blur amount / distance table).

  The blur amount / luminance image is an image having a luminance representing the distance from the focus of the camera 2 to the position of the subject corresponding to the edge of the image. In the present embodiment, the closer to the focal point, the higher the luminance. Here, the blur amount / distance table is created in advance and stored in the storage 16 as a storage unit of correlation information. The contents and creation procedure of the blur amount / distance table will be described later.

  When the user inputs a distance to be measured using the keyboard 3 and the mouse 4, the luminance threshold setting unit 229 is a unit that sets a blur amount / luminance image luminance threshold corresponding to the distance in the RAM 15. In the present embodiment, the distance is set within 1 m, within 2 m, within 3 m, etc. from the focal point. For example, when the brightness threshold is within 1 m from the focal point, the luminance threshold is a luminance representing the blur amount of the subject located 1 m from the focal point in the luminance image generated by the blur amount / luminance image generating unit 228.

  The luminance image extraction unit 22 a extracts pixels whose luminance exceeds the luminance threshold set by the luminance threshold setting unit 229 from the pixels of the image generated by the blur amount / luminance image generation unit 228. As a result, the luminance value of the pixel exceeding the luminance threshold value remains unchanged, and the luminance value of the pixel equal to or lower than the luminance threshold value is “0”. Here, the edges of the image of the subject located within 1 m, within 2 m, and within 3 m from the focus are individually extracted. Note that the luminance value of a pixel having a luminance exceeding the luminance threshold may be a constant value such as “1”.

  The luminance image synthesizing unit 22b synthesizes the three luminance images extracted by the luminance image extracting unit 22a. By this synthesis, for example, an image representing the edge of the subject within 1 m, within 2 m, and within 3 m from the focal point is generated. Here, within 3 m from the focal point includes within 2 m and within 1 m, and within 2 m includes within 1 m, so luminance values in overlapping ranges are not added. However, when the luminance value of a pixel having a luminance exceeding the luminance threshold is fixed to “1” or the like, by adding, “1” within 1 m from the focal point exceeds “3”, 1 m is exceeded, and “2” within 2 m is “2”. Exceeding and within 3 m is “1”, and exceeding 3 m is “0”.

<Bokeh amount / distance table creation procedure>
FIG. 7 is a diagram for explaining a procedure for creating a blur amount / distance table. Here, FIG. 7A is a graph showing the relationship between the distance from the lens and the size (diameter) of the circle of confusion when the focus of the lens having a diameter of 50 mm is matched with the distance of 10 m from the lens. This data is published by the camera manufacturer. FIG. 7B is a graph showing a blur amount estimated value obtained by calculating the blur amount of the image blurred by the Gaussian filter by the equation [3].

  As shown in FIG. 7A, there is a correspondence between the distance from the lens and the size of the circle of confusion. It is known that there is a correspondence between the size of the circle of confusion and the amount of blur, so if you create data that shows the correspondence between the amount of blur and the circle of confusion, you can see the correspondence between the amount of blur and the distance from the lens. Data to be shown, that is, a blur amount / distance table can be created.

  Therefore, the blur-free image obtained by the camera having the characteristics shown in FIG. 7A is processed by a Gaussian filter to generate a blur image, and the blur amount is calculated by Equation [3], whereby the data shown in FIG. 7B is obtained. create.

  In this figure, the numerical value of the blur amount by the Gaussian filter on the horizontal axis represents the size of the Gaussian filter, and the size n (n = 1 to 11) means (2n + 1) × (2n + 1). Also, the blur amount estimated value on the vertical axis is a value obtained by subtracting 1 from 2 times the value calculated by the equation [3]. The shaded portion is a range that the shadow removing unit 24 determines as a shadow.

  Here, the amount of blur by the (2n + 1) × (2n + 1) Gaussian filter means that one pixel of an image without blur is blurred to (2n + 1) pixels. Therefore, by acquiring data indicating the correspondence between (2n + 1) pixels and the size of the circle of confusion in FIG. 7A, data in which the value on the vertical axis in FIG. 7A is converted into the number of pixels can be created. By using the data shown in FIG. 7B, data in which the vertical axis value in FIG. 7A is the blur amount estimation value shown in the vertical axis in FIG. 7B, that is, the blur amount / distance table can be created. It is preferable that a plurality of blur amount / distance tables are provided using at least one of the focal length and the aperture value (F value) as a parameter. Note that data indicating the correspondence between (2n + 1) pixels and the size of the circle of confusion in FIG. 7A can be acquired from the specifications of the camera, such as the number of pixels of the image sensor.

<Operation of distance measuring means>
FIG. 8 is a flowchart showing the operation of the distance measuring means 22 in FIG. In this figure, the blur amount estimation processing in step S11 is the same as the entire flowchart (steps S1 to S7) shown in FIG.

  In the next step S12, the blur amount / luminance image generation unit 228 generates a luminance image representing the blur amount, that is, an image whose luminance corresponds to the blur amount. Here, an image having a higher luminance is generated for an image of a subject having a smaller blur amount, in other words, a subject closer to the focal point.

  Next, the brightness threshold value setting unit 229 obtains setting information of the distance measurement range by the user, that is, information indicating the distance to be measured that is input by the user using the keyboard 3 and the mouse 4 (step S13). Referring to the table, a luminance threshold value corresponding to the distance that the user wants to measure is set (step S14).

  Next, the luminance image extraction unit 22a extracts pixels whose luminance value is equal to or higher than the luminance threshold set in step S14 by the luminance threshold setting unit 229 from the pixels of the image generated by the blur amount / luminance image generation unit 228. (Step S15).

  Next, the luminance image synthesizing unit 22b synthesizes luminance images at the respective set distances (step S16). That is, the pixels corresponding to the brightness of each set distance extracted in step S15 are synthesized.

<Operation of Luminance Image Extraction Unit and Luminance Image Synthesis Unit>
FIG. 9 is a diagram for explaining the operation of the luminance image extracting means 22a and the luminance image synthesizing means 22b in FIG.

  Here, FIG. 9A shows an image extracted by the edge determination means 225. 9B, FIG. 9C, and FIG. 9D show images within 1 m, within 2 m, and within 3 m from the focal point extracted by the luminance image extracting means 22a, respectively. FIG. 9E shows a composite image generated by the luminance image composition means 22b. Note that although the actual image is a grayscale image, it is described here as a binary line drawing for convenience. In addition, the brightness of this composite image was made to correspond to the thickness of the line.

  As shown in FIG. 9A, there are images of a person 31, a building 32, and a tree 33. Further, as shown in FIGS. 9B, 9C, and 9D, the person 31 is within 1 m from the focal point, the parts other than the person 31 and the roof of the building 32 are within 2 m from the focal point, the person 31, the building 32, and the leaves of the tree 33 The lower half and the trunk are located within 3 m from the focal point.

  Then, as shown in FIG. 9E, the person 31 has the highest luminance, the portion other than the roof of the building 32 has medium luminance, the roof of the building 32 and the lower half of the leaves of the tree 33 and the trunk have low luminance, and the upper half of the leaves. The luminance value of is 0.

<Measurement target of distance measuring means>
FIG. 10 is a diagram illustrating an example of a measurement target of the distance measuring unit 22 in FIG.
The camera 2 is attached so as to image the monitoring area from above by holding means (not shown). The focal point 43 of the lens of the camera 2 is set between the head 42a and the torso 42b of the person 42 expected to enter the ground 41 in the monitoring area, that is, at the height of the neck. The depth of field is set to the length from the top to the center of the trunk.

  By setting the luminance image extraction means 22a shown in FIG. 6 and extracting the luminance image within the depth of field, the shadow formed on the ground 41 or the small animal that has entered the monitoring area is erroneously detected as a person. The situation can be prevented.

  According to the distance measuring unit 22 according to the embodiment of the present invention, based on the data indicating the correspondence between the blur amount of the subject image generated by imaging the subject with the camera 2 and the distance from the camera 2 to the subject, When estimating the distance from the camera 2 to the subject, estimation of the blur amount of the image can be realized by a simple process without frequency analysis.

<Functional block of object detection means>
FIG. 11 is a block diagram showing a functional configuration of the object detection means 23 in FIG.
As illustrated, the object detection unit 23 includes a detection range setting unit 231, a background distance image generation unit 232, an input distance image generation unit 233, a difference image generation unit 234, and an area recognition unit 235.

  The detection range setting means 231 is a means for setting an object detection range as a monitoring area based on the user setting of the focal length and the aperture of the camera 2. This object detection range corresponds to the depth of field portion in FIG.

  The background distance image generation unit 232 performs distance image generation processing on the generated image obtained by imaging the monitoring area when the detection target object does not exist in the object detection range set by the detection range setting unit 231, that is, an image This is a means for generating a background distance image by generating an image having luminance corresponding to the distance estimated from the amount of blur.

  The luminance value of the background distance image is a positive value corresponding to the distance from the focus for a subject within the object detection range, and “0” for a subject outside the object detection range. The background distance image generation process is the same as the process shown in steps S11 to S15 in FIG.

  The input distance image generation means 233 is a means for generating an input distance image by generating a background distance image and then performing a distance image generation process on the image generated by the camera 2 imaging the monitoring area.

  Similar to the luminance value of the background distance image, the luminance value of the input distance image is a positive value corresponding to the distance from the focus for the subject in the object detection range, and “0” for the subject outside the object detection range. The input distance image generation process is the same as the process shown in steps S11 to S15 in FIG.

  The difference image generation means 234 is a means for generating a difference image by taking the difference between the input distance image generated by the input distance image generation means 233 and the background distance image generated by the background distance image generation means 232.

  The luminance value of the difference image is a positive value corresponding to the distance from the focus for a portion where there is a difference between the input distance image and the background distance image within the object detection range, and “0” for a portion where there is no difference. Accordingly, the portion where the luminance of the difference image has a positive value represents the edge of the object that has entered the object detection range.

  The region recognition unit 235 is a unit that performs region recognition by adding a circumscribed rectangle to a portion with a positive luminance in the difference image generated by the difference image generation unit 234.

  The luminance of the difference image surrounded by the area recognition unit 235 corresponds to the distance from the focus of the camera 2. In addition, the size (vertical size, horizontal size) of the circumscribed rectangle generated by the region recognition unit 235 corresponds to the size of the object that has entered the object detection range. Therefore, it is possible to specify the intruding object.

<Operation of object detection means>
FIG. 12 is a flowchart showing the operation of the object detection means 23 in FIG.
First, the detection range setting unit 231 sets the object detection range based on an operation of adjusting the focal length and the aperture of the camera 2 using the keyboard 3 and the mouse 4 while the user views the image from the camera 2 on the monitor 5. (Step S21).

  At this time, an object simulating a human body is arranged in the monitoring area, and an image having a luminance corresponding to the distance from the focus of the camera 2 is displayed on the monitor 5 by the same processing as the background image generation processing. Yes. The user moves the object back and forth and adjusts the focal length and the aperture so that the brightness of the image of the object changes from 0 to a predetermined value, so that the front end of the detection range (the closest distance from the camera 2) and The rear end (the farthest distance from the camera 2) can be set.

  The detection range setting unit 231 stores the detection range setting information in which the combination of the focal length and the aperture is associated with the detection range (front end and rear end) in the ROM 14, thereby operating the object detection unit 23 thereafter. When the detection range setting information is read, automatic setting is possible. In addition, by storing a plurality of combinations of focal lengths and apertures in association with a plurality of different detection ranges, the detection range can be variably set. This detection range setting information is created by extracting necessary information from the blur amount / distance table used by the distance measuring means 22 described above.

  Next, the background distance image generation means 232 generates a background distance image (step S22). Next, the input distance image generation unit 233 generates an input distance image (step S23).

  Next, the difference image generation means 234 generates a difference image from the input distance image and the background distance image (step S24). Then, the area recognizing unit 235 recognizes the area of the object by generating a frame for the pixel having a positive luminance value (step S25).

<Operation of object detection means>
FIG. 13 is a diagram for explaining the operation of the object detection means 23 in FIG. Here, FIGS. 13A, 13B, 13C, 13D, and 13E show a background image, an input image, a background distance image, an input distance image, and a difference image, respectively. FIG. 13F shows a state in which a circumscribed rectangle is attached to the object in the difference image. Note that although the actual image is a grayscale image, it is described here as a binary image for convenience.

  In FIG. 13A, a hatched area 51 indicates the floor of the monitoring area. That is, the lower end of the monitoring area is the area 51, and the upper end is the upper end of the screen in the figure.

  As illustrated in FIG. 13B, the input image includes an object (here, a human body) 52 and a butterfly 53. Here, it is assumed that the object 52 is located in the monitoring area and the butterfly 53 is located in front of the monitoring area, that is, between the camera 2 and the monitoring area.

  As illustrated in FIG. 13C, in the background distance image, only the portion within the monitoring area is extracted from the background image illustrated in FIG. 13A. It can also be said that the portion outside the monitoring area (indicated by a broken line) is removed.

  As illustrated in FIG. 13D, in the input distance image, only a portion within the monitoring area is extracted from the input image illustrated in FIG. 13B. Accordingly, the object image 61 is extracted, but the butterfly image is not extracted.

  As illustrated in FIG. 13E, the difference image is a difference between the input distance image illustrated in FIG. 13D and the background distance image illustrated in FIG. Then, as shown in FIG. 13F, a circumscribed rectangle 72 is added to the object image 71 to recognize the object region.

  Thus, according to the object detection means 23, when detecting the intrusion of an object into the monitoring area based on the change in the image obtained by imaging the imaging range including the monitoring area, the depth of the real space of the imaging range is recognized. The detected object can be realized by a simple process.

<Functional block of shadow removal means>
FIG. 14 is a block diagram showing a functional configuration of the shadow removing unit 24 in FIG.
As illustrated, the shadow removal unit 24 includes an image acquisition unit 241, a monochrome conversion unit 242, a noise removal unit 243, an edge vector calculation unit 244, an edge determination unit 245, a luminance difference calculation unit 246, a blur amount calculation unit 247, An amount / luminance image generation unit 248, a luminance threshold setting unit 249, and a luminance image extraction unit 24a are provided.

  These means are configured similarly to the means of the same name in the distance measuring means 22. However, the shadow removing unit 24 separates and removes shadows in the image using the property that the shadow image of the object is blurred due to light diffraction and that the blur amount becomes larger as the distance from the object increases. It is desirable that the image acquisition unit 241 acquires an image in a state where the depth of field of the camera 2 is deep (defocus is unlikely to occur). It is also desirable to increase the shutter speed (exposure time: short) in order to prevent the occurrence of blurring (movement blur) due to movement of the subject. The image acquisition unit 241 is preferably configured to output a background difference image of the image from the camera 2.

<Operation of shadow removal means>
FIG. 15 is a flowchart showing the operation of the shadow removing means 24 in FIG. In this figure, the blur amount estimation process in step S31 is the same as the entire flowchart (steps S1 to S7) shown in FIG. However, here, the image is acquired in a state where the depth of field of the camera 2 is set deep.

  In the next step S32, the blur amount / luminance image generation means 248 generates a luminance image representing the blur amount, that is, an image whose luminance corresponds to the blur amount. Here, an image is generated in which the luminance decreases as the blur amount increases. Therefore, when an object exists in the monitoring area, the brightness of the shadow image is lower than that of the object image, and the brightness between the shadow images increases as the distance from the lower end of the object (in the case of a human body) increases. Lower.

  Next, the luminance image extraction unit 24a removes pixels lower than the predetermined threshold set by the luminance threshold setting unit 249 from the image generated by the blur amount / luminance image generation unit 248 (step S33). At this time, theoretically, the shadow image can be removed by setting the maximum blur amount of the shadow image as the luminance threshold. In this embodiment, the blur amount of the image generated by the Gaussian filter in FIG. 7B: 4 or more (estimated blur amount value: 0.36 or less) is determined as a shadow.

Since the blur amount of the shadow image changes depending on the brightness of the monitoring area and the irradiation direction of sunlight, it is desirable to dynamically change the threshold according to those conditions. Therefore, the threshold value is dynamically changed as in (1) and (2) below.
(1) The size (length or area) of the region surrounded by the edges extracted by the edge determination means 245 is calculated, and the threshold value is changed according to those values. At this time, since the area surrounded by the edge is a combination of the object and its shadow, the larger the length or area, the longer the shadow, in other words, the lower the solar radiation angle. If the shadow is long, the amount of blur of the shadow increases (the brightness decreases), so the threshold value is lowered.
(2) The average value of the brightness of the edges extracted by the edge determination means 245 is calculated, and the threshold value is changed according to the value. When the average brightness is high, the shadow is clear and the average value of the blur amount is small (the brightness of the brightness image representing the blur amount is high), so the threshold value is increased.

<Operation of shadow removal means>
FIG. 16 is a diagram for explaining the operation of the shadow removing means 24 in FIG. 16A is an output image of the noise removing unit 243, FIG. 16B is an output image of the blur amount / luminance image generating unit 248, and FIG. 16C is an output image of the luminance image extracting unit 24a.

  As shown in FIG. 16A, the output image of the noise removing unit 243 includes a human body image 81 and a human body shadow image 82. For convenience, the human body image 81 has vertical lines, and the human body shadow image 82 has meshes.

  As shown in FIG. 16B, in the output image of the blur amount / luminance image generation means 248, the human body image 83 with a small blur amount has a high luminance, and the human shadow image 84 with a large blur amount has a low luminance. Here, the thickness of the edges of the images 83 and 84 represents the brightness.

  As shown in FIG. 16C, in the output image of the luminance image extracting means 24a, a human body image 83 having a high luminance is extracted (a human shadow image 84 having a low luminance is removed).

  As described above, according to the shadow removing unit 24, since the shadow is separated and removed by using the blur of the edge of the shadow image, the shadow can be removed with high accuracy even if the light irradiation condition changes. Can do.

<Principle of speed measurement>
FIG. 17 is a view for explaining the measurement principle of the speed measuring means 25 in FIG. Here, FIG. 17A is a diagram for explaining the distance traveled by the vehicle to be measured within the exposure time of the camera 2, and FIG. 17B is a diagram for explaining blurring (moving blur) of the vehicle image captured by the camera 2. It is a figure for doing. Here, the camera 2 is installed so as to take an image of the vehicle from a direction orthogonal to the side surface thereof, that is, a direction orthogonal to the moving direction of the vehicle.

  As shown in FIG. 17A, when the speed of the vehicle 91 is V, the moving distance d of the vehicle 91 during the exposure time ts (shutter speed) of the camera 2 is “V × ts”. Also, as shown in FIG. 17B, the blur amount B of the vehicle image 92 captured by the camera 2 at this time corresponds to the number of horizontal pixels corresponding to the moving distance d, and therefore the vehicle speed V, exposure time ts, and This is a function of the distance from the camera 2 to the vehicle 91. In FIG. 17B, only the vertical line in the image 92 is shown by a slanted line.

  A high gradation occurs in the movement direction in the luminance of the vertical line (vertical edge) in the vehicle image 92. Theoretically, this gradation is a linear change, but in an actual image, it is a curve that approximates the cumulative distribution curve in the same way as out of focus.

  FIG. 18 is a graph of a characteristic curve showing a correspondence relationship between the luminance and the amount of blur in an image in which moving blur occurs, that is, a diagram showing an example of the above curve. In this figure, the horizontal axis represents the pixel position, and the vertical axis represents the luminance value. Here, the blur amount corresponds to the blur amount (B) in FIG. 17B.

As described above, the image blur amount is a function of the vehicle speed V, the exposure time ts, and the distance from the camera 2 to the vehicle 91. The vehicle speed V can be calculated by the following equation [5].
V = “number of pixels corresponding to the amount of blur (B)” × “distance in pixel units in the image” ÷ “exposure time ts” (5)

  In this equation, “the number of pixels corresponding to the blur amount (B)” can be obtained from the characteristic curve shown in FIG. Further, the “distance in pixel units in the image” can be obtained by measuring the number of pixels of an image of an object of a predetermined size captured by the camera 2. This distance becomes longer as the distance from the camera 2 to the position through which the measurement object passes is longer, that is, as the position through which the measurement object passes away from the camera 2.

  Therefore, if a table for calculating the speed from the blur amount of the image (hereinafter referred to as a blur amount / speed table) is created in advance using the distance from the camera 2 to the measurement object and the exposure time as parameters, the blur amount estimation means 21. Similarly, the blur amount B is estimated, and the velocity V can be obtained by referring to the blur amount / speed table.

<Functional block of speed measurement means>
FIG. 19 is a block diagram showing a functional configuration of the speed measuring means 25 in FIG.
As illustrated, the speed measurement unit 25 includes an image acquisition unit 251, a monochrome conversion unit 252, a noise removal unit 253, an edge vector calculation unit 254, an edge determination unit 255, a luminance difference calculation unit 256, a blur amount calculation unit 257, a blur amount calculation unit 257, and the like. A quantity / luminance image generation means 258 is provided.

  These means are configured similarly to the means of the same name in the distance measuring means 22. However, since the speed measurement unit 25 obtains the speed of the object by utilizing the fact that the blur (moving blur) of the object image corresponds to the speed of the object, the image acquisition unit 251 determines the depth of field of the camera 2. It is desirable to acquire an image in a deep state (where it is difficult for out-of-focus). By increasing the depth of field of the camera 2, for example, the speed of a traveling vehicle in two lanes can be measured.

  The vertical line detection unit 259 is a unit that detects a vertical line (vertical edge) in an image having luminance corresponding to the blur amount generated by the blur amount / luminance image generation unit 258. In the present embodiment, the front end, rear end, window, and the like of the vehicle 91 are detected as vertical lines. The speed calculation unit 25a refers to a blur amount / speed table prepared in advance, and calculates a speed corresponding to the movement blur (blur) of each pixel constituting the vertical line.

<Operation of speed measuring means>
FIG. 20 is a flowchart showing the operation of the speed measuring means 25 in FIG. In this figure, the blur amount estimation processing in step S41 is the same as the entire flowchart (steps S1 to S7) shown in FIG. Here, however, the image is acquired by vertically capturing the side surface of the vehicle traveling on the road in a state where the depth of field of the camera 2 is set deep.

  In the next step S42, the blur amount / luminance image generation means 258 generates a luminance image representing the blur amount, that is, an image whose luminance corresponds to the blur amount. Here, the smaller the blur amount, in other words, the slower the vehicle image, the higher the luminance.

  Next, the vertical line detection unit 259 detects the vertical line of the luminance image representing the amount of blur (step S43). Thus, in the case of the vehicle image 92 shown in FIG. 17B, the front and rear ends of the body, the front and rear ends of the door, and the like are detected.

  Next, the speed calculation means 25a calculates the speed corresponding to the brightness of the pixel of the vertical line detected by the vertical line detection means 259 from the blur amount / speed table (step S44).

  Here, theoretically (ideally), the luminance of each pixel constituting one vertical line is the same. In addition, the luminance of each pixel is the same between a plurality of vertical lines. Therefore, theoretically, the speed can be obtained from the luminance of one pixel of one vertical line. However, since it is conceivable that the luminance is actually different, it is preferable to calculate an average value of luminance for each vertical line or to calculate an average value of all the vertical lines. In addition, a moving object such as the vehicle 91 can be detected by detecting a horizontal line (horizontal edge) of the image, and a height of the moving object can be estimated by calculating an interval between the upper and lower horizontal lines. .

21: blur amount estimating means, 214 ... edge vector calculating means, 215 ... edge determining means, 216 ... luminance difference calculating means, 217 ... blur amount calculating means.

Claims (3)

Edge extraction means for extracting the edge of the image;
Based on the difference between the pixel values of a pair of pixels located immediately on both sides of the edge extracted by the edge extraction means, and the difference between the pixel values of at least a pair of pixels located on both sides of the pair of pixels, A blur amount calculating means for calculating the blur amount of the edge;
An image processing apparatus. The image processing apparatus according to claim 1,
The edge extraction means compares an edge vector calculation means for calculating an edge vector of each pixel constituting an image with an intensity of the edge vector and a predetermined threshold value, and determines an edge portion pixel constituting the edge. And an image processing apparatus. The image processing apparatus according to claim 2,
The blur amount calculating unit calculates a difference between pixel values of a pair of pixels adjacent to the edge portion pixel, which is calculated in the process of calculating the edge vector of the edge portion pixel by the edge extracting unit. An image processing apparatus having a difference between pixel values of a pair of pixels located at a position.

Images