JP4937318B2 - Image processing apparatus and image adjustment method - Google Patents

Description

  The present invention relates to an image processing apparatus that processes an image captured by a camera, and more particularly to an image adjustment technique for correcting camera distortion.

  In recent years, development of a monitoring system using an image (video) photographed by a camera has been promoted particularly by realizing high performance and low price of a monitoring camera. The monitoring system includes an image processing device that processes an image captured by the camera, and outputs information (image information) for detecting an event in the monitoring target range using the image processing result.

  In such a monitoring system, it is possible to monitor a relatively wide space (monitoring target range) with a camera. However, in the monitoring system, since the monitoring accuracy depends on the quality of the image taken by the camera, it is important to perform adjustment work on the camera in advance.

  In a surveillance system, the number of cameras installed is directly related to the cost of the system, so it is desirable to reduce the number of cameras as much as possible. For this reason, in order to widen the monitoring range corresponding to one camera, a wide-angle lens is often adopted as a camera lens. However, in general, an image using a wide-angle lens (wide-angle image) is distorted in a barrel shape, and the distortion becomes more prominent in the periphery of the image.

  Such image distortion becomes a factor of deteriorating the monitoring performance of the monitoring system. Therefore, in order to remove the distortion of the image, adjustment work (initial adjustment work) for performing camera calibration is required for each camera. In addition, after the installation of the monitoring system, on-site adjustment work is required to designate the monitoring target area in the monitoring environment. Since these adjustment operations are required for each camera, it takes much time to install a large number of cameras. Therefore, an effective method for reducing the adjustment man-hour is desired.

  By using a method for correcting distortion of a wide-angle image, it is possible to improve the efficiency of adjustment work related to a camera using a wide-angle lens. As a method for correcting distortion of a wide-angle image, a method of correcting distortion of a wide-angle image by comparing the object height with an object on the image under a condition where the actual object height is known (for example, , See Patent Document 1).

JP 2008-52589 A

  The prior art, which is a method for correcting the distortion of the wide-angle image described above, requires an object for adjustment. For example, when applied to a monitoring system that monitors a person walking in a monitoring target area, It is difficult to realize an adjustment function that automatically performs correction. For this reason, even if the method of the prior art is simply applied, it is not easy to improve the efficiency of the adjustment work related to the camera of the surveillance system using the wide-angle image.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image processing apparatus capable of automatically performing a process for correcting distortion of an image from a camera, thereby improving the efficiency of adjustment work related to the camera. It is in.

  An aspect of the present invention calculates a movement trajectory line on an image when processing an image obtained by shooting a moving object with a camera, and calculates a lens distortion correction parameter of the camera as adjustment information based on the movement trajectory line. An image processing apparatus having a function of

  An image processing apparatus according to an aspect of the present invention includes an image input unit that inputs image information captured by a camera, a unit that extracts a moving object region from the image information, and the moving object region according to a moving locus of the moving object. Trajectory calculating means for calculating the moving trajectory line information, and adjusting means for calculating adjustment information for correcting distortion of the image information according to lens distortion of the camera based on the moving trajectory line information. It is a configuration.

  According to the present invention, it is possible to automatically perform the process of correcting the distortion of the image from the camera, so that the efficiency of the adjustment work related to the camera can be improved.

1 is a block diagram showing a main part of an image processing apparatus according to an embodiment of the present invention. The block diagram for demonstrating the structure of the monitoring system regarding this embodiment. 6 is a flowchart for explaining image adjustment processing according to the embodiment. The figure for demonstrating the effect of the image adjustment process regarding this embodiment.

  Embodiments of the present invention will be described below with reference to the drawings.

[Configuration of image processing apparatus]
FIG. 1 is a block diagram for explaining the configuration of an image processing apparatus according to this embodiment. FIG. 2 is a block diagram for explaining the configuration of the monitoring system according to this embodiment.

  The image processing apparatus 10 of the present embodiment is applied to a monitoring system 1 described later, and processes an image (video signal) transmitted from a monitoring camera (hereinafter simply referred to as a camera) 20. As shown in FIG. 1, the image processing apparatus 10 includes, for example, an image processing dedicated board (board) on which electronic components including a microprocessor and a memory are mounted, and is incorporated in a computer 100 described later. The image processing apparatus 10 is mainly composed of computer hardware and software.

  The image processing apparatus 10 includes an image input unit 11, a human extraction unit 12, a feature amount extraction unit 13, an adjustment amount calculation unit 14, an initial adjustment unit 15, and an adjustment result output unit 16. As shown in FIG. 2, the image input unit 11 inputs an image (video) taken by the camera 20 installed at an angle of view over which the monitoring target areas 200 and 210 are viewed. The image input unit 11 converts the input image into a format that can be calculated as digital image data (hereinafter referred to as image information).

  The human extraction unit 12 processes the image information from the image input unit 11 to extract a moving object region on the image. As will be described later, the present embodiment is an image processing apparatus 10 that handles a human as a moving object in order to assume a system that measures the number of people in a monitoring area as the monitoring system 1 to be applied. In addition, as a moving body, according to application of the monitoring system 1, an animal, a vehicle, etc. are contained.

  The feature amount extraction unit 13 extracts a feature amount from a human region on the extracted image, and calculates movement trajectory line information based on the feature amount as described later. The adjustment amount calculation unit 14 calculates adjustment amount information necessary for the initial adjustment based on the movement trajectory line information calculated by the feature amount extraction unit 13. Here, the adjustment amount information is parameter information for correcting lens distortion (aberration distortion) of the camera 20 that is particularly noticeable when a wide-angle lens is used.

  The initial adjustment unit 15 uses the adjustment amount information calculated by the adjustment amount calculation unit 14 to perform initial adjustment on the image information. That is, the initial adjustment unit 15 executes camera calibration processing (image adjustment processing) for removing image distortion corresponding to lens distortion of the camera 20. The adjustment result output unit 16 outputs the image information initially adjusted by the initial adjustment unit 15 to the monitoring processing unit 30 included in the monitoring system 1.

  The monitoring processing unit 30 is a main processing unit of the monitoring system 1 and executes processing for measuring the number of people and distribution existing in the monitoring area from image information (after initial adjustment) taken by the camera 20. Here, as shown in FIG. 2, the monitoring system 1 includes a computer 100 including an image processing 10 and a monitoring processing unit 30, a computer 110 for performing air conditioning control, lighting control, power control, and the like, and a monitoring area (monitoring). (Target space) 200 and 210 are configured by a plurality of cameras 20 and a network 120.

  The computer 100 of the monitoring system 1 monitors the number and distribution of humans existing in the monitoring areas 200 and 210 by inputting and processing images captured by the cameras 20. The monitoring system 1 controls the air conditioning, lighting, and power supply in cooperation with the computer 110 by monitoring the number and distribution of people existing in the monitoring target area in, for example, buildings and factories. Maintain the environment optimally.

[Image adjustment processing]
Next, an initial adjustment process will be described as an image adjustment process of the present embodiment with reference to FIGS.

  First, in general, a wide-angle lens or a super-wide-angle lens (including a fish-eye lens) that can cover a wide range of a monitoring target area is used as a mounting lens of the camera 20 used in the monitoring system 1. An image captured by such a camera 20 may be an image distorted in a barrel shape due to lens distortion (aberration distortion), for example, as shown in FIG. If the image is distorted, modeling from three-dimensional information in real space to two-dimensional information on the camera image is difficult to handle, and there is a possibility that human extraction performance and people counting performance on the image may be reduced.

  Here, FIG. 4A is an image of a black and white calibration rectangular pattern photographed by a camera using a wide-angle lens. As shown in FIG. 4A, the distortion rate increases particularly in the periphery of the image. As shown in FIG. 4B, the image processing apparatus 10 according to the present embodiment performs an initial adjustment process for each camera so that an image with corrected distortion can be acquired.

  Hereinafter, an initial adjustment process, specifically, an adjustment process method for correcting lens distortion (aberration distortion) of an image taken by the camera 20 will be described with reference to the flowchart of FIG.

  First, in the initial adjustment mode, the image processing apparatus 10 inputs an image captured by the camera 20 with the image input unit 11 (step S1). Here, it is assumed that the camera 20 captures a scene in which a human (that is, a moving object) walks (moves) within the monitoring target area 200. At this time, it is assumed that a human walks as straight as possible.

  The human extraction unit 12 performs image processing such as background difference processing on the image information from the image input unit 11, thereby tracking the moving object (human) on the image and extracting the moving object region (human region) (step) S2).

  The feature amount extraction unit 13 extracts a feature amount from the moving object region on the extracted image. In other words, the feature quantity extraction unit 13 extracts a human head or torso from a moving body region, and tracks a trajectory line (movement) that combines the center points of the region (hereinafter sometimes referred to as human extracted points) in time series. (Track line information) is calculated (step S3). Here, as an example of a specific method of extracting the head, a contour is extracted from image information by image processing such as a Sobel filter, and the extracted contour image is converted into a binary image with an appropriate threshold value. To do. Further, a Hough transformation is performed on the obtained pixels of the contour portion to vote on the Hough space. By this Hough transform, a circular portion in the image is extracted and extracted as a human head.

  By the way, since the obtained movement trajectory line information includes a human walking error and an extraction error due to image processing, if this is used as it is as straight line data, there is a possibility that lens distortion cannot be corrected with sufficient accuracy. . Therefore, the feature amount extraction unit 13 calculates the straight line input data (reference trajectory line information) by executing the following two processes to eliminate the error (step S4).

  First, the feature amount extraction unit 13 calculates an average vector from human extracted points on the past N images in order to eliminate data when the human walk is obviously not a straight line. Further, the feature amount extraction unit 13 compares the average vector with the past M images (however, N> M), the vector obtained from the human extracted points on the current image, and has an angle greater than the set value. In such a case, tracking is terminated by determining that there is a sudden change in walking direction and that the walking is not a straight line.

  Next, the feature quantity extracting unit 13 reduces the estimation error of the human extracted point by applying a quadratic approximate curve to each movement trajectory line, and uses this as “linear input data” for subsequent processing. Use. As the straight line input data is obtained in the peripheral portion on the image, that is, in a region where the influence of distortion is relatively strong, the accuracy of distortion correction described below increases.

Next, the adjustment amount calculation unit 14 calculates parameter information (lens distortion coefficients k ₁ and k ₂ ) for correcting lens distortion (aberration distortion) of the camera 20 as adjustment amount information necessary for initial adjustment ( Step S5). The parameter information is lens distortion (aberration distortion) that is particularly noticeable when a wide-angle lens is used. Hereinafter, a method for calculating the adjustment amount information will be described with reference to Equations (1) to (6).

First, a pinhole camera model is assumed in which lens distortion is identified, and projection from 3D information in real space to 2D information on a camera image can be expressed by a matrix. That is, the generation of lens distortion is modeled as follows. Here, the lens center (principal point) of the camera is defined by the following mathematical formula (1). Further, image coordinates observed under the influence of lens distortion before correction are defined by the following mathematical formula (2). Furthermore, ideal image coordinates (ideal image coordinates) that are not affected by lens distortion after correction are defined by the following mathematical formula (3).

Further, model equations (4) to (6) relating to lens distortion correction are as follows.

Here, the lens distortion coefficients k ₁ and k ₂ are parameter information for correcting lens distortion (aberration distortion) of the camera 20 as adjustment amount information necessary for initial adjustment. Α is a parameter relating to the aspect ratio of the image sensor of the camera 20, and can generally be approximated by “α = 1”. Therefore, the actual lens distortion correction parameter information is the lens distortion coefficients k ₁ and k ₂ . R represents the distance from the lens center (principal point) of the camera.

By applying parameters such as lens distortion coefficients k ₁ and k ₂ to the model formulas (4) to (6) for each pixel on the image of the linear input data calculated by the feature amount extraction unit 13, The ideal image coordinates shown in Expression (3) are calculated. A voting is performed on the Hough space by executing the Hough transformation on the ideal image coordinates. If the correction is correct, the value of the point on the Hough space corresponding to each straight line input data has a sharp peak. By evaluating the sharpness of the peak, the optimum lens distortion coefficients k ₁ and k ₂ can be determined, and the lens distortion can be corrected.

  The initial adjustment unit 15 uses the adjustment amount information (optimal lens distortion parameter) calculated by the adjustment amount calculation unit 14 to perform initial adjustment on the image information. That is, the initial adjustment unit 15 executes camera calibration processing (image adjustment processing) for correcting image distortion in accordance with lens distortion of the camera 20. The adjustment result output unit 16 outputs the image information initially adjusted by the initial adjustment unit 15 to the monitoring processing unit 30 included in the monitoring system 1.

  As described above, in the case of the image processing apparatus 10 according to the present embodiment, a moving object such as a person existing in the monitoring target area is extracted based on a photographed image from the camera 20 that uses a wide-angle lens, and the number of people and people In the monitoring system 1 that monitors the distribution of the image, it is possible to realize a function of automating the initial adjustment for correcting the distortion of the image according to the distortion of the lens (aberration distortion). This makes it possible to improve the efficiency of the initial adjustment required for each camera, particularly when a large number of cameras are installed and applied to a system that monitors a wide monitoring area. In other words, a person existing in the monitoring target area is photographed, and the photographed image is analyzed by image processing, so that an initial adjustment can be easily performed on site after installation of the camera 20 without requiring a special adjustment device. This makes it possible to reduce the adjustment man-hours.

  For example, in a building or factory, a monitoring system for grasping the presence / absence, distribution, number of people, etc. in the monitoring area when realizing an energy-saving system in cooperation with an air conditioning / lighting / power supply equipment system In addition, it is effective if the image processing apparatus 10 of the present embodiment is applied.

  Here, in the present embodiment, a case where a human being is monitored as a moving object is described. In this case, assuming that the person walks linearly, the feature extraction unit 13 calculates the center of the human head or torso as the movement trajectory line. This is assumed as linear input data existing on the image. In other words, since a human is walking in a straight line in the real space, it can be assumed that the movement trajectory line measured by the image processing also shows a straight line on the image.

  However, in practice, due to the influence of the aberration distortion of the lens, it is not actually a straight line but a distorted movement trajectory line. Therefore, the optimal lens distortion correction parameter is calculated from the model equations (1) to (6) by making the movement locus line a straight line. In other words, a process for excluding the movement trajectory line when it is clearly not a linear walk from the reference movement trajectory line is performed using the movement trajectory line of the linear walking as the reference movement trajectory line for lens distortion correction.

  Note that the image processing apparatus 10 according to the present embodiment can also be applied to a case where a moving object other than a human being, for example, an animal or a vehicle is to be monitored. However, the movement trajectory line information calculated by image processing differs depending on the characteristics of the moving object.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Monitoring system, 10 ... Image processing apparatus, 11 ... Image input part, 12 ... Human extraction part,
13 ... feature amount extraction unit, 14 ... adjustment amount calculation unit, 15 ... initial adjustment unit,
16 ... Adjustment result output unit, 20 ... Monitoring camera, 30 ... Monitoring processing unit,
100, 110: Computer.

Claims (12)

Image input means for inputting image information captured by the camera;
Means for extracting a moving object region from the image information;
Trajectory calculation means for calculating movement trajectory line information on an image according to the moving trajectory of the moving object from the moving object region;
An image processing apparatus comprising: adjusting means for calculating adjustment information for correcting distortion of the image information in accordance with lens distortion of the camera based on the movement trajectory line information. The adjusting means includes
Using the model with the lens distortion coefficient of the camera, each of image coordinates and ideal image coordinates included in the movement trajectory line information as parameters, determine the optimum value of the lens distortion coefficient,
The image processing apparatus according to claim 1, wherein the adjustment information is calculated based on the optimum value. As the model, in addition to the lens distortion coefficient, image coordinates of the lens center, image coordinates according to lens distortion before correction as image coordinates included in the movement trajectory line information, and lens distortion after correction as the ideal image coordinates Using a model built with each of the image coordinates unaffected by
The adjusting means includes
The image processing apparatus according to claim 2, wherein an optimum value of the lens distortion coefficient is determined based on a result of calculating the ideal image coordinates from the model.   The image processing apparatus according to claim 1, wherein the trajectory calculating unit calculates moving trajectory line information according to a linear movement of the moving object. The adjusting means includes
4. The apparatus according to claim 3, further comprising means for executing a Hough transform on the ideal image coordinates calculated using the model and determining an optimum value of the lens distortion coefficient based on a result of the Hough transform. Image processing apparatus. The locus calculating means includes
6. The apparatus according to claim 1, wherein a human feature is extracted from the moving object region as a moving object, and movement trajectory line information corresponding to the linear movement of the human is calculated. The image processing apparatus according to item 1. The locus calculating means includes
The moving track line information corresponding to the moving track line of the representative part of each tracked moving body is calculated by tracking each moving body in the image from the moving body region. The image processing apparatus according to item.   The movement trajectory line information calculated by the trajectory calculation means includes means for invalidating the movement trajectory line information when the inclination of the movement trajectory line does not change uniformly. The image processing apparatus according to claim 1. Using the image processing apparatus according to any one of claims 1 to 8,
Means for measuring the number of persons present in the monitoring target range based on image information from the camera that images the monitoring target range;
A monitoring system comprising: means for performing camera distortion correction processing on the image information using the adjustment information output from the image processing apparatus.   The image processing apparatus according to claim 1, comprising an electronic component including a microprocessor and a memory mounted on a board. An image adjustment method applied to an image processing apparatus that processes image information captured by a camera,
A process of inputting the image information;
A process of extracting a moving object region from the image information;
A process of calculating movement trajectory line information on an image corresponding to the movement trajectory of the moving object from the moving object region;
A process for calculating adjustment information for correcting distortion of the image information according to lens distortion of the camera based on the movement trajectory line information. A process of inputting the image information;
A process of extracting a moving object region from the image information;
A process of calculating movement trajectory line information corresponding to the movement trajectory of the moving object from the moving object region;
A program for causing a computer to execute a procedure including a process of calculating adjustment information for correcting distortion of the image information in accordance with lens distortion of the camera based on the movement trajectory line information.

Images